# Triton-distributed overlaps compute and communication in one kernel

> Triton-distributed is ByteDance Seed's compiler for kernels that overlap computation with communication, built on OpenAI Triton and aimed at Nvidia and AMD GPUs. It has moved fast since 2025, with expert-parallel dispatch work landing almost monthly, and it now builds as an out-of-tree plugin against stock Triton rather than as a fork.

**ByteDance-Seed/Triton-distributed** — Distributed Compiler and Optimized Parallel Kernels

- Repository: https://github.com/ByteDance-Seed/Triton-distributed
- Website: https://triton-distributed.readthedocs.io/en/latest/
- Stars: 1,570 · Forks: 177
- Language: Python
- License: MIT
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/bytedance-seed-triton-distributed

## The problem is the collective between the matrix multiplies

The README describes Triton-distributed as a distributed compiler designed for computation-communication overlapping, based on OpenAI Triton. The claim it makes is that programmers using it can develop efficient kernels comparable to highly optimised libraries, naming NVIDIA's Distributed-GEMM example and ByteDance's own FLUX as the bar.

The target workload is visible in the news list. Most of the work since mid-2025 is expert parallelism for mixture-of-experts models: dispatch and combine, all-to-all traffic around GEMMs, tensor parallel MoE operators, Mega-EP. The point of overlapping is that in MoE inference a layer alternates between a compute-bound GEMM and a communication-bound all-to-all, and a conventional implementation waits through the collective. Fusing them into one kernel with tiles that issue communication while other tiles compute is the whole idea.

The hardware scope is stated plainly: Triton-distributed currently mainly targets Nvidia GPU and AMD GPU, and can also be ported to other hardware platforms, with the team asking to be contacted if you want to use it on your own hardware.

Because it builds on Triton rather than replacing it, the model is familiar. You write a Triton kernel and gain the ability to express communication inside it.

## Moving from a fork to an out-of-tree plugin

The most consequential line in the README is about branches, and it changed in August 2026.

The news entry for 08/17/2026 says Triton-distributed was upgraded to Triton 3.7 and now builds as an out-of-tree plugin against stock upstream Triton, with the Triton 3.4 line kept on a separate `triton-v3.4` branch. The default branch tracks Triton 3.7.

That is a significant change in maintenance burden. A fork has to carry upstream by hand, and every Triton release becomes a merge. A plugin builds against the Triton you already have, so adopting it means pinning a Triton version rather than pinning a Triton fork.

The caveat sits immediately below it, and it is the one to plan around. Ascend is not on the 3.7 plugin build path yet: `triton-ascend` has not rebased onto 3.7, the README directs Ascend users to the `triton-v3.4` branch for a working build, and the 3.4-era Ascend sources are parked under `ascend/` on the main branch until a plugin-based port exists.

So the branch you need is determined by your accelerator, not by your preference.

## FlashComm is where the recent engineering went

FlashComm is the project's CUDA communication library, introduced on 07/12/2025 with expert-parallel overlap, TMA support, a `TRITON_DIST_CGA_CLUSTER_SIZE` environment knob, Triton host-overhead optimisation, and AMD mori-shmem expert-parallel intra-node and low-latency kernels.

The two most recent entries show the direction of travel.

On 09/01/2026 came fixed-buffer chunked expert parallelism, where dispatch and combine run in chunks over a fixed-size buffer. The stated effect is that peak expert-parallel memory no longer scales with the worst-case token count, which is the kind of change that matters when one long sequence would otherwise size your buffers for the worst case. The same entry notes that overlap buffers are now sized on demand and that the kernels cover a wider range of hidden sizes.

On 09/17/2026 came fused MXFP8 intra-node dispatch, letting expert-parallel traffic send packed FP8 instead of BF16, with bitwise checks against fused and prequantised paths so the two are known to agree. The entry also mentions MXFP8 launch templates, caller-owned expert-parallel output buffers and tightened pinned-buffer lifetime checks.

Elsewhere there is fused inter-node CuTeDSL dispatch and combine on Hopper and Blackwell, an Ascend RDMA support change on 08/06/2026, and an AMD MORI backend update on 08/10/2026 restoring fused-MoE cooperative shared memory APIs.

The repository layout backs this up: a `FlashComm/` directory, `shmem/`, `csrc/`, `include/`, `plugin/`, `lib/` and `3rdparty/` at the root, with `tutorials/` and `unittest/` alongside them.

## Kernels, profilers and papers

Beyond the communication library, several other pieces show how the project is being used.

MegaTritonKernel arrived on 08/13/2025 with a Qwen3 tensor parallel demo on H20 and H800, documented under docs/getting-started/megakernel. An Intra-Kernel Profiler followed on 09/03/2025, documented at docs/getting-started/profiler/intra_kernel_profiler.md, which is the tool you would reach for when a fused kernel is slower than expected and you cannot see inside it with a normal profiler.

The end-to-end inference acceleration demo landed on 07/24/2025 with unified support for both NVIDIA and AMD GPUs. The project also reports support for inference acceleration for ByteDance-Seed/Seed-OSS-36B-Instruct with a 1.33x speedup on 08/24/2025, a figure the project states for its own model without publishing the measurement setup.

The underlying method was documented in the paper `TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives`, accepted at MLSys 2025.

The tests are the most practical entry point for an evaluator. The news list links specific ones by path, including test_allreduce.py, test_gemm_ar.py for GEMM plus AllReduce on H800, test_ag_moe.py and test_moe_reduce_rs.py. A directory of passing tests per accelerator is a better answer to what works than a README claim.

## Version numbers to keep in mind

The release history says research code, which is not a criticism so much as a warning about pinning.

There is a tag called `experimental` for v0.0.1 on 2025-07-11, then v0.0.1-rc on 2025-08-20, then v0.0.2-rc on 2025-09-12 with the title Enhance overlap kernels. Since then the news list has continued at a high rate with no new tag, through the Triton 3.7 upgrade in August 2026 and the September 2026 FlashComm work.

So the newest published version is a release candidate from the middle of 2025, while the default branch carries a year of further work. Anyone pinning to a release is pinning to a Triton 3.4 era build, and anyone tracking main is tracking unreleased code that reorganised itself around a plugin build.

The licence is MIT, with a LICENSE and a NOTICE file at the root. The last push was on 2026-09-18, with `.readthedocs.yaml` at the root and the documentation site at triton-distributed.readthedocs.io.

## Conclusion

Adopt Triton-distributed if you are running MoE inference on Nvidia or AMD hardware and have a problem where collective communication sits between GEMMs, since that overlap is what the project exists to solve. Do not adopt it on Ascend from the default branch, since the README says triton-ascend has not rebased onto Triton 3.7 and points you at the triton-v3.4 branch for a working build. Verify first that your Triton version matches the branch you check out, that your collectives are supported for your topology, and that you can read the tests under python/triton_dist/test/nvidia as the specification of what currently works.

## FAQ

### Which Triton version does Triton-distributed track?

The default branch tracks Triton 3.7 and builds as an out-of-tree plugin against stock upstream Triton. The earlier Triton 3.4 line is kept on the triton-v3.4 branch, which is also the branch the README points Ascend users at.

### Does Triton-distributed support Ascend hardware?

Not on the 3.7 plugin build path yet. The README states triton-ascend has not rebased onto 3.7, recommends the triton-v3.4 branch for a working Ascend build, and notes the 3.4-era Ascend sources are parked under ascend/ until a plugin-based port exists.

### What is FlashComm in Triton-distributed?

It is the project's CUDA communication library, introduced with expert-parallel overlap, TMA support, the TRITON_DIST_CGA_CLUSTER_SIZE environment variable and AMD mori-shmem kernels. Recent work added fixed-buffer chunked dispatch and combine and fused MXFP8 dispatch.

### Which GPUs does Triton-distributed target?

The README says it currently mainly targets Nvidia GPU and AMD GPU, with fused CuTeDSL dispatch and combine work on Hopper and Blackwell and an AMD MORI backend. The team invites contact for other hardware.

## Sources

- [ByteDance-Seed/Triton-distributed on GitHub](https://github.com/ByteDance-Seed/Triton-distributed)
- [License: MIT](https://github.com/ByteDance-Seed/Triton-distributed/blob/main/LICENSE)
- [Project website](https://triton-distributed.readthedocs.io/en/latest/)
- [README](https://github.com/ByteDance-Seed/Triton-distributed/blob/main/README.md)
- [Releases](https://github.com/ByteDance-Seed/Triton-distributed/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/bytedance-seed-triton-distributed