Yunchang merges Ulysses and Ring attention into one sequence parallel path

Two attention methods, each with a ceiling

The repository is called long-context-attention, and the code is published under the name Yunchang, with a tech report titled USP: A Unified Sequence Parallelism Approach for Long Context Generative AI. It sits on top of zhuzilin/ring-flash-attention and refers to DeepSpeed-Ulysses, and the README explains the problem by explaining why neither method alone is enough. Ulysses is sensitive to the number of attention heads, because the parallelism degree cannot exceed the number of heads, which rules it out for Grouped Query Attention and Multi-Query Attention scenarios, and the text notes that Ulysses does not operate effectively with a single head. On the Ulysses side the constraint compounds, since Tensor Parallelism also divides across the head dimension, so making the two compatible is difficult. Ring-Attention, in turn, is described as less efficient than Ulysses in computation and communication, because it segments QKV into smaller blocks, which lowers efficiency when FlashAttention is used, and even with full overlap of communication and computation the total execution time lags behind Ulysses. Ring also relies on asynchronous peer-to-peer communication, which has lower bandwidth utilisation than collective methods and risks deadlocks at large scale.

LongContextAttention as a drop-in replacement

The class at the centre of the package is LongContextAttention, which the README also calls Unified Sequence Parallelism and Hybrid Sequence Parallelism. The usage pattern is short, and the README walks through it in three steps with the zigzag ring implementation as the example. First you apply set_seq_parallel_pg to set the process group. Second you extract local tensors with zigzag_extract_local, because the input tokens have to be reordered for load balanced ring attention. Third you apply LongContextAttention with ring_impl_type set to zigzag as a drop-in replacement for your attention implementation. The naming makes the hybrid structure explicit, since the process group is configured with two degrees, one for Ulysses and one for the ring, and in the example sp_ulysses_degree is 2 while sp_ring_degree is 4, supporting a world size of 8. The ring implementation type is a parameter, so the same class covers several strategies, and the attention type is a second parameter, with the README naming FA, FA3 and TORCH_EFFICIENT as the options.

FlashAttention decides which path you take

Installation is where this project is most likely to cost you an afternoon, and the README says so directly. FlashAttention is the most important external dependency and is often the cause of errors when installing and using yunchang. The supported range is flash_attn 2.6.x and 2.7.x, covering both v3 and v2 versions, and the package also supports running without flash_attn, which the README says is suitable for NPUs. The mapping from hardware to path is laid out as four cases. For H100 and B100 hardware that supports FA v3, ring_flash_attn uses FA v3. For A100 and L40 hardware that supports FA v2, it uses FA v2. For hardware such as NPUs that does not support FA, attention is computed with torch, there is no need to install flash_attn, and you apply LongContextAttention with ring_impl_type basic and attn_type AttnType.TORCH_EFFICIENT, and the backward pass is not supported for that type. For Intel XPU, the Xe GPU and BMG, you use sgl-kernel-xpu which provides flash_attn_varlen_func for XPU, with ring_impl_type basic_xpu and attn_type AttnType.XPU, and the backward pass is not supported there either.

Installing the package and running the tests

The pip route is the simple one, and the package is named yunchang on PyPI.

`pip install flash-attn`

`pip install yunchang`

If you prefer to build from the local checkout, the command is the same one you would expect, and the README also points at separate install guides for AMD GPUs in docs/install_amd.md and for Intel XPU in docs/install_xpu.md.

`pip install .`

FlashAttention 3 is a special case, because it is still beta released and has to be installed from source following the upstream instructions, and the README records the exact commit the project applied, the Nov 10 2024 commit b443207c1fc4c98e4532aad4e88cfee1d590d996, which is the sort of detail that saves an hour of version archaeology. If you did not install the package, the tests can still be run by putting the project root on the Python path.

export PYTHONPATH=$PWD:$PYTHONPATH

The test scripts are then launched with torchrun. The example in the documentation runs four processes with a Ulysses degree of two and the zigzag ring implementation.

torchrun --nproc_per_node=4 ./test/test_hybrid_attn.py --sp_ulysses_degree 2 --ring_impl_type "zigzag" --causal --attn_impl fa --use_bwd

The package metadata tells you the rest of the constraints. The version is 0.6.4, it requires Python 3.7 or newer, and its only hard dependency is torch at 2.3.0 or newer, with flash-attn at 2.6.0 or newer and sglang-kernel-xpu as optional extras.

The call site, in the shape you will copy

The example code in the README is worth reading in full, because it shows the pieces you have to own when you integrate this. You import the attention classes and the process group helper from yunchang, and the attention type enum from yunchang.kernels.

from yunchang import (
    AsyncLongContextAttention,
    LongContextAttention,
    set_seq_parallel_pg,
    EXTRACT_FUNC_DICT
)
from yunchang.kernels import AttnType

sp_ulysses_degree = 2
sp_ring_degree = 4

You then set the process group with those two degrees plus the rank and world size, and construct the attention module with the ring implementation type and the attention type.

# attn_type could be FA, FA3, TORCH_EFFICIENT.
longctx_attn = LongContextAttention(ring_impl_type="zigzag", attn_type=AttnType.FA)

The extraction step is where the load balancing happens, and the call takes the global Q, K and V and returns the local shard.

# extract a local shard for the global Q, K, V.
local_q = EXTRACT_FUNC_DICT["zigzag"](
        Q, rank, world_size=world_size, rd=sp_ring_degree, ud=sp_ulysses_degree
    ).detach().clone()

The attention call itself takes dropout, a causal flag with a comment noting that zigzag and stripe are the load balance strategies for causal=True, a window size, softcap, alibi slopes, a deterministic flag and return_attn_probs. The presence of alibi slopes and softcap in the signature tells you this is aimed at long context autoregressive models rather than a general attention utility.

Where the two-way split breaks down

The two forward-only paths are the sharpest limitation. On NPU hardware without FlashAttention, the torch implementation gives you inference but no backward pass, which means you can serve a long context model on that path and you cannot train one there. The same applies to the Intel XPU path, where the sgl-kernel-xpu kernel is forward only. That is a coherent design decision rather than an oversight, since these are inference-oriented targets, but it means the feature matrix is not uniform and the README has to be read as a table of which combinations support training. A second limitation is the dependency floor. FlashAttention has to be compiled or matched to your GPU generation, and the README treats FlashAttention 3 as a source install pinned to a specific commit, so reproducible environments need that pin recorded. A third is the scope. This is a research artifact from a paper, published with a tech report, and the package version has moved slowly, with 0.6.3 in April 2025, a post release 0.6.3.post1, and 0.6.4 in January 2026.

The uptake signal is in NVIDIA's transformer engine

The most useful piece of evidence about this code is not the paper but the adoption. The README states that USP has been applied in NVIDIA's TransformerEngine, in the function AttnFuncWithCPAndKVP2P, and that you can use it through the API attn_forward_func_with_cp. That is a meaningful check on the approach, because TransformerEngine is the reference implementation many teams use for fused attention on Hopper and Blackwell hardware, so a technique that survives there has been tested against real training runs rather than a small experiment. Two other signals in the repository are worth noting. The project builds on zhuzilin/ring-flash-attention, so the ring implementation lineage is traceable rather than invented from scratch. And the repository includes a benchmark/ directory, patches/ and a scripts/ directory, plus a docs/ directory with the AMD and Intel installation guides, which is the shape of a project that expects people to run it on several hardware types. The licence is Apache-2.0, with the LICENSE.txt file in the repository root and the classifier License :: OSI Approved :: Apache Software License, which is permissive enough to use inside a commercial training stack without copyleft obligations on your own code.

Editorial conclusion

Adopt Yunchang when you are training or serving a long context model on more GPUs than your attention head count allows, and you need Ulysses style efficiency without giving up Ring's reach. Skip it if a single node fits your context, since the whole point is distributed sequence parallelism, and skip it if you cannot install FlashAttention, because the README names that dependency as the most common cause of installation errors. Before you integrate it, read the Ulysses head limit in the README yourself and confirm your hardware path, because the attention type you pick determines whether a backward pass exists at all, and the TORCH_EFFICIENT and XPU paths are forward only.

Frequently asked questions

Official sources

1. feifeibear/long-context-attention on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases