Shard: Pipeline-Parallel LLM Inference Across Machines on the Open Internet
Pipeline-parallel LLM inference across GPUs on separate machines.
At a glance
- What is it?
- Shard splits a transformer into contiguous layer blocks, one per GPU, and streams activations between them over WAN links. Its own receipts claim 30.15 tok/s for a 284B model on six RTX 5090s in four countries, and roughly 30 tok/s for a 744B model across six US states. The design is latency-first, and the documentation is honest about where the levers stop paying.
- Who is it for?
- Adopt Shard if you have several GPUs on separate networks and a model that will not fit on one card, and you are willing to run one engine per model rather than a generic server. Do not adopt it if you need a single-machine deployment, a stable plugin API, or a serving stack with published compatibility guarantees across model families.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 37 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap Shard is aimed at: models bigger than any single card, on machines you do not own
The problem is specific. A frontier-size model does not fit in the VRAM of one GPU, and the usual answers are tensor parallelism inside a single host (fast interconnect, one failure domain) or pipeline parallelism inside a datacenter (NVLink or InfiniBand between nodes). Shard targets a third case: GPUs on separate machines, on separate networks, connected only by the public internet. The README frames the ambition as a permissionless compute network, describing the project as BitTorrent-like where participants share VRAM and compute rather than disk. The stated audience is whoever wants to pool heterogeneous consumer or prosumer cards into a swarm that runs a model too large for any one of them. The repository is a protocol spine (the README names wire, ring, receipts, placement) plus one engine per model. Three engines are listed: MiniMax-M2.5, Kimi-K3, and DeepSeek-V4-Flash. That per-model structure is the first thing to notice, because it tells you what kind of project this is. This is not a generic inference server with a model registry. It is a set of tuned implementations that share a transport and a placement layer, with a stated long-run direction of driving all of them behind a single ModelRuntime interface documented in docs/MODEL_RUNTIME.md. If your model is not one of the three, the interface document is where you find out how much work is in front of you.
Contiguous layer blocks, a direct-return ring, and why the round trip is the scarce resource
The mechanism is pipeline parallelism by layer range. A transformer is a stack of layers; Shard splits the stack into contiguous blocks and assigns one block per GPU. A token is produced by passing activations through the blocks in order. No node holds the whole model, and the README states that each node loads only its own block. In the GLM-5.2 run, 78 layers are split 13 layers per node across six GPUs. The coordinator is a special role: in that run it holds no model layers at all, only the token embedding and head plus a small GLM-4-9B draft model. The README's central claim about performance is that over WAN the round trip, not compute, is the scarce resource. Plain KV decode is latency-bound at one round trip per token, which the project's own step table puts at 1.87 tok/s. Everything after that is an attempt to convert a latency-bound loop into a throughput-bound one. Two changes do most of the work. Ring direct-return lets the tail node return results to the coordinator in one hop instead of relaying back through every intermediate node, which the table credits with moving 1.99 to 2.94 tok/s. Async pipelining then overlaps many verify traversals so the pipeline runs at its throughput rather than its latency, which the table credits with 16.6 tok/s and drops the WAN to roughly 5% of the loop. The final step, CUDA-graphing the draft model, is credited with reaching about 30 tok/s. The interesting part is the ordering: the draft model only became the bottleneck after the network was hidden. That is a property of this architecture, not a general one.
Speculative decoding does the heavy lifting, and the acceptance trade-off is 1:1
Speculative decoding is normally a marginal optimization in a datacenter, where a round trip is cheap. Shard inverts that. A small draft proposes K tokens, the distributed model verifies them in a single pipeline traversal, and greedy acceptance commits the verified prefix. One traversal then yields several tokens, which is exactly what you want when traversals are expensive. The README is candid about the cost. It states that throughput is useful in-flight work over round-trip latency, and that every lever which fills the pipe costs acceptance at roughly 1:1, so forcing the pipe to 99.6% of its cap halves throughput because the added frames speculate on a history the ring does not take. That is a real constraint, not a tuning note, and it means there is an interior optimum rather than a monotonic dial. The same section says the draft block has an interior optimum that only appears if you measure the middle rather than the endpoints. Two consequences follow for anyone evaluating this. First, the headline numbers are the product of a specific operating point, and moving off it degrades throughput in a way the documentation describes as roughly proportional. Second, the DeepSeek-V4-Flash run uses DeepSeek's own DSpark drafter, whose three MTP blocks tap the last three layers and therefore live entirely on the tail box. Draft placement is a design decision that varies per model, which is another reason the per-model engine structure exists.
Getting it running: what the material actually specifies
The README does not give an install command, a pip invocation, or a launch script. What it does give is a set of file paths that define the operational surface. The engine implementations referenced are research/glm_swarm_nvfp4_cg.py and research/glm_swarm_nvfp4_cg_diff.py, the latter being the differential check that the CUDA-graphed path is byte-identical to the eager path. The architecture and interface documents are docs/MODEL_RUNTIME.md, docs/V4_FLASH_ENGINE.md, and docs/PROOF.md. The evidence artifacts are under docs/receipts/, with named examples including docs/receipts/v4-flash-matrix-20260802.json and docs/receipts/glm52-nvfp4-wan-20260618.json. The technical report lives at docs/paper/main.pdf. Releases are tagged with a sidecar prefix and a linux-amd64 platform suffix, so sidecar-v0.2.0 and sidecar-v0.1.0 are the two artifacts published so far. That release naming is worth reading carefully: the project ships a sidecar component, and the release notes in the supplied material do not describe what the sidecar does or how it relates to the engines. Anyone planning a deployment should treat that as an open question to resolve from the repository rather than assume the sidecar is the full serving path. Beyond paths and release tags, the material does not specify configuration keys, environment variables, or a CLI. I am not going to invent them. The honest summary is that the repository documents an architecture and a proof methodology more thoroughly than it documents installation.
The receipts are the strongest part of the project, and they are also the boundary
Most distributed inference projects publish a throughput number and a topology diagram. Shard publishes signed receipts. The README states that every run emits a verifiable receipt containing distinct GPU UUIDs, public IPs, regions, measured WAN edge RTTs in the 22 to 75 ms range, the output token hash, and a lossless-optimization check. It points a skeptic at docs/PROOF.md for the verification procedure. The DeepSeek-V4-Flash matrix is described as 51 of 51 cells bit-identical to a greedy baseline on the same ring, with 68 of 68 signed receipts and zero faults, and the headline figure is given as a median of three consecutive warm runs within 0.17 of each other (30.18, 30.29, 30.12). The GLM-5.2 run is described as greedy and deterministic across six US states. This is the correct way to present a claim like this, and it is the reason the numbers are worth taking seriously at all. It is also the boundary of what you can conclude. Bit-identical output to a single-machine baseline demonstrates that the sharding did not change results. It does not demonstrate throughput on your hardware, your interconnects, or your model. The RTT range of 22 to 75 ms is quoted for one set of links; the README does not give a model for how throughput degrades as RTT grows past that, and the acceptance-versus-pipe-fill trade-off suggests it would not be a gentle curve.
Where Shard is the wrong tool
The clearest case against Shard is a single machine with enough VRAM. If the model fits on one card, pipeline parallelism across the network adds a round trip to every traversal and buys nothing. The project's own baseline table supports this: plain KV decode over WAN is 1.87 tok/s, and the entire engineering effort from there exists to recover speed that a local deployment never loses. A second case is a model outside the three listed engines. The README is explicit that Shard is a spine plus one engine per model, because getting a model to interactive speed over the open internet means tuning down to the kernels. There is no claim that an arbitrary Hugging Face checkpoint will run. The ModelRuntime interface in docs/MODEL_RUNTIME.md is described as a long-run direction, not a finished abstraction, so a new model family means writing an engine. A third case is a latency-sensitive interactive workload on unreliable links. The architecture assumes a ring of nodes that stay reachable; the README describes placement, receipts, and verification, but the supplied material does not describe failure handling when a node in the middle of the ring drops out mid-request. That is a gap worth confirming before anything production-facing. Finally, if you need a serving stack with a stable API surface and published compatibility guarantees across model families, this is not that. It is a research-grade engine with a protocol layer, and the per-model tuning that produces its numbers is the same thing that makes it hard to generalize.
How this differs from vLLM with pipeline parallelism
The closest well-known alternative is vLLM configured for pipeline parallelism, which splits a model across GPUs in the same way conceptually. The difference is the assumption about the interconnect. vLLM's pipeline mode assumes GPUs that can exchange activations over NVLink or a fast local fabric, and it is normally deployed inside one host or one datacenter. Shard assumes the opposite: separate machines, separate networks, tens of milliseconds between hops, and no shared host. That single assumption cascades. Because round trips are expensive, Shard invests heavily in speculative decoding and async pipelining, which vLLM treats as optional accelerators rather than the core of the design. Because nodes are not co-located and not necessarily trusted, Shard adds signed receipts with GPU UUIDs, public IPs, regions, and output token hashes, plus a lossless-optimization check. vLLM has no equivalent, because in a datacenter you trust the fabric. The trade is generality for reach. vLLM serves a very wide range of models through a common runtime; Shard serves three, each with its own engine, and is working toward a common interface. If your GPUs are in one chassis, vLLM's approach is simpler and will be faster. If they are in six states, Shard is addressing a problem vLLM's pipeline mode was not designed for.
Maintenance, licensing, and what to check before you commit
Shard is Apache-2.0, which permits commercial use, modification, and redistribution provided you retain the license and notices. That is a permissive license with no copyleft obligation on your own code. It is not legal advice, and the specifics of attribution and patent terms are worth reading in the LICENSE file rather than taking from a summary. Maintenance cost is the harder question, and the material points in two directions. On one side, the project is active: the most recent push in the supplied metadata is dated 2026-08-09, and two sidecar releases landed in July 2026, with sidecar-v0.1.0 on 2026-07-21 and sidecar-v0.2.0 on 2026-07-29. On the other side, the per-model engine structure means upgrade cost is not uniform. A change to the protocol spine or the ring behavior can require retuning each engine, and the README's own account of the CUDA-graph work shows how deep that tuning goes: making the static KV cache honor speculative rollback under graph capture required driving the write slot through a static-address position tensor. That is the kind of change that has to be re-derived per model rather than inherited. The release cadence also matters for a different reason. Only sidecar artifacts are tagged, and the supplied material does not explain what the sidecar contains or whether the engines are distributed through it. Before deploying, confirm from the repository which component you are actually installing and how it relates to the engine files under research/. The receipts are the other thing to verify directly. docs/PROOF.md describes how to check one, and a single verified receipt from docs/receipts/ tells you more about whether the claims hold on your reading than any summary, including this one.
Editorial conclusion
Adopt Shard if you have several GPUs on separate networks and a model that will not fit on one card, and you are willing to run one engine per model rather than a generic server. Do not adopt it if you need a single-machine deployment, a stable plugin API, or a serving stack with published compatibility guarantees across model families. Before committing, read docs/PROOF.md and verify one receipt under docs/receipts/ yourself, then check whether your target model already has an engine in the repository or only the ModelRuntime interface in docs/MODEL_RUNTIME.md.
Community notes