FlashRT: A No-Compile Kernel Engine for Small-Batch Realtime VLA and LLM Inference

The workload FlashRT Was Built Around: One Decision at a Time

Most inference stacks optimise for throughput under batch. TensorRT compiles a model through tactic search into a frozen engine. vLLM and SGLang schedule many concurrent requests against a paged KV cache. Both are reasonable designs, and both impose costs that a robot control loop cannot pay. A vision-language-action policy issues one decision, waits for it, acts, and issues the next. The batch is one. The metric that matters is the wall-clock latency of that single decision, and the compile step that makes TensorRT fast at serving time makes it slow to iterate on at development time. FlashRT takes the position that this cell, small-batch and latency-sensitive, deserves its own engine rather than a configuration of an existing one. The README states the target directly: "The pattern is workload-shaped (small-batch realtime), not model-class-shaped." The flagship integration is VLA control, with frontends for Pi0, Pi0.5, GROOT N1.6, GROOT N1.7 and Pi0-FAST, validated on LIBERO where applicable. The same kernel set is also used for Qwen3.6-27B NVFP4 long-context serving, BAGEL world-model paths, Higgs Audio v3 TTS, and Wan2.2 / Motus video-policy paths. The audience is narrow and identifiable: engineers running a fixed checkpoint on a Jetson AGX Thor, an A100, or an RTX 4090/5090, who have already found that the host framework's own execution path is the bottleneck.

Static Graphs of Hand-Written Kernels Instead of a Compile Step

The mechanism is composition, not compilation. FlashRT ships a general kernel library covering norm, activation, fusion, RoPE, FP8 and NVFP4 GEMM, and attention primitives, which the README says cover standard transformer, DiT and SigLIP blocks. Those kernels are composed into static graphs. There is no ONNX export, no engine compilation, and no per-driver rebuild. That last point is the one that changes operational behaviour: a TensorRT engine is tied to the driver and GPU combination it was built against, so a driver update can force a rebuild. FlashRT's claim is that the graph is assembled from kernels that are already compiled, so the driver does not enter into it. The README also notes that the composition pattern itself is hardware-agnostic, while the shipped implementations are NVIDIA-only, spanning Jetson AGX Thor through A100, RTX 4090 and RTX 5090. That gap between the pattern and the implementations is worth reading carefully. A hardware-agnostic composition layer is an architectural statement about how kernels are wired together; it is not a claim that the code runs on anything other than NVIDIA hardware today. The kernel list is also the boundary of what the engine can express. If a model needs an operator outside norm, activation, fusion, RoPE, FP8/NVFP4 GEMM and attention, the composition approach has nothing to compose.

What the Demos Actually Measure, and What They Do Not

The README's demo section is unusually specific about methodology, which is worth noting because most projects are not. Each film shows one checkpoint executed several ways, side by side on one wall clock, with each pane advancing at the rate it was measured at. The GIFs on the page are sped up, but the linked real-time recordings and the Hugging Face walkthrough are described as playing one second per second. For Pi0.5 on a Jetson AGX Thor running LIBERO-spatial task 1 with one flow-matching sample held fixed, the reported per-decision latency is 266.0 / 303.6 ms for the two hosts as their authors ship them, 36.1 ms for one accelerated form, and 29.2 ms for the hand-written FlashRT pipeline, which the README translates to 3.8 Hz rising to 34.2 Hz of policy decisions, with the task finishing in 6.0 s instead of 23.5 s. On an RTX 5090, the same checkpoint is shown at 107.5, 58.9, 25.6 and 21.6 ms across four execution strategies. GR00T N1.7 on the 5090 is listed at 44.4, 25.4, 17.8 and 15.8 ms. The cross-host comparisons are the more interesting result: Pi0.5 under LeRobot goes 107.5 to 25.6 ms and under OpenPI 41.7 to 28.8 ms, so two hosts that start 2.58x apart end within 12 percent of each other. For GR00T N1.7 under Isaac-GR00T and the LeRobot port, the two hosts are reported to agree to a cosine of 0.999995 on one decision made from a byte-identical observation. What these numbers do not tell you is how the engine behaves under sustained load, thermal throttling on the Jetson, or a checkpoint outside the listed frontends. They are single-task, single-sample measurements, and the README presents them as such.

Getting It Running: Quantisation on Adopt and the LLM Path

The README does not include an install command, a build invocation, or a repository layout section, so the exact steps to build FlashRT from source cannot be confirmed from the supplied material. What it does document is one configuration key and its effect. In the Qwen3.6-35B-A3B demo on a single 32 GB card, 67 GB of BF16 weights do not fit in memory. The key `quantize_on_adopt` regrids the expert banks before the model reaches the device, bringing the resident footprint to 22.3 GiB, at which point the model runs. That is a load-time transformation rather than a post-load quantisation pass, and the name reflects it: the weights are quantised as they are adopted onto the device. The same demo reports decode at 51.6, 203.5 and 284.9 tok/s across three arms, the third being the checkpoint's own draft head at 3.43 tokens accepted per round. The README is explicit that the draft head is "shown on the prompt where it wins, dropped on the one where it does not", which is a more honest presentation of speculative decoding than a single aggregate number. The second LLM demo attaches FlashRT inside vLLM on a Jetson AGX Thor at 1, 4, 8 and 16 concurrent requests, with aggregate throughput reported as 38.4 to 77.4, 77.5 to 182.8, 98.9 to 245.5 and 211.4 to 302.1 tok/s. The README's explanation for why every level gains is that a routed mixture of experts does not dilute the way a dense model does, because each token reads its own experts, so expert weight traffic grows with the batch instead of being shared. Anyone evaluating the project should treat that as the project's own reasoning about its measurements, not as an independent result.

The Cost of a Fixed Pipeline: No Dynamic Shapes, No Model Swapping

Composing kernels into a static graph buys the absence of a compile step and pays for it in flexibility. A static graph is fixed at the point it is assembled. TensorRT's tactic search exists precisely because the optimal kernel for a given shape and precision is not always obvious, and it resolves that at build time by measuring. FlashRT resolves it by hand, which is why the kernel set is enumerated rather than generated, and why the README describes the kernels as hand-written and hand-tuned. The consequence is that a model whose operator set falls outside norm, activation, fusion, RoPE, FP8/NVFP4 GEMM and attention has no path through the engine without new kernels. The frontend list reinforces the point: Pi0, Pi0.5, GROOT N1.6, GROOT N1.7 and Pi0-FAST are named as production frontends, and the LLM coverage is described as single-stream with Qwen3.6-27B NVFP4 long-context serving, with Qwen3.6-35B-A3B appearing in the demos. There is no indication of a general model-loading path that accepts an arbitrary checkpoint. The vLLM attachment demo is the exception that tests the rule: it puts FlashRT's kernels underneath vLLM's scheduler rather than replacing it, which means the small-batch assumption is relaxed by borrowing someone else's batching layer. That is a sensible integration, but it is a different deployment shape from the standalone engine, and the README treats it as such.

How FlashRT Differs From TensorRT and vLLM

The README positions both alternatives explicitly. TensorRT is described as tactic-search compile to frozen engines; vLLM and SGLang are described as high-batch LLM serving. The difference with TensorRT is the compile step and its coupling to the driver. A frozen engine is fast, but producing it takes time and the artifact is tied to the environment that produced it. FlashRT removes both the build latency and the rebuild-on-driver-change problem, at the cost of the per-shape autotuning that TensorRT performs. The difference with vLLM is the batching assumption. vLLM's scheduler, paged attention and continuous batching are designed to keep a GPU busy with many concurrent sequences; a single robot policy decision has nothing to batch. FlashRT targets that single-sequence case directly. The two are not mutually exclusive, and the README's own demo attaches FlashRT inside vLLM, which suggests the intended relationship is complementary rather than competitive for LLM serving. For VLA control there is no equivalent attachment story in the material, so the comparison there is closer to a straight substitution of the host framework's execution path. A team already running Pi0.5 through LeRobot or OpenPI is the clearest candidate, because the README measures both hosts directly and reports the two converging to within 12 percent of each other.

Maintenance, Licensing, and the Missing Release Artifacts

FlashRT is licensed Apache-2.0, which permits commercial use, modification and redistribution provided the licence and notices are preserved. That is a permissive licence with a patent grant, and it is the same licence used by much of the surrounding ecosystem, so there is no obvious compatibility problem with linking against it. Nothing here constitutes legal advice, and a team embedding FlashRT in a shipped product should have its own counsel review the NOTICE requirements and any third-party kernel attributions in the repository. On maintenance, the material is thin in one specific way: no releases were retrieved, and the README describes the project as shipping with NVIDIA implementations today, with the composition pattern described as hardware-agnostic. There is no versioning scheme, no changelog and no stated support window in what is available. The last push to main is dated 2026-09-10. For a project with no tagged releases, adopting it means tracking main, which in turn means that a kernel change can alter numerics without a version number changing. The README's own cosine-of-0.999995 comparison between two GR00T hosts shows the project is attentive to numerical agreement, but that is a measurement of two hosts against each other, not a guarantee of stability across commits. The arXiv paper and the separate FlashRT-HF-kernels repository are the two places to check whether the kernel set in main matches what is documented, since the README links both without stating how they relate to the main tree.

Editorial conclusion

FlashRT fits teams serving a fixed set of checkpoints at batch size one on NVIDIA edge or desktop hardware, where TensorRT build time and vLLM's batch-oriented scheduler are both costs rather than benefits. It does not fit anyone who needs per-request model swapping, non-NVIDIA accelerators, or a supported release artifact, because the repository has no tagged releases and the composition pattern is described as hardware-agnostic in principle while only NVIDIA implementations ship today. Before adopting, verify the pinned CUDA and driver versions for your target board, confirm the specific checkpoint you intend to serve appears in the frontend list, and check whether the arXiv paper and the FlashRT-HF-kernels repository match the kernel set in main.

Official sources

1. flashrt-project/FlashRT on GitHub
2. Issues
3. License: Apache-2.0
4. README