InferenceX: A Live Benchmark Platform for Continuous LLM Inference Performance

The Problem: Stale Benchmarks in a Fast-Moving Field

LLM inference performance is driven by two pillars: hardware and software. Hardware delivers step jumps with new GPUs and XPUs, but software evolves daily through kernel optimizations, distributed inference strategies, and scheduling innovations. A benchmark conducted at a fixed point in time quickly goes stale, representing only what was possible with the software packages available that day. InferenceX was built to address this challenge by providing a continuous, automated benchmark that moves at the same speed as the ecosystem. It tracks the most popular open-source inference frameworks, including SGLang, vLLM, and TensorRT-LLM, and captures their progress in near real-time. This is for engineers and operators who need to know which framework and hardware combination actually delivers the best performance today, not six months ago.

How InferenceX Works: Continuous Tracking and Live Dashboards

InferenceX operates as an open-source, automated benchmark platform. It continuously runs inference workloads on supported hardware and frameworks, capturing performance metrics over time. The results are published on a live dashboard, available publicly at https://inferencex.com/, and the dashboard code itself is open-sourced in a separate repository. The platform tracks model releases from day zero, as seen with Kimi K3 2.8T, MiniMax M3, and DeepSeek V4 Pro, providing a near real-time indicator of inference performance progress. The README emphasizes that only the official SemiAnalysisAI/InferenceX repository contains the official results; forks and other repositories are unofficial and may have different benchmark setups, leading to subpar benchmarking. This centralization ensures consistency in how benchmarks are run, which is critical for comparing results across time and hardware.

Getting Started: What You Need to Run It

The repository is Python-based and licensed under Apache-2.0. To get started, you would clone the repository and follow the documentation, which includes a docs/index.md that links to architecture, configuration, workflow, eval, runner, and troubleshooting references. The README does not provide explicit installation commands, so you would need to consult the docs for exact steps. However, the platform is designed to run on specific hardware, including GB200 NVL72, GB300 NVL72, MI355X, B200, B300, MI325X, MI300X, H200, and H100. If you have access to such systems, you can contribute benchmarks or run your own. The project also encourages contributions via pull requests, with a CONTRIBUTING.md and a PR review checklist. For most users, the easiest path is to use the public dashboard rather than running the platform yourself, given the hardware requirements.

Supported Hardware and Software Stacks

The officially supported hardware list is extensive, covering NVIDIA and AMD data center GPUs. As of the latest update, GB300 NVL72, GB200 NVL72, MI355X, B300, B200, MI325X, MI300X, H200, and H100 are all marked as supported. The platform is actively used to benchmark models like GLM5.3, Kimi K2.7-Code, MiniMax M3, DeepSeekv4, and Qwen3.5. It also tracks specific software stacks, including SGLang, vLLM, TensorRT-LLM, CUDA, and ROCm. The README notes that AMD contributors have been responsive in debugging and optimizing performance on AMD GPUs, and NVIDIA teams have supported access to GB200 and B200 systems. This breadth of support makes InferenceX a comprehensive resource for comparing performance across major hardware vendors. However, TPU support is listed as 'Coming Soon', so Google's accelerators are not yet covered.

A Real Limitation: The Barrier to Entry and Trust

The most significant limitation is the hardware requirement. Running InferenceX requires access to expensive, high-end GPU systems like GB200 NVL72 or MI355X. This is not a tool for a small team with a few A100s; it is designed for operators of large-scale token factories. The README explicitly mentions being trusted by OpenAI, Meta, Microsoft, and Oracle, which indicates the scale of the target audience. For anyone else, the value is in consuming the results, not producing them. Another limitation is the reliance on the official repository for trustworthy results. Unofficial forks may produce different outcomes due to varying benchmark setups, so users must be careful to only reference the official data. This creates a single point of trust, which is fine for a research platform but could be a concern if the official maintainers ever stop updating it.

Alternative Approaches: Static Benchmarks vs. Continuous Tracking

The primary alternative to InferenceX is static benchmarking, where performance is measured once per release or per hardware launch. Projects like the LMSYS Chatbot Arena, which the README references in a blog post about GB300, provide point-in-time comparisons. Static benchmarks are easier to run and require less infrastructure, but they quickly become outdated as software improves. Another alternative is in-house benchmarking, where an organization sets up its own test harness to evaluate frameworks on its specific workloads and hardware. This offers full control over the benchmark parameters but is time-consuming and requires ongoing maintenance. InferenceX differentiates itself by automating the continuous aspect, freeing users from the need to re-run benchmarks manually. The trade-off is that you lose the ability to customize the benchmark to your exact use case unless you contribute to the project.

Maintenance and Upgrade Cost

InferenceX is actively maintained, with recent releases and a last push in August 2026. The project has a clear contribution process, including a PR review checklist and a maintainer documentation map. The Apache-2.0 license allows for commercial use and modification, which is favorable for integration into internal tooling. However, the maintenance cost for users is non-trivial if they choose to run the platform themselves. You would need to keep up with software updates for the frameworks being benchmarked, as well as the InferenceX codebase itself. The project also has a queue backport release for TileRT, indicating that it tracks low-level runtime changes. For most users, the maintenance burden is low because they only consume the dashboard, but for contributors, it requires a commitment to keeping benchmarks current and accurate.

Editorial conclusion

Adopt InferenceX if you run large-scale LLM inference and need current, comparable performance data across frameworks and hardware, especially for NVIDIA and AMD GPUs. Skip it if you lack the compute resources or need benchmarks for unsupported hardware like TPUs. Before relying on results, verify the exact software versions and hardware configurations used, since performance varies daily. The platform is valuable for tracking progress, but its credibility rests on the official repository's setup, so always reference the official results and not unofficial forks.

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes