InferenceX publishes day-zero benchmarks and five placeholder rows in its hardware table
Open Source Continuous Inference Benchmark Research Platform, Kimi K2.7-Code, MiniMax M3, DeepSeekv4, GLM5 - GB200 NVL72 vs MI355X vs B200 vs GB300 NVL72 & soon TPUv6e/v7/Trainium2/3 | , Kimi K2.7-Code MiniMax M3 DeepSeekv4 GLM5 - GB200 NVL72 vs MI355X vs B200 vs GB300 NVL72 TPUv6e/v7/Trainium2/3.
At a glance
- What is it?
- This is a continuously-run inference benchmarking platform whose central argument is that fixed-point benchmarks go stale. The parts worth reading are the ones that undercut its own pitch: a support matrix shipped with unfilled rows, three of six project directories marked experimental, a single release that is a queue backport, and official numbers that partly come from rented cloud hardware.
- Who is it for?
- Treat this as a research platform rather than a reference number generator. Its value is the same-day re-measurement model, and that is genuinely different from a benchmark suite you run once, but the result depends on whose hardware and whose software stack produced it, which the project itself concedes.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- SemiAnalysisAI/InferenceX2,041 ★ · +752
| Repository | 2026-07-28 | 2026-11-15 |
|---|---|---|
| SemiAnalysisAI/InferenceX | 1.3k | 2k |
752 Stars in the last 110 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
The hardware support matrix ships five placeholder rows
The officially supported hardware table has eighteen rows. Twelve name real products and are marked as supported: three Blackwell-class rack systems, two individual Blackwell parts, four AMD accelerators across two generations, two Hopper parts, one TPU generation, and one workstation-class GPU part.
The remaining six are marked as coming soon, and only one of those six is a real product. The other five are unfilled template rows reading as chip number one from hardware vendor number one, chip number two from the same vendor, and one chip each from three further vendors.
The numbering is inconsistent as well. One vendor has two placeholder entries while three others have a single entry each, so a reader counting distinct vendors gets a different answer depending on whether they count rows or names.
None of this is dishonest, and a status table that shows intent is arguably more useful than one that omits planned hardware. But it is a partially filled document published as a finished matrix inside a project whose entire argument is about measurement credibility, and that mismatch is the most quotable thing on the page.
Day zero is the product, and the argument for it is sound
The thesis is stated directly in the rationale section, and it is a good one.
Inference performance rests on two things. Hardware delivers step changes roughly once a year, as new accelerator systems ship. Software delivers continuous gains on top of those steps, every day, through kernel-level optimisation, distributed inference strategies and scheduling improvements that move the achievable frontier in incremental releases that can be days apart.
The consequence is stated next: a benchmark run at a fixed point in time goes stale and stops representing what the latest software can achieve.
So the platform's answer is to keep measuring rather than to measure once, and the news list shows that as a repeated commitment rather than a slogan. At least five model launches are described as having continuous benchmarks live from day zero, alongside entries for adding later hardware and for the second version of the platform itself comparing three hardware generations.
That is a real methodological difference from a one-shot benchmark suite, and it is the reason to care about the project at all.
The project concedes that unofficial runs produce worse numbers
There is an important callout near the top of the page, and it is about provenance rather than about the software.
It states that only the original repository contains official results, and that every fork and other repository is unofficial. The stated reason is specific: the benchmark setup and the quality of machines or clouds in unofficial repositories may differ, leading to subpar benchmarking. Unofficial versions must be explicitly labelled as such, and the text adds that forks may not remove the disclaimer.
So the project competes partly on the authenticity of the measurement. That is a legitimate position for a platform whose value is the credibility of its numbers, and it is also an admission that a benchmark result is a property of the run rather than of the model.
On enforcement: an instruction in a README is not binding, and the same page describes the project as Apache licensed, whose notice retention requirement is what actually carries weight when someone redistributes. The trademark symbol in the name is doing similar work. Neither mechanism prevents a fork, but both shape how a fork can legitimately present itself.
Three of six project directories are marked experimental
The repository is not one project. The index lists six directories and three of them carry an experimental or beta label.
The end-to-end inference serving benchmarks are the one with no such qualifier, and they are the flagship. Networking and collective communication benchmarks are marked experimental beta. Operator and kernel level benchmarks are also marked experimental beta. Alongside them sit a system level power modelling directory, a directory for shared components, and one described as the home for remaining experiments.
Read that as a maturity map rather than a feature list. The mature surface is end-to-end serving. The two most technically interesting areas, collective communication and kernel-level operator work, are the ones the project is least sure about. Power modelling is presented without a qualifier at all, which is ambiguous rather than reassuring.
Anyone depending on this should build against the end-to-end directory and treat the other two as something to evaluate rather than to rely on.
The official numbers partly come from rented cloud hardware
The acknowledgements section is unusually specific about hardware provenance, and the specificity is the interesting part.
Two named executives are thanked for providing particular AMD accelerators. Two others are thanked for supporting the work with access to a full rack system and individual accelerators, and the rack access is qualified as having come through a cloud provider.
So the machines behind the official numbers include vendor-donated accelerators and a rack obtained through a hosted cloud, rather than sitting entirely in either vendor's own lab.
Put that next to the fork disclaimer and the tension is visible. The platform argues that unofficial runs are subpar because machine and cloud quality may differ, while its own official figures depend on the generosity of vendors and the behaviour of a cloud provider. That is not a contradiction, since being transparent about it is better than silence, but it does mean that a number on the dashboard is a statement about a particular machine on a particular day as much as it is about a model.
That is true of every benchmark ever published. It is worth remembering here because this project has made measurement provenance its headline.
The only release is a queue backport
The repository has exactly one release, and its tag is a mouthful.
It names a component, a version with a post-release segment, the project name, and a counter, and the release title describes it in two languages as a queue backport for the project.
Now set that against the news section, which describes a first version launch in late 2025 and a second version launch in early 2026. Neither launch is a release. Neither version number appears as a tag.
So the release history and the versioning narrative are two separate things. There is one tag, it describes an incremental change to one subsystem, and there is no artefact you can install by version to reproduce a published result.
For a research platform that is a defensible choice, since the results live on a continuously updated dashboard rather than in pinned packages. It does mean there is nothing to cite as a versioned release, and that a reader who finds a number has no way to tie it to a tagged state of the code.
A standalone mode exists for anyone without a cluster scheduler
The contributing section contains the most practically useful line on the page, and it is easy to miss.
It points to a document for benchmarking an existing server with neither the project's own continuous integration nor a cluster scheduler, covering client installation and a direct command from a profiling tool. So the harness can be pointed at a machine you already have, without the infrastructure the platform itself runs on.
That converts the project from something you read into something you can run, which matters more here than for most research repositories because the whole thesis is about stale numbers. If you can reproduce a number locally on your own hardware, the dashboard stops being the only source of truth.
Two smaller details from the same section. The contributor documentation is organised as a map with an index linking architecture, configuration, workflow, evaluation, runner and troubleshooting references, and there is a separate review checklist for pull requests.
And the repository uses git submodules, so a plain clone does not necessarily give you the whole thing, which is worth knowing before you follow that document.
Editorial conclusion
Treat this as a research platform rather than a reference number generator. Its value is the same-day re-measurement model, and that is genuinely different from a benchmark suite you run once, but the result depends on whose hardware and whose software stack produced it, which the project itself concedes. Two things to weigh before quoting any figure. The support matrix contains placeholder rows, so the coverage story is not finished, and three of the six project directories are labelled experimental, leaving one directory as the mature surface. And with exactly one release, which is a backport of a queue component, there is nothing to install by version and nothing to cite as a tagged artefact.
Frequently asked questions
What is InferenceX?
It is an Apache 2.0 licensed inference performance research platform that continuously analyses and benchmarks the open-source inference frameworks used by major model operators. A live dashboard is published free at inferencex.com, and the dashboard application itself lives in a separate repository.
Why does InferenceX run benchmarks continuously?
Because hardware delivers performance steps roughly yearly while inference software delivers continuous gains daily, so a benchmark taken at a fixed point in time goes stale. The platform keeps re-measuring as the software ships, which is why news entries repeatedly describe benchmarks going live on day zero of a model launch.
Which hardware does InferenceX support?
Twelve SKUs are marked supported, spanning rack-scale and individual Blackwell systems, several AMD accelerators, two Hopper parts, a TPU generation and a workstation-class GPU. Six more are listed as coming soon, but five of those six are unfilled placeholder rows naming an unnamed chip and vendor.
Can I run InferenceX on a server I already have?
Yes. The project documents a standalone mode for benchmarking an existing server without needing its continuous integration or a cluster scheduler, including client installation and a direct command from a profiling tool. Note that the repository uses git submodules, so a plain clone may not fetch everything.
What benchmark projects does InferenceX contain?
Three: end-to-end inference serving benchmarks with no maturity qualifier, networking and collective communication benchmarks marked experimental beta, and operator and kernel level benchmarks also marked experimental beta. Alongside them are a system level power modelling directory, a shared components directory, and one for remaining experiments.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/semianalysisai-inferencex)Star history chart for your README
[](https://hysenlabs.com/projects/semianalysisai-inferencex)Was this useful?