# InfiniTensor is a graph executor split across three layers

> InfiniTensor imports, transforms and executes computation graphs, with a native CPU runtime and accelerator execution delegated to InfiniOps and InfiniRT. It suits teams already building the lower stack, and it leaves single-device execution, ABI matching and hardware support outside its own code.

**InfiniTensor/InfiniTensor** — InfiniTensor is a high-performance inference engine tailored for GPUs and AI accelerators. Its design focuses on effective deployment and swift academic validation.

- Repository: https://github.com/InfiniTensor/InfiniTensor
- Stars: 528 · Forks: 78
- Language: C++
- License: Apache-2.0
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/infinitensor-infinitensor

## A graph executor, not a model library

InfiniTensor describes itself as a high-performance inference engine tailored for GPUs and AI accelerators, with a design focused on effective deployment and swift academic validation. The README is more concrete than that description: InfiniTensor imports, transforms and executes computation graphs. Those three verbs are the scope.

Importing means reading a model through a frontend, and the project ships an ONNX frontend with its own test target. Transforming means rewriting the graph before execution, which is where the research contributions live, with EinNet providing derivation-based tensor program optimization and PET providing partially equivalent transformations with automated corrections. Executing means running the result, and here the runtime splits.

The audience is therefore people who already have a model and a reason to care about how it is executed: a system engineer fitting a workload onto a particular accelerator, a researcher evaluating a transformation technique, or a platform team that wants the graph machinery without writing a runtime from scratch. It is not a model zoo, not a training framework and not a serving platform with an HTTP interface.

## The three layers, and why only one of them lives here

InfiniTensor provides a native CPU runtime. For everything else it uses InfiniOps for the operator implementations and InfiniRT for the runtime layer, and the README is explicit that hardware-specific SDK headers and implementations stay outside InfiniTensor.

That single sentence is the most important operational fact in the project. The accelerator devices and operators you can use are determined by the installed InfiniOps and InfiniRT build, not by InfiniTensor itself. So the question when you evaluate the project is never what does InfiniTensor support, but what does the InfiniOps build on this machine support.

The repository layout shows the layers as separate pieces of work. There is a 3rd-party directory and a proto directory for the protocol definitions, src and include for the C++ core, a python directory plus pyinfinitensor for the Python binding, examples for distributed and Python usage, scripts for helper tooling, and an env.sh for environment setup. A CMakeLists.txt and a Makefile sit at the root, and the Makefile passes the lower stack paths straight through to CMake as INFINIOPS_ROOT and INFINIRT_ROOT.

The roadmap points the same way. RefactorGraph is described as the next-generation graph framework under development, and accelerator support is said to continue evolving through InfiniOps and InfiniRT, which means new hardware arrives through those projects rather than through new code here.

## Single-device execution is the boundary to plan around

The generic Infini backend currently supports single-device execution. That is stated in one line of the README and it is the limitation most likely to affect your design, because distributed example code exists under examples/distributed and a BUILD_DIST option appears in the build system, yet the general backend still runs on one device.

The practical reading is that multi-device work is either specific to a particular InfiniOps build or not yet generalised in the generic path. If your workload is sharded across several GPUs or accelerators, you need to check the InfiniOps build you intend to use rather than assume the engine will spread a graph across devices for you.

The other boundary is operator coverage. Unsupported operators fail explicitly instead of silently falling back to the CPU, and the README says model compatibility is not limited to a fixed model list but depends on ONNX importer coverage, operator semantics, data types and shapes, and the implementations available in the selected InfiniOps backend. The explicit failure is a deliberate design choice and a good one for correctness, but it converts a model you want into a coverage question you have to answer per backend.

## Building the CPU-only Python package

The lower stack sources are pinned as submodules, so the first step after cloning is to initialise them:

```bash
git submodule update --init --recursive
```

For the native CPU runtime with no accelerator kernels, the install target is a single make invocation:

```bash
make install-python INFINI=OFF PYTHON="$(command -v python3)"
```

That target builds the C++ project and installs pyinfinitensor as an editable Python package. In the Makefile it compiles with CMake into a build directory under build/Release, copies the backend shared objects and libInfiniTensor.so into the pyinfinitensor source tree, and then runs pip install -e on that directory. Because the install is editable, the Python package points back at your working copy rather than a snapshot.

The install step also runs a check script afterwards, scripts/check_install.py, which can be pointed at a specific backend with the CHECK_BACKEND variable and at provider modules with PROVIDER_MODULES. Running the check is worth doing, since it is the fastest way to find out whether the built runtime sees the backends you expected.

## Building against an accelerator means matching two prefixes

To use an accelerator you first install matching InfiniOps and InfiniRT prefixes for the target machine, then build InfiniTensor against them:

```bash
make install-python \
  INFINI=ON \
  INFINIOPS_ROOT=/path/to/infiniops-prefix \
  INFINIRT_ROOT=/path/to/infinirt-prefix \
  PYTHON="$(command -v python3)"
```

The two root paths are where the ABI question lives. The build notes state that INFINIOPS_ROOT and INFINIRT_ROOT must refer to matching lower-stack builds, and that ATen-backed InfiniOps builds must use the same Python and PyTorch C++11 ABI as InfiniTensor. A mismatch there produces link failures or subtle crashes rather than a clear error message.

Provider modules are the other moving part. If the selected InfiniOps build requires Python provider modules, you pass them with PROVIDER_MODULES during installation and with INFINIOPS_PROVIDER_MODULES at runtime. The INFINIOPS_PROVIDER_LIBRARY_DIRS setting is only needed when provider libraries sit outside the selected Python and PyTorch environment.

One more build switch matters for anything touching the protocol definitions: USE_PROTOBUF=ON requires a compatible Protobuf installation. The README also notes that BACKTRACE=OFF is the default for portable Release builds and should only be enabled when libdw-dev is available.

## The Makefile is the interface you will actually use

The root Makefile exposes a small set of targets, and they cover the whole loop. make build compiles the C++ project, make install-python builds and installs pyinfinitensor, make test-cpp runs the C++ tests, make test-onnx runs the ONNX frontend tests, make test-api runs the Python API tests, and make clean removes the generated build files.

Defaults live in the Makefile too, and they are worth reading before you change anything. TYPE defaults to Release, JOBS to 8, ATEN to ON, INFINI to AUTO, TEST to ON, BACKTRACE to OFF, DIST to OFF and NNET to OFF. Setting TEST=OFF skips the test targets during compilation, which is what you want in a slim build image. The ATen flag controls whether the ATen-backed InfiniOps kernels are compiled in, and the NNET flag turns on the NNET build, which corresponds to the EinNet branch listed in the roadmap.

Docker support is built into the same file, with DOCKER_NAME, DOCKER_IMAGE_NAME, DOCKER_FILE and DOCKER_RUN_OPTION variables and a default of infinitensor_ubuntu_22.04.dockerfile, so a containerised build is a make variable away rather than a separate script.

Contributing is also documented in the README: run the tests and formatting, use test/script/clang_format_inplace.sh for C++ formatting, include the ctest result in the pull request when C++ code changes, keep the pull request title concise because squash merges use it as the commit message, and get at least one reviewer approval.

## No hardware matrix, no versioned releases, and an Apache licence

Two maintenance facts stand out. First, InfiniTensor does not maintain a separate static hardware matrix, and the project tells you to check the exact InfiniOps and InfiniRT build, vendor SDK, driver and device runtime used on the target machine. Compatibility is therefore a property of the stack you assembled rather than a claim the project publishes, which is honest but means you own the verification.

Second, the release history is thin. The only tagged release is called test-models and dates from 2023-12-11, while the last push to the repository was on 2026-10-18. Work is happening, but there is no versioned distribution stream to pin against, so you build from a commit and keep your own record of it.

The licence is Apache-2.0, which is permissive and includes an explicit patent grant, so embedding InfiniTensor in a commercial product is a low-friction decision from the licence side. The documentation links are in Chinese, and the repository also carries README_CN.md, a Chinese introduction, a Chinese documentation index and a Chinese installation guide, with the user guide and installation guide both pointing at Chinese pages. English readers get the README and the source, while the deeper guides are Chinese only.

If you use the research results, the project asks you to cite EinNet and PET, and the README gives the two BibTeX entries for the OSDI 23 and OSDI 21 papers.

## Conclusion

InfiniTensor fits a research or platform team that already has InfiniOps and InfiniRT builds for its target machines and needs a graph-level executor with an ONNX frontend on top. It does not fit a team that wants a single installable package covering hardware support, since device SDKs and operators live outside this repository, nor a multi-device deployment, since the generic backend supports single-device execution only. Before you commit, verify that the InfiniOps and InfiniRT prefixes you point at match the driver and runtime on the machine, confirm the Python and PyTorch C++11 ABI line up with any ATen-backed build, and read the operator coverage of your chosen backend, because unsupported operators fail rather than fall back to the CPU.

## FAQ

### How do I install InfiniTensor with only the CPU runtime?

Initialise the pinned lower stack sources with git submodule update --init --recursive, then run make install-python INFINI=OFF PYTHON="$(command -v python3)". The target builds the C++ project and installs pyinfinitensor as an editable package.

### How do I build InfiniTensor for a GPU or accelerator?

Install matching InfiniOps and InfiniRT prefixes for the target machine first, then build with make install-python INFINI=ON, passing INFINIOPS_ROOT and INFINIRT_ROOT. The available devices and operators come from those installed builds.

### Does InfiniTensor fall back to the CPU when an operator is missing?

No. The documentation states that unsupported operators fail explicitly instead of silently falling back to the CPU, so compatibility depends on the ONNX importer coverage and the operator implementations in the selected InfiniOps backend.

### What happens if my InfiniOps build does not match InfiniTensor?

INFINIOPS_ROOT and INFINIRT_ROOT must refer to matching lower-stack builds, and ATen-backed InfiniOps builds must use the same Python and PyTorch C++11 ABI as InfiniTensor. Provider modules are passed with PROVIDER_MODULES at install time and INFINIOPS_PROVIDER_MODULES at runtime.

### Can InfiniTensor run a model across multiple devices?

The generic Infini backend currently supports single-device execution. Multi-device work is handled through the particular InfiniOps build you select rather than through a generic path in InfiniTensor.

## Sources

- [InfiniTensor/InfiniTensor on GitHub](https://github.com/InfiniTensor/InfiniTensor)
- [Issues](https://github.com/InfiniTensor/InfiniTensor/issues)
- [License: Apache-2.0](https://github.com/InfiniTensor/InfiniTensor/blob/master/LICENSE)
- [README](https://github.com/InfiniTensor/InfiniTensor/blob/master/README.md)
- [Releases](https://github.com/InfiniTensor/InfiniTensor/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/infinitensor-infinitensor