What is ONNX?

What a model file actually contains

An ONNX file is a graph. It holds a set of nodes, each one naming an operator and its inputs and outputs, plus a description of every tensor in between: its element type and its shape. The weights live in the file too, as initializers, separate from the graph structure. At the top sits metadata: the producer, the IR version, and the version of the operator set the graph is written against.

That last part is the mechanism that makes the whole idea work. Operators are not free-form: the standard defines a versioned set of them, with defined shapes and semantics, and a graph is only valid against a particular operator set version. A runtime reads that version and refuses, or warns, when it meets an operator it does not implement. The format is therefore a contract with a moving surface, which is the main thing to understand about it.

Because the file is data and not code, exporting is a translation problem. A framework's modules become nodes, its parameters become initializers, and control flow and shape inference have to be resolved into something the format can express. Our analysis of onnx/onnx describes the project as an open standard defining a common format for representing trained models so a model trained in one framework can be exported and run in a different runtime, with a Python package that provides tools for creating, validating and inspecting ONNX graphs. onnx.ai is the canonical reference for the specification itself.

What the format deliberately does not carry is a training loop, an optimiser, a data pipeline or an execution schedule. It is a description of a computation, not a program that trains one, and the runtime decides how to execute it.

The two halves: the format and the engine

Confusing these two is the most common misunderstanding about ONNX, and it decides who owns a failure when something breaks.

The format is onnx/onnx, the specification plus the tooling to build, check and inspect graphs. Producing a valid file is the exporter's job, and the export is where a surprising amount of the work happens: framework features that have no counterpart in the standard, dynamic shapes that have to be resolved, and fused kernels that were faster in the framework and mean nothing to a runtime that will re-plan them.

The engine is onnxruntime, Microsoft's open-source runtime for executing those graphs. Our analysis describes it as an engine for running and accelerating models exported from PyTorch, TensorFlow, scikit-learn, LightGBM and XGBoost across multiple hardware targets, and it notes the repository's README is intentionally minimal with the documentation living at onnxruntime.ai. The runtime maps each operator to a kernel, decides execution order, fuses what it can, and places work on CPU, GPU or other accelerators.

The other two halves are the glue. Backends translate the graph for a specific accelerator, and onnx/onnx-tensorrt is the clearest example: our analysis describes it as the C++ parser library and Python backend that turns an ONNX file into a TensorRT engine, and it is for teams already committed to NVIDIA hardware who need control over how that conversion happens. Simplifiers clean up the exported graph, and onnxsim/onnxsim is that in one file: our analysis says it runs shape inference, graph optimization and constant folding to a fixed point over an ONNX model, then optionally checks the result against the original on random inputs.

Alternative runtimes exist because the reference engine is not always the right one. sonos/tract is Sonos' self-contained inference engine in Rust, which our analysis describes as loading ONNX, NNEF and legacy TensorFlow Lite models, optimising them, and running them on CPUs, GPUs or WebAssembly. k2-fsa/sherpa-onnx is a C++ library built on next-gen Kaldi and ONNX Runtime that runs speech-to-text, text-to-speech, speaker diarization, voice activity detection and source separation entirely offline, targeting embedded hardware, mobile platforms and edge devices with bindings for twelve programming languages.

When it earns its place

ONNX is worth the conversion step in four situations, and the conversion is not free in any of them.

Deploying to a platform the training framework does not target. A browser, a phone, an embedded board or an NPU is often easier to reach through a runtime that already exists for it than through a framework build. Our analysis of sonos/tract notes it runs models on WebAssembly, and our analysis of sherpa-onnx lists Raspberry Pi, RISC-V, several NPUs and x86_64 servers among its targets.

A language your framework does not serve. deeplearning4j/deeplearning4j is the case in point: our analysis describes it as a multi-library deep learning suite for Java, Scala and Kotlin with a high-level network training API, a C++-backed linear algebra engine, an automatic differentiation framework and support for importing models from Keras and TensorFlow, and its description also lists onnx and PyTorch model import, so a JVM team can take a model from a Python training stack without retraining it.

Decoupling releases. A model file plus a runtime version means the model and the engine can be upgraded separately, which matters when the engine is where the hardware support lives.

Keeping a model in house across tool changes. If the framework you trained in is going to be replaced, an exported graph is a hedge.

It does not earn its place when you are training and serving in one framework on one platform, where the framework's own deployment path is better integrated and the conversion buys nothing. It does not earn its place during exploration, and it does not earn its place when your model uses framework features that the standard does not cover, because that is the case where the export is most likely to be the difficult part.

Where it goes wrong

Five problems account for most ONNX trouble, and none of them are mysterious once you know the format is a contract.

Operator coverage. A framework operation with no counterpart in the operator set either fails to export, exports into a more expensive equivalent, or is dropped. The graph loads, the output is wrong, and nothing complains. This is the failure that costs the most time, because the symptom appears far from the cause.

Dynamic shapes and control flow. The format describes shapes, and a model with genuinely input-dependent shapes forces the runtime to reason at run time. Some operations are implemented for one shape profile and not another, which produces a model that works on the example you tested and fails on the example you did not.

The exported graph is a mess. Framework exports routinely contain redundant operators, constant subexpressions and casts that no longer mean anything once the runtime re-plans the graph. onnxsim exists because this is the normal state of an export, not an exception.

Numerics drift. Constant folding, fusion and precision conversion all change floating point results slightly. A model compared at one decimal place will match, and a model compared against a stored golden output will not. Deciding what tolerance is acceptable is part of deploying, and onnxsim's option of checking the simplified result against the original on random inputs is the pattern to copy.

The model collection is not a source. The onnx/models repository, the Model Zoo, had its Git LFS downloads shut down on July 1, 2025, and our analysis of it describes it as preserved for historical reference only. A plan that fetches models from there is a plan that stopped working, and it is worth knowing before a build depends on it.

How the projects divide

The repositories here sit in four groups, and the group tells you which question each one is answering.

The standard and the reference engine. onnx/onnx is the format and its tooling; microsoft/onnxruntime is the engine that most pipelines end up using.

The conversion layer. onnx/onnx-tensorrt targets one vendor's accelerator, and onnxsim/onnxsim repairs and shrinks the graph before it gets there.

The runtimes. microsoft/onnxruntime and onnx/onnx-tensorrt aside, there are engines written for other constraints: sonos/tract for a small self-contained Rust binary that can run on a CPU, a GPU or WebAssembly, k2-fsa/sherpa-onnx for offline speech work across embedded and mobile hardware, and deeplearning4j/deeplearning4j for teams that live in the JVM and want a training API as well as an import path.

The deployments. RapidAI/RapidOCR is the clearest case of ONNX used as a delivery format rather than as an interchange goal: our analysis says it takes PaddleOCR's detection and recognition models, converts them to ONNX, and ships runtimes for Python, C++, Java, C# and more, and it is explicit that this is a deployment project rather than a training project. That distinction decides whether it fits your pipeline.

The model collections, where the licence is the thing to read. PINTO0309/PINTO_model_zoo is a numbered collection of computer vision and audio models already converted between TensorFlow, PyTorch, ONNX, OpenVINO, TensorFlow Lite, EdgeTPU and CoreML. Our analysis of it notes the conversion scripts are MIT while each model folder carries its own LICENSE, and that the folder licence is where the real work of adopting it begins. A converted model inherits the licence of the model it was converted from, and the converter's licence says nothing about that.

In practice

ONNX is a contract for the artifact rather than a promise about execution, so the failure modes live at the two boundaries: an export that does not map cleanly onto the operator set, and a runtime that implements a subset of what the file asks for. Practically, export early, check the graph with a tool such as onnxsim/onnxsim, compare outputs against the source framework with a tolerance you chose deliberately, and pick the runtime that matches where the model will run, which is what the alternatives such as sonos/tract and k2-fsa/sherpa-onnx are for. Read the licence in the model folder rather than the repository, and do not build a dependency on a collection whose downloads have been switched off.