Torch-TensorRT: compiling PyTorch models for NVIDIA GPUs without leaving PyTorch

What Torch-TensorRT is for, and who it is not for

The README states the goal plainly: bring TensorRT to PyTorch so that a model already written in PyTorch can run through NVIDIA's inference compiler. The problem it addresses is the gap between training-framework convenience and deployment-framework speed. TensorRT by itself expects a network description it understands; PyTorch models are Python objects with dynamic control flow. Torch-TensorRT sits in that gap and accepts three input forms named in the repository description: PyTorch, TorchScript and FX. The audience is therefore narrow and identifiable. You need an NVIDIA GPU, a PyTorch model you intend to serve rather than train, and a reason to care about inference latency. The README's headline claim is acceleration of up to 5x compared to eager execution, described as achievable in one line of code, and that claim is the project's own marketing, not an independent measurement. If your workload is training, or your deployment target is CPU or a non-NVIDIA accelerator, nothing here applies. The platform table is explicit that Linux ppc64le is not supported, and that Windows support is limited to the Dynamo path.

The two entry points: torch.compile and the export workflow

There are two distinct ways in, and choosing between them is the main architectural decision. The first is a drop-in backend for torch.compile. The README's example imports torch_tensorrt for its side effect, defines a model in eval mode on CUDA, creates a representative input tensor, then calls torch.compile(model, backend="tensorrt"). The comment in the sample notes that compilation happens on the first run and that the second call is the fast one. That is a just-in-time path: the model is compiled when it first executes, which suits interactive work and services that can absorb a warm-up. The second path is ahead-of-time export. Here you call torch_tensorrt.compile(model, ir="dynamo", inputs=inputs) with a list of representative inputs, then serialize with torch_tensorrt.save. The README shows two output formats: "trt.ep" for the PyTorch runtime and "trt.ts" via output_format="torchscript" for C++ deployment. The README is direct about the constraint behind that split: PyTorch only supports a Python runtime for an ExportedProgram, so if you need C++, you take the TorchScript file. Representative inputs are not optional decoration in either path; they are how the compiler learns shapes and dtypes.

Installation, dependencies and the version pinning problem

Stable builds come from PyPI with pip install torch-tensorrt. Nightly builds come from the PyTorch index with pip install --pre torch-tensorrt --index-url https://download.pytorch.org/whl/nightly/cu130 --extra-index-url https://pypi.org/simple. The README also points at the NVIDIA NGC PyTorch container, which it says ships all dependencies at the proper versions with example notebooks included. That container is the path of least resistance, and the dependency list explains why. The versions used to verify the test cases are Bazel 8.1.1, Libtorch 2.15.0.dev (latest nightly), CUDA 13.2 and TensorRT 11.2.1.2. The README states that Torch-TensorRT can work with other versions but that tests are not guaranteed to pass. Read that as the central operational risk: this is a compiler that binds PyTorch, CUDA and TensorRT together, and the project only vouches for one combination. The badge row at the top of the README names PyTorch 2.14, CUDA 13.2 and TensorRT 11.2.1. If your production image is on an older CUDA, the honest answer is that you are outside the verified set.

Deploying the compiled artifact in Python and in C++

The deployment side is where the export path earns its keep. In Python, the README loads the serialized module with torch.export.load("trt.ep").module() and calls it with the input list; it notes that torch_tensorrt.load("trt.ep").module() also works, and that this can run in a fresh Python session. That last detail matters because it means the compiled artifact is not tied to the process that produced it. For C++, the sample includes torch/script.h and torch_tensorrt/torch_tensorrt.h, loads "trt.ts" with torch::jit::load, and calls trt_mod.forward({input_tensor}). The practical consequence is a deployment that carries no Python dependency, which is often the real reason teams reach for this project rather than for a pure latency win. The trade-off is that the C++ route forces the TorchScript serialization format rather than the ExportedProgram one, so you give up the newer PyTorch export representation in exchange for the libtorch runtime. Neither format is described in the README as interchangeable with the other, and the sample comments treat them as parallel outputs of the same compile step.

Where the platform table and deprecation policy constrain you

The support matrix is short and worth reading before anything else. Linux AMD64 and Linux SBSA are listed as supported on GPU. Windows is supported for GPU but marked Dynamo only, which rules out the TorchScript export route there. Jetson is a different story: GPU and DLA are both listed as source compilation supported on JetPack 4.4 and later, with a note pointing to the NVIDIA L4T PyTorch NGC container for the PyTorch libraries. Source compilation is a heavier commitment than a wheel install, and the README does not describe the build steps beyond the dependency list. Linux ppc64le is not supported at all. The deprecation policy is unusually concrete for a project of this kind. Beginning with version 2.3, notices are communicated in the release notes, deprecated functions carry a statement in the source, and deprecated methods and classes emit runtime warnings when used. The migration window is six months, during which the API keeps working, after which removal follows semantic versioning. For anyone pinning a version in a long-lived service, that six-month clock is the number to plan around.

The limitation that matters: graph breaks and the wrong-model case

The obvious failure mode is a model the compiler cannot take whole. The README's resource list includes an entry titled "Tools to resolve graph breaks and boost performance" that is marked coming soon, with an empty link. That is a candid signal: graph breaks are a known problem area, and the tooling to diagnose them is not yet published. When a graph breaks, the compiler falls back rather than failing loudly, which is worse in practice because you get a partially accelerated model and no clear error. The second limitation is the version triangle. CUDA, TensorRT and libtorch move on their own schedules, and the project verifies one combination. A mismatch does not necessarily produce a compile error; it can produce a model that compiles and then behaves differently at runtime. The third case where this is the wrong tool is any deployment that needs portability across hardware vendors. Once you serialize to a TensorRT engine, you have committed to NVIDIA. If your roadmap includes running the same artifact on AMD or on CPU, the export path is a dead end and the torch.compile backend is only marginally better, since it still requires the CUDA device at compile time.

The alternative: ONNX Runtime with the TensorRT execution provider

The natural comparison is ONNX Runtime configured with its TensorRT execution provider. The difference is in what gets compiled and when. Torch-TensorRT consumes the PyTorch program directly, through torch.compile or through an export step that stays inside the PyTorch package, and it can hand back a module that reloads with torch.export.load or runs under libtorch. ONNX Runtime requires an intermediate conversion to ONNX first, which inserts a serialization format between your model and the compiler. That extra step is a cost, and it is also the benefit: an ONNX graph is a static artifact you can inspect, version and hand to a different runtime if the TensorRT provider disappoints you. Torch-TensorRT keeps you in one ecosystem and one set of APIs, which is simpler if PyTorch is already your whole stack, and more confining if it is not. Neither approach removes the dependency on a specific TensorRT version; both inherit that constraint. The choice is really about whether you want a portable intermediate representation or a shorter path from Python to a running engine.

Maintenance cost and the licence

The release cadence visible in the repository is roughly every six to eight weeks, with v2.12.1 in June, v2.13.0 in July and v2.14.0 in September. That is frequent enough that a pinned version will fall behind within a quarter, and the six-month deprecation window means an upgrade cannot be deferred indefinitely without accumulating removals. Each upgrade also reopens the version triangle, because a new release may move to a newer TensorRT or CUDA. Budget for a recompile-and-revalidate step on every bump, not just a dependency change. The licence is BSD-3-Clause, a permissive licence that generally allows commercial use and modification provided the copyright notice and disclaimer are retained. This is not legal advice; if you redistribute a compiled artifact or ship it inside a product, have your own counsel read the LICENSE file and the notices that ship with TensorRT itself, which is a separate NVIDIA product with its own terms. The repository is not archived and the last push is recent, so the project is active, but activity is not the same as stability across the version matrix.

Editorial conclusion

Adopt Torch-TensorRT if you are already deploying PyTorch on NVIDIA GPUs and can pin the exact CUDA, TensorRT and libtorch versions the release was verified against; skip it if your target is CPU, AMD, ppc64le, or a model whose control flow survives neither tracing nor the FX path. Before committing, run the two-step export flow in the README (torch_tensorrt.compile with ir="dynamo", then torch_tensorrt.save to trt.ep and trt.ts) on your own model and confirm that the serialized module reloads and produces matching outputs, because that reload step, not the compile step, is where graph breaks and version drift show up.

Official sources

1. License: BSD-3-Clause
2. Project website
3. pytorch/TensorRT on GitHub
4. README
5. Releases