tensorrt_cpp_api: A No-Throw C++ Wrapper Around TensorRT Engine Building

The Problem: TensorRT's C++ API Is Verbose and Easy to Get Wrong

Raw TensorRT C++ code asks you to manage a builder, a network definition, a parser, an optimization profile, a serialized engine blob, a runtime, an execution context, and every device buffer by hand. Each of those is a place to leak memory, forget a synchronize, or reuse a serialized engine that no longer matches the model on disk. The README frames this library as replacing that with a small surface: name-keyed tensors at the boundary, caller-owned CUDA streams, explicit host and device transfers, and a Status/Result error model instead of exceptions. The target audience is narrow and stated: Linux, CUDA 12, TensorRT 10 or newer, C++20, and CNN-style vision models. If you are writing a Windows service or serving a decoder-only language model, the Scope section says those are out of scope, and the rest of the README does not contradict that.

Engine Cache Keyed by ONNX Hash, Options, TensorRT Version and GPU UUID

The feature the README spends the most words on is the cache. Building an engine is the slow part of a TensorRT workflow, so most projects serialize the result and reload it. The failure mode there is a stale blob: you change the model, or the precision, or upgrade the driver, and the old engine still loads and still produces output, just the wrong output. This library keys the cache on the ONNX content hash plus the build options plus the TensorRT version plus the GPU UUID, writes a JSON sidecar, and uses atomic writes. The README states that a stale cache is detected and rebuilt rather than silently misused. That is a design claim about invalidation, not a benchmark, and it is the kind of thing worth reading the source for if you plan to ship it. The atomic write matters for a different reason: two processes starting at once on the same cache directory. The README does not describe the concurrency behaviour of the cache directory itself, only that writes are atomic.

Dynamic Shapes, Execution Context Leasing, and Where the Stream Lives

Dynamic shapes in TensorRT need per-input min/opt/max optimization profiles, and the README says this library builds them per input and supports a Shape type that understands -1. It also states that one optimization profile is required per execution context for concurrent dynamic-shape inference. That constraint is not a library choice; it is how TensorRT works, and the library surfaces it through EnginePool, which leases execution contexts for multi-stream inference. Every call runs on a caller-provided Stream, so the library does not own your scheduling. The README describes the engine as thread-compatible and the pool as thread-safe, which is a meaningful distinction: you can share the engine across threads, but you coordinate the contexts. The inferSingle call takes a map of input name to tensor view and returns a Result. The example then calls toHost(stream), which the README annotates as explicit device-to-host plus sync, never implicit. That is a deliberate trade against convenience: you always know when a copy and a synchronization happen.

Building It: CMake Options, the Install Prefix, and the Downstream find_package

TensorRT and CUDA are treated as externally provided, which the README states plainly. The build sequence given is cmake -S . -B build -DTRT_CPP_API_BUILD_PREPROC=ON, then cmake --build build -j$(nproc), then cmake --install build --prefix /opt/trtcpp. If TensorRT came from a tarball rather than a package manager, the README says to add -DTensorRT_DIR=<root>. Downstream, the installed package is consumed with find_package(tensorrt_cpp_api REQUIRED) and linked against tensorrt_cpp_api::tensorrt_cpp_api and tensorrt_cpp_api::preproc. The preproc target is separate, which matches the README's description of fused preprocessing as optional. The Python bindings install with pip install . and are built through scikit-build-core into a wheel named trtcpp. The README points at docs/install.md for the apt versus tarball distinction and the full build option list, and at docs/quickstart.md for core concepts. There is also a docs/upgrading_from_v6.md, which tells you the API changed enough at some point to warrant a migration guide. No release notes were retrieved for this repository, so the version history is not something this review can describe.

Preprocessing and the Python Bindings Are Optional, and That Is the Point

Two features sit behind build flags or separate targets. The first is tensorrt_cpp_api::preproc, described as one CUDA kernel that performs letterbox-resize, BGR to RGB conversion, per-channel normalization, HWC to NCHW layout change, and a cast, with no intermediate buffers. If your pipeline already does preprocessing elsewhere, or you feed preprocessed tensors, you do not need this target. The second is the Python bindings, which accept CuPy, PyTorch, or Numba GPU arrays through __cuda_array_interface__ or DLPack and return them the same way, with the GIL released during inference. The README claims the Python path runs within about 13% of C++ and points at examples/python/benchmark_parity.py for that measurement. Treat that as a claim from the project, not an independent result. The zero-copy framing is the interesting part: if your Python pipeline already holds tensors on the GPU, the binding avoids a host round-trip, which is usually the cost that makes Python inference wrappers slow.

The Latency Table, and What It Does and Does Not Tell You

The README includes a table measured on an RTX 3080 Laptop GPU with TensorRT 10, using a preallocated zero-copy enqueue loop from examples/benchmark. It lists YOLOv8n at 1.07 ms and 937 inferences per second in FP16, YOLOv8n at 2.00 ms and 499 per second in FP32, and MobileNetV2 at 0.31 ms and 3199 per second in FP16. The README's own interpretation is that inference time is TensorRT-bound, being the enqueueV3 cost of the engine, so the wrapper adds no measurable inference overhead. That is the right way to read the numbers: they are evidence about the hardware and the engines, not about the wrapper's efficiency. The wrapper's work is everything around the call, which the README describes as zero-copy name-keyed IO with no per-call allocations or nested-vector copies, a stream-ordered allocator, and the no-throw API. None of those show up as a latency delta in a table like this. What the table does usefully show is the FP16 to FP32 gap on the same model and GPU, roughly 1.9x here, which is a reasonable sanity check on whether your own precision setting is taking effect.

Limitations: Platform Scope, CNN Assumption, and a Thin Release Trail

The scope statement is the first limitation and it is self-declared: Linux only, CUDA 12, TensorRT 10 or newer, CNN-style vision models, with Windows and LLM or transformer-specific features explicitly out of scope. If you need any of those, this is the wrong tool and no amount of wrapper quality changes that. The second limitation is the build dependency model. TensorRT and CUDA are not vendored or fetched; you supply them, and the README distinguishes apt installs from tarball installs with a separate -DTensorRT_DIR flag, which means the install path is the part most likely to go wrong on an unfamiliar machine. The third is the absence of retrieved releases. There is an upgrading_from_v6.md document, so the API has had at least one breaking transition, but with no release list available, there is no way from the supplied material to say how frequently the API moves or what a version pin should look like. The fourth is the precision promise: the README says precision is version-aware and errors clearly when a mode is not achievable rather than silently degrading. That is a good property, but it also means an unsupported precision on your TensorRT build is a runtime failure you have to handle, not a fallback.

What It Replaces, and Where the Difference Actually Lies

The obvious alternative is writing against the TensorRT C++ API directly. The difference is not speed; the README's own position is that inference is TensorRT-bound, so a wrapper cannot make enqueueV3 faster. The difference is in three places. First, error handling: direct TensorRT code returns null pointers and status codes that are easy to ignore, while this library returns Status or Result and the README's example checks the Result before touching the value. Second, header hygiene: the README states that no nvinfer1, OpenCV, or spdlog types appear in any public header, achieved with PImpl and a version-gated generated build_config.h, so consumers need TensorRT at runtime rather than compile time. That is a real build-system difference if you have ever tried to keep a TensorRT include path out of a large codebase. Third, the cache: a hand-rolled TensorRT integration usually serializes an engine and reloads it, and the invalidation logic is whatever the author remembered to write. Here it is ONNX hash plus options plus TensorRT version plus GPU UUID. A second alternative is the ONNX Runtime TensorRT execution provider, which builds engines internally and manages the cache for you, at the cost of giving up direct control over optimization profiles, execution context pooling, and stream ownership. If you need per-context profiles for concurrent dynamic-shape inference, that control is the reason to be here.

Editorial conclusion

Adopt it if you ship CNN vision inference on Linux with CUDA 12 and TensorRT 10 or newer, and you want engine caching, dynamic shape profiles, and no TensorRT headers in your public API. Do not adopt it for Windows, for LLM or transformer-specific serving, or if you need a supported release with a changelog; the repository shows no retrieved releases. Before committing, verify that your TensorRT install is the apt or tarball layout that docs/install.md covers, run examples/download_models.sh plus the benchmark example on your target GPU, and confirm the engine cache invalidation behaves as documented when you change build options.

Official sources

1. cyrusbehr/tensorrt-cpp-api on GitHub
2. Issues
3. License: MIT
4. README