YOLOv8-TensorRT-CPP: A C++ Inference Wrapper Around Ultralytics Models

The gap between a trained YOLOv8 model and a C++ executable

Ultralytics ships YOLOv8 as Python. That is fine for training and fine for a demo, but a production vision service written in C++ needs to load a model, run it on the GPU and decode the output without a Python interpreter in the loop. This repository is the glue for that last step. It takes an ONNX export of a YOLOv8 model and drives it through the TensorRT C++ API, then does bounding box decoding and non-maximum suppression in C++ outside the model graph. The README states it supports object detection, semantic segmentation and body pose estimation, and that it works out of the box with Ultralytics pretrained models in all three families.

The audience is narrow and specific: engineers on Ubuntu who already have a CUDA toolchain and a reason to avoid Python at inference time. The README does not claim to be a general inference framework. It says it demonstrates how to use the TensorRT C++ API for YOLOv8, and it defers the heavy lifting to a sibling project, tensorrt-cpp-api, which the README describes as running inference behind the scenes. That framing matters when you evaluate maintenance: you are really adopting two repositories, and the one you clone is the thinner of the two.

How the ONNX graph and the C++ postprocessing split the work

The design decision the README is most explicit about is where decoding happens. When you export with the Ultralytics conversion script, the end2end flag must be disabled. The README explains why: that flag adds bounding box decoding and NMS directly to the model, whereas this implementation does those steps externally in C++. So the ONNX file you feed in produces raw predictions, and the C++ side owns the tensor parsing, the box math and the suppression step.

That split has consequences you can see in the repository's own benchmark table. On the RTX 3080 Laptop GPU listed, yolov8n-seg spends 12.353 ms in postprocess against 2.732 ms in inference. For the segmentation model the mask handling dominates the runtime, and it dominates it in C++, not in TensorRT. The same table shows yolov8n at 1.829 ms postprocess versus 1.703 ms inference, and yolov8n-pose at 0.407 ms postprocess versus 1.609 ms inference. The pose path is the cheapest to decode; the segmentation path is roughly an order of magnitude heavier than either. If you are choosing which of the three tasks to put behind this code, that asymmetry is the first thing to look at.

The benchmark numbers come from the README and were produced on the stated hardware with a 640x640 BGR image already in GPU memory at FP16. They are not a prediction of your throughput. The README also notes that the first run of any executable can take five minutes or more while TensorRT builds and caches an optimized engine file from the ONNX model, and that subsequent runs load that cached engine. That build step is a deployment artifact you have to plan for, not a one-off inconvenience.

What you need installed before the build will succeed

The prerequisite list is not decorative. Ubuntu 20.04 or 22.04, and the README states plainly that Windows is not supported at this time. CUDA 12.0 or newer recommended. cuDNN 8 or newer recommended. TensorRT 10 or newer, required, downloaded separately. OpenCV 4.8 or newer, and specifically built with CUDA support; the README points at a build_opencv.sh script in the tensorrt-cpp-api repository for compiling it from source. That is the step most people underestimate. A stock apt OpenCV will not do.

After extracting TensorRT, the README instructs you to open CMakeLists.txt and replace the TODO with the path to your TensorRT installation. There is no environment variable or find_package fallback described; you edit the file. Then the clone itself has a requirement that is easy to miss: git clone https://github.com/cyrusbehr/YOLOv8-TensorRT-CPP --recursive. The README bolds the note that the --recursive flag is necessary because the repo uses git submodules. Clone without it and the build has nothing to link against.

Model conversion happens in scripts/. Install ultralytics with pip3, then run python3 pytorch2onnx.py --pt_path <path to your pt file>. The README says the code also accepts segmentation exports such as YOLOv8x-seg and pose exports such as yolov8x-pose.onnx.

Building and running the three executables

The build sequence in the README is the conventional one: mkdir build, cd build, cmake .., make -j. That produces three executables, and each takes a --model argument pointing at your ONNX file plus an --input argument whose meaning changes per tool.

For a throughput measurement: ./benchmark --model /path/to/your/onnx/model.onnx --input /path/to/your/benchmark/image.png. The README advises ensuring the GPU is unloaded before benchmarking, and suggests the images/640_640.jpg file. To get per-stage numbers rather than a single total, recompile with cmake -DENABLE_BENCHMARKS=ON .. and rerun; that flag turns on the preprocess, inference and postprocess breakdown shown in the tables.

For a single annotated image: ./detect_object_image --model /path/to/your/onnx/model.onnx --input /path/to/your/image.jpg. For live webcam inference: ./detect_object_video --model /path/to/your/onnx/model.onnx --input 0, where 0 is the camera index. Both accept a --trt_model option if you already have a built TensorRT engine and want to skip the ONNX conversion path. Running any executable with no arguments prints the full argument list, which is the practical way to discover options the README does not enumerate.

INT8 is opt-in and needs calibration data. The README's commands are --precision INT8 --calibration-data /path/to/your/calibration/data, and it advises using 1K or more calibration images representative of real inference data. It warns that if you hit an "out of memory in function allocate" error, you must reduce Options.calibrationBatchSize so the whole batch fits in GPU memory. The README also states the tradeoff directly: INT8 can speed up inference at the cost of accuracy reduction due to reduced dynamic range. The FP32 to FP16 to INT8 row for yolov8x in the README's table goes 25.819 ms, 10.147 ms, 7.32 ms total, which is the shape of the gain you are buying.

Where this repository is the wrong choice

The Windows exclusion is not a footnote. If your deployment target is Windows, the README closes that door explicitly, and nothing in the material suggests a workaround. The OpenCV-with-CUDA requirement is the second filter: if your build environment cannot compile OpenCV from source, or your platform's package manager only offers CPU builds, you are stuck before you reach the TensorRT step.

The third limitation is maintenance. The README carries a section headed "Looking for Maintainers" asking for people to help guide the project, with a LinkedIn contact. There are no tagged releases retrieved for this repository. For an engineer deciding whether to build a product on top of it, that combination means you should expect to read the source rather than a changelog, and you should assume the tensorrt-cpp-api submodule can move independently of the code you cloned. Pin the submodule commit.

The fourth is the hardcoded TensorRT path in CMakeLists.txt. Editing a TODO in a tracked file is workable for a demo and awkward for a CI pipeline that builds on more than one machine, because every environment needs its own edit or a patch step. Nothing in the README describes an override mechanism.

Finally, the external postprocessing is a deliberate trade, not a defect, but it is a trade. Keeping NMS in C++ gives you control over the decode path and keeps the ONNX graph portable. It also means the segmentation postprocess cost sits on your CPU-side code, and the README's own yolov8n-seg row shows that cost at 12.353 ms, several times the inference time it is decoding.

How it differs from running the model through ONNX Runtime

The obvious alternative for C++ inference is ONNX Runtime with the CUDA or TensorRT execution provider. The difference in approach is where the optimization happens. ONNX Runtime takes your ONNX file and, with the TensorRT execution provider, builds TensorRT engines under the hood while keeping a general-purpose graph runtime around it. This repository instead talks to the TensorRT C++ API directly, through the tensorrt-cpp-api submodule, and expects you to handle the YOLOv8-specific decode and NMS yourself in C++.

That means this code is narrower and more explicit. You get the TensorRT engine build and cache behavior described in the README, and you get hand-written postprocessing for exactly three YOLOv8 task heads. You do not get a graph runtime that can also execute an arbitrary ONNX model. If your service runs one YOLOv8 variant and nothing else, the narrower path removes a layer. If you need to serve several unrelated models from the same process, ONNX Runtime's generality is the thing you would miss. The README does not present this as a comparison, and it does not benchmark against ONNX Runtime, so treat the choice as architectural rather than performance-driven on the evidence available.

Licence, upgrade cost and what the MIT grant does not cover

The repository is MIT licensed. That is permissive: it allows use, modification and redistribution provided the copyright notice and permission notice are retained. It is not legal advice, and two things sit outside that grant. First, the submodule tensorrt-cpp-api is a separate repository with its own licence, which you should check independently rather than assume matches. Second, the Ultralytics YOLOv8 models and the ultralytics package you install with pip3 carry their own licensing terms, and those terms apply to the weights and the export tooling regardless of what this C++ wrapper is licensed under. If you ship a product, the model licence is the one to read carefully.

Upgrade cost is dominated by the toolchain, not by the code. The README pins expectations at TensorRT 10 or newer, CUDA 12.0 or newer recommended, cuDNN 8 or newer recommended, OpenCV 4.8 or newer. Moving any of those forward means rebuilding OpenCV with CUDA support, re-running cmake with the TODO path corrected, and accepting that the first inference after an upgrade rebuilds the TensorRT engine, which the README describes as potentially five minutes or more. Because there are no tagged releases, there is no version boundary to upgrade across; you track the default branch and pin commits yourself. Budget for reading diffs in both repositories before you pull.

Editorial conclusion

Adopt this if you are deploying Ultralytics YOLOv8 detection, segmentation or pose models on Ubuntu with a recent CUDA, cuDNN and TensorRT 10 stack, and you want the postprocessing in C++ rather than baked into the ONNX graph. Do not adopt it if you need Windows, if you cannot compile OpenCV with CUDA support, or if you expect a maintained library with tagged releases. Verify first that your ONNX export has end2end disabled, that the TODO path in CMakeLists.txt points at your TensorRT 10 install, and that a cloned build with --recursive actually pulls the tensorrt-cpp-api submodule before you plan any integration work around it.

Official sources

1. cyrusbehr/YOLOv8-TensorRT-CPP on GitHub
2. Issues
3. License: MIT
4. README