YOLOv8-TensorRT Setup, Engines, and Inference

Why YOLOv8-TensorRT Exists

YOLOv8-TensorRT addresses the deployment step between a trained Ultralytics YOLOv8 model and a TensorRT inference process. The repository accepts trained weights, exports them to ONNX, builds a TensorRT engine, and supplies Python and C++ paths for consuming that engine. It is aimed at engineers who already have YOLOv8 weights and an NVIDIA software stack but need task-specific inference wrappers rather than only an exported model file.

Five YOLOv8 tasks are covered: detection through `det`, segmentation through `seg`, pose through `pose`, oriented bounding boxes through `obb`, and classification through `cls`. The Python side consolidates these under `infer.py`, while the C++ repository contains separate executables for corresponding operations. Python and C++ can share engines and class files, according to the README, but that does not mean every engine is valid for every consumer.

The main deployment constraint is TensorRT engine specialization. `export-det.py` creates an End2End detection engine with NMS included. Its documented outputs are `num_dets`, `bboxes`, `scores`, and `labels`. `export-seg.py` creates a segmentation engine with `outputs` and `proto` outputs. A native Ultralytics ONNX export keeps raw model output, with `[1, 84, anchors]` given as an example. These layouts have different assumptions and must be routed to compatible handlers. For example, `infer.py --task det` expects the End2End layout, while `yolov8_detect`, pose, OBB, and classification paths consume raw exports.

Python, C++, and TensorRT Architecture

The repository separates shared C++ mechanisms from thin task applications. Under `csrc/core`, `libyolov8_core` owns TensorRT and CUDA resources through RAII wrappers, engine handling, pre-processing, post-processing, profiling, and a compatibility layer named `trt_compat`. The README states that `trt_compat` is the only location that branches on TensorRT version. Task binaries under `csrc/apps` sit above this core instead of duplicating engine-management logic.

The build system detects TensorRT versions across the documented 8, 10, and 11 families and detects OpenCV versions. The README reports compatibility with TensorRT 8.6, 10.8, 10.16, and 11.0, plus OpenCV 4.5, 4.6, and 4.11. That is a documented compatibility set, not a promise for arbitrary versions outside it. For OpenCV 4.7 and newer, the build selects class-aware NMS.

Python has a parallel organization under `models/`, with engine-building code, backend implementations, label handling, compatibility code, and task-specific handlers. `infer.py` offers three documented backends: `torch`, `cudart`, and `pycuda`. The `cudart` backend uses `cuda-python`, while `pycuda` uses the `pycuda` package. TensorRT 10 support is documented for the `cudart` and `pycuda` paths.

Batched inference submits one batch per engine call and decodes results per image. A fixed-batch engine remains tied to its exported batch size, while a dynamic-batch engine follows `--batch N`. A single image or batch-one engine retains the same behavior. Optional components include C++ tests under `csrc/tests`, Python tests, `benchmark.py`, and a DeepStream bounding-box parser under `csrc/deepstream`.

Installing CUDA, TensorRT, and Python Dependencies

The project does not install CUDA or TensorRT for the user. The README requires CUDA 11.4 or newer and TensorRT 8.4 or newer to be installed system-wide before Python dependencies are added. It also directs the user to confirm that both `nvidia-smi` and `trtexec --version` work before continuing.

The following commands verify that the NVIDIA driver and TensorRT command-line tools are available in the environment. Both should complete successfully before installing the Python packages.

nvidia-smi
trtexec --version

After confirming those tools, install the repository requirements and the additional Python packages used by export or optional inference backends.

pip install -r requirements.txt
pip install ultralytics
pip install cuda-python
pip install pycuda

`ultralytics` is required for ONNX export. `cuda-python` is needed for `infer.py --backend cudart`, and `pycuda` is needed for `infer.py --backend pycuda`. The repository's `requirements.txt` lists those last two packages as comments, so installing only `-r requirements.txt` does not install either optional backend. It pins `numpy<=1.23.5` and includes `onnx`, `onnxsim`, `opencv-python`, `torch`, `torchvision`, and `ultralytics`.

Exporting Detection ONNX and Building an Engine

Start with detection because it exposes the distinction between End2End and raw output. The repository provides `export-det.py` with exported weights, ONNX simplification, input dimensions, thresholds, a top-k detection count, and CUDA device selection.

This command exports `yolov8s.pt` as a 640 by 640 End2End detection model with built-in NMS. The README states that Ultralytics can download pretrained weights such as `yolov8s.pt` on first use.

python export-det.py --weights yolov8s.pt --sim --input-shape 1 3 640 640 \
    --iou-thres 0.65 --conf-thres 0.25 --topk 100 --device cuda:0

Build the corresponding TensorRT engine in FP16 with the repository helper. The resulting file is `yolov8s.engine`.

python build.py --weights yolov8s.onnx --fp16 --device cuda:0

The README also permits direct construction through `trtexec`. This is an alternative to `build.py`, not an additional processing stage.

/path/to/tensorrt/bin/trtexec --onnx=yolov8s.onnx --saveEngine=yolov8s.engine --fp16

Run inference against the `data` directory and save results under `output`. This uses the End2End detection engine required by `infer.py --task det`.

python infer.py --task det  --backend torch  --engine yolov8s.engine     --imgs data --out-dir output

The output location is controlled by `--out-dir`. The README also documents `--show` for display, `--conf-thres` and `--iou-thres` for relevant task paths, `--device` for the Torch backend, and `--batch` for dynamic-batch engines.

Segmentation, Pose, and Raw Exports

Detection is not the only route. For segmentation, `export-seg.py` produces a dedicated TensorRT engine. The resulting outputs are `outputs` and `proto`, and the matching Python invocation uses `--task seg`. The `cuda-python` dependency is installed separately because this example selects the `cudart` backend.

This command builds the exported segmentation engine by passing the documented thresholds and backend selection to `infer.py`.

python infer.py --task seg  --backend cudart --engine yolov8s-seg.engine  --imgs data --conf-thres 0.25 --iou-thres 0.65

Pose, oriented boxes, and classification use raw Ultralytics exports rather than the two repository-specific export scripts. The README gives the following Ultralytics command for pose:

yolo export model=yolov8s-pose.pt format=onnx opset=11 simplify

After building the matching TensorRT engine, inference can select the `pycuda` backend and request a display window:

python infer.py --task pose --backend pycuda --engine yolov8s-pose.engine --imgs data --show

This route is a poor fit if the available environment cannot install `pycuda`. In that case, the documentation lists the `torch` and `cudart` alternatives, but it does not claim that switching backends changes the engine layout. The export still has to match the pose handler.

Building and Running the C++ Detection Path

The C++ path can use the same engines and class files as Python. Configure the repository with the location of a TensorRT installation, then compile the applications and tests.

The following command creates the CMake build directory and configures TensorRT discovery.

cmake -S . -B build -DTensorRT_ROOT=/path/to/TensorRT

Compile the configured project. The build can use C++17 `std::filesystem`, while C++14 falls back to vendored `ghc::filesystem` according to the README.

cmake --build build -j

Before launching a dynamically linked binary, place the TensorRT libraries on `LD_LIBRARY_PATH`.

export LD_LIBRARY_PATH=/path/to/TensorRT/lib:$LD_LIBRARY_PATH

This documented detection command reads `yolov8s.engine`, processes `data/bus.jpg`, and writes results to `output`.

./build/bin/yolov8_detect yolov8s.engine data/bus.jpg --out-dir output

The C++ interface also documents `--show`, `--profile`, and `--labels`. The profiler can produce a per-layer report, but the README gives no example of its output. Executable selection matters: `yolov8_detect` consumes the raw Ultralytics layout, while `yolov8_detect_e2e` consumes the End2End layout produced by `export-det.py`. An engine from the earlier detection export therefore belongs with the latter binary unless it is rebuilt for raw-output use.

Engine Mismatches, Upgrades, and License Boundaries

The clearest failure mode is an engine-layout mismatch. Passing an End2End engine to a raw-output consumer, or using `infer.py --task det` with a native Ultralytics export, violates the routing documented by the README. The task association must remain intact through export, engine construction, and inference. Segmentation has its own `outputs, proto` layout, while pose, OBB, and classification use raw Ultralytics exports.

The dependency floor is another boundary. CUDA must already be available at version 11.4 or newer, TensorRT at version 8.4 or newer, and the documented compatibility checks are narrower than those minimum versions. OpenCV selection can also change post-processing because version 4.7 and newer selects class-aware NMS. TensorRT engines are built artifacts, but the README does not provide a broader portability matrix beyond its listed TensorRT and OpenCV versions.

Maintenance spans several compatibility layers. The repository contains CUDA compatibility code, Python backend compatibility code, and the C++ `trt_compat` layer. Its last push was on 2026-08-14, and the repository has no GitHub releases. The README therefore defines no tagged release sequence for upgrades. A TensorRT upgrade requires checking the build's detected version and the Python backend compatibility code, rather than assuming an existing engine can be reused unchanged.

The repository is licensed under MIT. That is the project license, not a statement about every bundled or separately installed component. The README also mentions model weights, Ultralytics, CUDA, TensorRT, OpenCV, and label sets such as COCO, ImageNet, and DOTA. It does not document their individual licensing or redistribution terms, so those must be checked separately. This is a compliance boundary, not legal advice.

Editorial conclusion

Adopt YOLOv8-TensorRT when trained Ultralytics YOLOv8 weights need TensorRT execution from either Python or C++ and the required engine layout is understood. Do not choose it when CUDA and TensorRT are unavailable, or when one engine must be assumed to work with every task. Verify first that the ONNX export matches the selected `infer.py` task or C++ binary, because engine-layout compatibility is an explicit boundary in this repository.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. triple-mu/YOLOv8-TensorRT on GitHub