depth-detect: a bicycle rear-view mirror that fuses YOLO with depth

The problem is a cyclist, not a warehouse

The repository describes itself as a bicycle smart rear-view mirror system. That framing is worth holding on to, because every design decision follows from it.

The core pipeline fuses YOLO object detection with monocular or stereo depth estimation, using Depth-Anything, Lite-Mono or YOLO-Depth, and then applies what the project calls an approach judgement on top of the detections: the depth is dropping, the target's bounding box is getting larger, and the trend is sustained, so the object is treated as a dangerous target and is highlighted and reported.

Three conditions, not one. That is the part of the design that is worth stealing even if you never build the rest. A single depth reading tells you an object is close. A single bounding box size tells you an object is large in frame, which happens when something passes near the camera. Requiring depth to fall and box height to grow at the same time, with the trend sustained rather than momentary, is what separates a vehicle actually coming up behind you from a pedestrian walking past on the pavement.

The second model is Yolo-Depth, which fuses the two tasks rather than running them side by side, and the deployment target is equally specific: x86 and aarch64, with Jetson TX2 and Jetson Nano named explicitly, and TensorRT on Jetson and GeForce.

So this is a system for one hardware shape and one mounting position, built by someone who wanted a mirror that tells them when a car is closing.

Three scheduling modes for two models that want different clocks

Running a detector and a depth model on the same frames wastes hardware if both run every frame. The pipeline therefore has three scheduling modes, and the difference between them is the order in which the inference stage is submitted.

Serial is the default, where detection and depth are called one after the other through the synchronous interface. Overlap is enabled with prefer.overlap=true and is used when stagger is off: both models are submitted with runInferenceAsync and the results are collected back on a CUDA stream, so the two pieces of GPU work run concurrently. Stagger is enabled with prefer.stagger_infer=true and schedules detection and depth separately according to detect_interval and depth_interval, with the frame number taken modulo each interval. When a stagger frame happens to hit both intervals, the frame is forwarded to the overlap path instead.

The detail in that last case is the interesting one. Under the yolo_depth depth model, the collision frame switches to a lightweight detection model rather than running the full one. So the system degrades the expensive model on exactly the frames where two models would otherwise contend, instead of dropping the frame or stalling.

The backend has its own limitation stated plainly: the ONNX backend has no asynchronous capability, so the overlap path automatically degrades to synchronous when it is in use. That is the trade for running on CPU, and it is the kind of thing that is easy to miss and then expensive to diagnose.

All three paths hang off one entry point, process(), which dispatches on the schedule mode. Everything above the inference stage is identical between them, which is why the model layer and the scheduling logic are unaware of the backend type.

Backend selection is a fallback chain, and the qnn option is a placeholder

The backend is chosen by prefer.backend, which defaults to auto, and the default is a historical fallback chain rather than a clever selection: if use_gpu is true and a GPU is available and an engine path is configured, TensorRT is used, otherwise ONNX Runtime.

The explicit values are stricter. tensorrt uses only TensorRT and errors out directly if loading fails, with no silent fallback. onnxruntime uses only ONNX Runtime on the CPU. qnn is reserved, intended for the Qualcomm HTP and NPU, and the documentation says the code entry point is ready but the implementation is switched on once it lands.

That qnn entry is worth flagging as a signal. It tells you the author has the interface boundary in the right place, since adding a third backend does not disturb the pipeline, and it tells you where the project is going next. It also means a reader should not expect a QNN path to work today.

What makes the backend abstraction work is that the model layer reports where its output tensors live, as TensorLocation::Host or CudaDevice, and the pre and post processing paths are chosen from that. So the same code picks an OpenCV CPU path or a CUDA kernel path depending on what the backend returned, rather than on a compile-time flag.

The parallel processing itself lives in op_kernel as CUDA .cu files for the pre and post processing, which is where the acceleration claim in the feature list comes from.

Building and running it, and where the models come from

The quick start is short. You clone the repository and change into the directory, and the same steps work on Linux and Jetson:

git clone https://github.com/yzfzzz/depth-detect.git
cd depth-detect

Then one script starts it:

./start.sh

The Docker route is the recommended one, and the build script is the entry point:

./build.sh
./start.sh

The container image is built from a TensorRT base, and the base image is an argument rather than a hardcoded line, because the build script detects the local CUDA version and passes it in:

ARG BASE_IMAGE=nvcr.io/nvidia/tensorrt:25.04-py3
FROM ${BASE_IMAGE}

The image installs the system libraries the build needs, including libyaml-cpp-dev, libeigen3-dev, libopencv-dev, pybind11-dev, clangd, clang-format, build-essential and libbenchmark-dev, and it installs the Python packages ultralytics, onnxsim, timm and carla. The timezone is set to Asia/Shanghai, which tells you where the project was developed and that timestamps in the logs follow that.

The models are the part that is not in the tree. The model/ directory is a git submodule that holds the model repository, with onnx and engine subdirectories, so a clone of this repository alone does not give you the weights. That is the first thing that will stop a new user, and it is why the documentation lists the model repository as a directory in the structure rather than as a folder of files.

One more piece of history is worth knowing. The readme warns that for the older code shown in the Bilibili video, you should check out the release/v1.0 tag, and it gives the two commands that do it, with the note that switching puts you in a detached HEAD state.

What comes out of a run, and how the approach judgement is recorded

The run artefacts are listed precisely, and each one is tied to a configuration key rather than to a fixed behaviour.

Output results land in bin/out_dir/, and whether you get video, images, both or none is decided by io_manager.save_mode. Run logs go to bin/logs/, controlled by logger.save_file. The per-track log is bin/track_log.csv, controlled by io_manager.save_track_log, and it records the class, the raw depth, the box area and the frame count for each track. Finally, the terminal prints a summary of the average, P95 and P99 timing for each stage when the program exits.

That CSV is the interesting artefact, because it is the evidence the approach judgement is built on. A row per track with the raw depth, the bounding box area and the number of frames is exactly what you need to check afterwards whether a highlight was correct, and it means the system is auditable after the fact rather than only observable in the moment.

The tracking itself is BYTETracker, with a Kalman filter from OpenCV and a LAPJV assignment step, and the thresholds are all configuration driven rather than compiled in. The motion state engine then turns the tracked objects into motion records, which is where the approach judgement is applied, and the danger alert handler assembles the report.

The report itself is JSON over TCP, built with nlohmann/json, and the sending side lives in a third_party submodule called JsonSenderTest. The disk write path has a bounded buffer: when the disk cannot keep up, new frames are dropped and counted for a warning, and the buffer is flushed on exit. For a rear-view camera running at a fixed frame rate, dropping the newest frame rather than blocking the pipeline is the right failure mode, and having the drop count visible is what makes it diagnosable.

The source tree separates the pipeline, the kernels and the tools

The directory layout in the project documentation is worth reading, because it shows where the boundaries are.

Under cpp/ there is main.cpp, the entry point that reads a frame, runs inference, handles the alert, draws and writes to disk or the display. Beside it, include/ holds frame.h, memory.h with the CUDA and TensorRT smart pointers, and public.h with the CHECK_CUDA macro. The core/ directory carries the Pipeline, the IOManager, the MotionStateEngine and the DangerAlertHandler, which is the four components that define the behaviour.

The inference/ directory is split in two: infer_backend/ holds the TensorRT and ONNX Runtime abstraction and its factory, and infer_models/ holds BaseModel, YoloDetectModel, DepthModel and YoloDepthModel. op_kernel/ holds the CUDA pre and post processing kernels as .cu files. bytetrack/ holds the tracker, the Kalman filter and the LAPJV assignment. tools/ holds the VisualManager, the ControlPanel and a ScopedTimer, and utils/ holds the ConfigManager and the LoggerManager.

Alongside that, benchmark/ holds the Google Benchmark targets and the regression test targets, scripts/ holds config_tool.py, which is described as a gen, read and refresh tool in one, and it is the companion to start.sh. third_party/ carries spdlog, onnxruntime, the JsonSenderTest submodule and OneEuroFilter. There is also a stereo_shot.py at the root, which is the natural companion to the stereo depth mode.

The stack underneath is C++14 with CMake 3.10, TensorRT with support for both 8.x and 10.x, ONNX Runtime 1.10 as the CPU fallback, CUDA for the parallel work, OpenCV 4.x for the vision code, yaml-cpp and spdlog for configuration and logging, Eigen3 for the numerics, and nlohmann/json over TCP for the reporting.

One configuration detail deserves attention: config_tool.py does generation, reading and refreshing in one tool, so the configuration file is treated as something with a toolchain around it rather than as a hand-edited file. The run artefacts are also configuration driven, with io_manager.save_mode, logger.save_file and io_manager.save_track_log deciding what is written.

Where depth-detect is the wrong tool

The first limit is the hardware. The deployment path is TensorRT on x86 or Jetson, and the container base is a TensorRT image. The ONNX Runtime path exists as a CPU fallback and the documentation is explicit that it degrades the overlap scheduling to synchronous because that backend has no async support. So a team without an NVIDIA device is not running this project in the configuration it was designed for, and a team with a modern GPU but no TensorRT installed is building its own environment.

The second is that this is a research prototype with a narrow purpose, not a detection library. The frame loop, the tracker, the motion state engine and the alert handler are all shaped around a rear-facing camera on a bicycle, and the approach judgement in particular encodes assumptions about a camera position and a road scene that will not transfer to a warehouse or a traffic camera. The MIT licence makes reuse easy, but the reuse will be a rewrite of the judgement logic, not a configuration change.

The third is the model situation. The weights live in a separate model repository that is a git submodule, so the repository alone is not runnable, and the model/ directory carries both onnx and engine directories, which suggests the TensorRT engines are checked in alongside the ONNX graphs. Engines are built for a specific TensorRT version and a specific GPU architecture, so an engine that loads on the author's machine may not load on yours, and the documentation's answer to a load failure in tensorrt mode is that it errors out rather than falling back.

The fourth is the release discipline. There are no GitHub releases at all, so there is no version to pin and no changelog to read, and the only marker of age is the last push, which was on 2026-10-15. The older code that appeared in the Bilibili video is still reachable through the release/v1.0 tag, and the readme asks you to fetch the tags and check that tag out, which leaves you in a detached HEAD state. For a project with a C++ core, a TensorRT integration and a submodule full of model engines, that means the only sensible way to depend on it is to vendor the tree yourself.

The fifth is the benchmark story. The repository has a benchmark directory with Google Benchmark targets and regression tests, and the run prints per-stage average, P95 and P99 timings. The project states the benchmark methodology but the repository does not carry the published numbers, so the performance claims in the readme are not verifiable from what is here.

Editorial conclusion

depth-detect is worth reading if you are building an on-device proximity or approach warning system, because the approach judgement is the part of the project that carries the idea and it is implemented with three separate signals rather than a single threshold, and because the pipeline is written so the model layer and the scheduler are indifferent to which inference backend runs them. It is the wrong starting point if you want a general object detection library, since the frame loop, the tracker and the alert path are all shaped around a rear-view camera. Before you try to build it, check the three things that decide whether it will run for you: that you have an NVIDIA TensorRT environment, because the ONNX Runtime path is documented as a CPU fallback and the base Docker image is a TensorRT one; that you have the model repository, which is a git submodule carrying the onnx and engine directories rather than files in the tree; and that the configuration file is generated rather than hand-written, since scripts/config_tool.py handles generation, reading and refreshing in one tool and the start.sh script expects the result.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. yzfzzz/depth-detect on GitHub