depth-anything.cpp: Monocular Metric Depth and Camera Pose Without Python

What depth-anything.cpp solves, and for whom

Depth Anything 2 and 3 are PyTorch projects. Running them means a Python process, a torch install, and whatever CUDA or MKL stack your platform wants. That is fine for a notebook and awkward for anything that ships as a native binary or runs inside a service written in C++ or Go.

depth-anything.cpp is the LocalAI team's from-scratch C++17 port of the Depth Anything 3 architecture on top of ggml, with Depth Anything 2 checkpoints supported through the same engine. The README frames the goal as dependency-free monocular metric depth plus camera pose: one self-contained GGUF file, no Python, no PyTorch, no CUDA toolkit at inference, just a small native library and CLI. The audience is people embedding depth estimation into a native application, a plugin, or a server process, and the LocalAI backend is the reference consumer.

The output surface is wider than depth alone. The README lists a dense metric depth map, per-pixel confidence, camera extrinsics as a 3x4 matrix, intrinsics as a 3x3 matrix, an optional sky mask, a back-projected 3D point cloud, and export to glb, COLMAP and PLY. If you only need a grayscale depth image, most of that is overhead you will ignore.

How the ggml engine and the metadata-driven loader fit together

The architectural choice that explains most of the project's behaviour is that every dimension, hyperparameter and preprocessing constant lives inside the GGUF file. The loader reads them. Nothing is hardcoded and no external config or vocab is shipped. That is why one binary can drive the whole DA3 family, from ViT-S up to the nested giant-plus-large metric model, without a per-model code path, and why the same engine can also pick up DA2 checkpoints.

DA2 and DA3 are not the same computation, and the README is explicit about the split. DA2 is single-image depth only, with no confidence, pose or sky. Relative DA2 models emit inverse depth through a ReLU; metric DA2 models emit depth in metres through a Sigmoid scaled by max_depth, which the README gives as 20 for the Hypersim indoor checkpoints and 80 for the VKITTI outdoor ones. DA3 adds the confidence head, the pose branch and the sky mask. The engine branches on metadata rather than on a model name.

The parity story is the second structural decision. The README states that every component is gated against PyTorch-dumped reference tensors, and that end-to-end depth matches the reference net() at correlation 1.0. Bit-exact parity is claimed at every quant down to f16. That is a strong claim, and it is the reason the conversion path needs torch and opencv-python-headless: the requirements file notes that cv2 is there for bit-exact resize parity in conversion. The parity gate lives on the Python side of the fence, not at inference.

Installing depth-anything.cpp and running the CLI on a first image

There is no release and no package manager entry, so the path is to build from source. The top level of the repository holds CMakeLists.txt, src/, include/, scripts/, models/, server/ and third_party/, which is a conventional CMake layout with ggml vendored under third_party/. The README does not reproduce a build command, so the exact CMake invocation is not something this article can quote. What the README does document is the CUDA switch for the GPU path, spelled as -DDA_GGML_CUDA=ON, and it names the inference artifacts as da3-cli and libdepthanything.so.

Conversion is a separate, Python-side step. The requirements file is explicit that these dependencies are for conversion and parity checks only, and that inference needs none of them. It also points at the upstream package you need for the reference model definitions:

pip install -r requirements.txt
pip install "git+https://github.com/bytedance-seed/depth-anything-3"

The requirements file pins torch>=2.4, numpy>=1.26, gguf>=0.19.0, huggingface_hub>=0.24, safetensors>=0.4, einops>=0.8, pillow>=10, matplotlib>=3.8 and opencv-python-headless>=4.9. A CPU-only torch build is called out as fine, which matters if you are converting on a machine with no NVIDIA card. scripts/download_model.py is the entry point named for pulling checkpoints, and the README links a hosted GGUF set at mudler/depth-anything.cpp-gguf, so you can skip conversion entirely if one of those files matches the model you want.

Once you have a GGUF, the CLI is the first real use. The README does not print a full command line, so the flags are not something to guess at here. What it does describe is the output: the CLI writes a colorized depth map, and the repository's demo image is captioned as real photos through the actual CLI, input next to the colorized depth in the turbo colormap. The export formats are named as glb, COLMAP and PLY. Expect a depth image plus, depending on the checkpoint, confidence, pose and the optional sky mask.

The parity gate is the feature, and the conversion step is the price

Bit-exactness against the reference is unusual to claim and easy to get wrong. The way this project earns it is by refusing to hand-roll the parts that would silently drift. The requirements file names opencv-python-headless specifically for bit-exact resize parity in conversion, which is the kind of detail that shows where the authors expected divergence: image preprocessing, not the transformer. If your pipeline resizes differently from the reference, you are outside the parity guarantee no matter what the engine does.

The cost is that adoption has two halves with different dependency profiles. The inference half is clean: a native library and a CLI, no Python, no PyTorch, no CUDA toolkit. The conversion half is a Python environment with torch, safetensors and the upstream ByteDance package. Anyone who wants to use a checkpoint the hosted GGUF set does not cover has to stand up that environment once. That is a real friction point for a C++ team, and it is not something the project can remove while the parity gate depends on PyTorch-dumped reference tensors.

Quantization is the other lever. The README gives q4_k as 99 MB, described as 0.25x the f32 size and near-lossless, with f16, q8_0, q6_k and q5_k also available. Bit-exact parity is stated to hold down to f16. Note the boundary: the parity claim stops at f16, so q8_0 and below are near-lossless by the project's own description rather than bit-exact.

Where the performance numbers come from, and what they do not cover

The published table is one machine: an AMD Ryzen 9 9950X3D, 16 cores and 32 threads, threads=16, 504x336 input, sustained at repeat=25, against a PyTorch f32 baseline. In that configuration the README reports 1.20x over PyTorch at f32 and 1.31x at q8_0, with peak RAM dropping from 1328 MB to 614 MB at f32 and 363 MB at q8_0, and load time falling from 749 ms to 112 ms at f32 and 40 ms at q8_0. q4_k is the smallest file at 99 MB but slower than f32 in that table, at 395.2 ms versus 346.4 ms, which is worth knowing before reaching for the smallest quant on a latency-sensitive path.

The README attributes the speedup to a specific fix: two positional embeddings, the DPT head's UV embedding at roughly 90 ms and the backbone's bicubic pos-embed at roughly 10 ms, were being recomputed every forward with single-threaded scalar sin/cos and bicubic loops, even though they depend only on input geometry. Caching them removed about 95 ms of per-forward host overhead. That is a well-identified bottleneck, and it also tells you the win is a host-side win, which is why the GPU picture looks different.

On an NVIDIA GB10 with -DDA_GGML_CUDA=ON, the README reports the ggml CUDA path with flash attention tying PyTorch's tuned cuDNN at 47.3 versus 47.3 ms per forward, roughly 47 ms across every quant, with 1.75-2.9x faster loading and a cold start of about 548 ms versus about 926 ms. So on GPU the inference advantage largely disappears and what remains is load time and memory. The README notes these are x86 plus oneDNN numbers and points at benchmarks/BENCHMARK.md for methodology, which is where you should look before treating any of this as predictive for your hardware.

Depth Anything V2 support is narrower than the DA3 headline suggests

The README title and the repository description both lead with DA2 and DA3, but the tables do not treat them equally. The DA3 table lists seven models with depth plus confidence plus pose, and the giant variant adds 3D Gaussians. The DA2 table lists six model families, and every one of them is depth only. The README states it plainly: DA2 is depth only, no pose, no confidence. The ViT-g DA2 checkpoint is not shipped either, because the README says its Depth-Anything-V2-Giant Hugging Face repository is gated or unreleased.

That asymmetry matters for anyone arriving from a DA2 workflow. If your pipeline was built around DA2 and you were hoping to pick up pose or confidence by switching to this engine, you will not get it from a DA2 checkpoint. You need a DA3 checkpoint for the full output surface, and that means a different model with different quality characteristics.

The metric semantics also differ by family in a way that is easy to trip over. DA2 relative models emit inverse depth through a ReLU, not depth. DA2 metric models emit metres, scaled by a max_depth constant that is baked into the checkpoint family: 20 for Hypersim, 80 for VKITTI. DA3 metric variants use their own metric branch. If you swap checkpoints without checking which convention you are in, your depth values will be wrong by an inversion, not by a scale factor.

The alternative: staying in PyTorch, and what that actually costs

The obvious alternative is to run the upstream ByteDance Depth Anything 3 package directly, which is what the conversion script already depends on. The difference in approach is not accuracy, since this project claims bit-exact parity with that reference. It is deployment shape. Upstream gives you a Python process with torch, which is the right answer if your depth estimation is a batch job, a notebook, or a service that already runs Python. It also gives you the full upstream model zoo without a conversion step, including checkpoints this project's supported table does not list.

What you give up by staying upstream is the thing this project was built for: a native library with a flat C API in include/da_capi.h that can be embedded from C, C++, Go or Rust, and a GGUF file small enough at q4_k that model distribution stops being a packaging problem. The README states the LocalAI backend is built on that C API, which is a concrete signal that the embedding path is exercised rather than decorative.

A third option is to use the hosted GGUF files at mudler/depth-anything.cpp-gguf directly if they cover your model. That skips the Python conversion environment entirely, which is the largest single piece of friction in adopting this project from a C++ codebase. Check the hosted set against the supported-model table before you plan around it.

Maintenance, licence and what you are taking on

The repository is not archived, and the last push was on 2026-09-07. There are no releases retrieved, which means there is no tagged version to pin to and no changelog to read between updates. Building from master is the practical path, and that has the usual consequence: your build is coupled to whatever the default branch looked like on the day you cloned it.

The licence is MIT, which is permissive and imposes no copyleft obligation on your own code. Two things sit outside that grant and are worth naming without pretending to give legal advice. First, the model weights are not covered by the repository's MIT licence; the README points at official Depth-Anything-3 checkpoints on Hugging Face and at the project's own converted GGUF set, and those carry their own terms. Second, the repository uses git submodules for third_party, so the ggml code and any backend you enable arrive under their own licences, and the CUDA path pulls in NVIDIA components. Check those separately if you are redistributing a binary.

Upgrade cost is dominated by the conversion step rather than the C++ build. If the GGUF format or the metadata schema changes, existing converted files may need regenerating, and regenerating them means the Python environment with torch and the upstream package. The parity gate is the safety net here: it is the mechanism that would catch a silent drift after an upgrade, but only if you run it against reference tensors you dumped yourself.

Editorial conclusion

Adopt depth-anything.cpp if you need monocular depth and camera pose inside a C++ or Go process, on CPU, without shipping PyTorch, and you are willing to convert the checkpoint yourself with the Python script in scripts/. Skip it if you need DA2 pose or confidence, since the README states DA2 is depth only, or if you want a prebuilt package, because there is no release and no package manager entry. Before committing, verify three things: that your target checkpoint is in the supported table, that your converted GGUF reproduces the reference tensors through the parity gate, and that your quant choice holds up on your own images, since the published table covers one AMD Ryzen 9 9950X3D configuration at 504x336.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. localai-org/depth-anything.cpp on GitHub
4. README