BOP Toolkit scores a pose estimator the same way every lab scores it

A scoring package, and the benchmark it belongs to

The BOP Toolkit is a Python toolkit of the BOP benchmark for 6D object pose estimation, and the homepage is bop.felk.cvut.cz. The primary language of the repository is listed as Jupyter Notebook, which is a fair signal about how the work is used.

6D object pose estimation is the problem of finding where an object is and how it is oriented, in six degrees of freedom. The BOP benchmark is the community reference for that problem, and the toolkit is the software that turns your estimated poses into the error numbers the leaderboard compares.

The repository is organised in three parts. bop_toolkit_lib is the core Python library for i/o operations and calculation of pose errors. docs holds documentation and conventions. scripts holds scripts for evaluation, rendering of training images and visualisation of 6D object poses.

That split is the shape of the whole thing. The library is the error computation and the file handling, and the scripts are the entry points that a user actually runs.

The licence is MIT, the default branch is master, and there are no GitHub releases, so the version that matters is the one you get from the package index or the repository. The last push was on 2026-06-19.

Two things are worth noting about the packaging before you go further. The pyproject.toml names the distribution bop_toolkit_lib at version 0.1.0, with authors Tomas Hodan and Martin Sundermeyer, and it declares requires-python of 3.8 or newer and below 3.13. There is also a package.xml at the repository root and a data-files entry installing it into an ament index resource path, which is the ROS packaging convention, so the toolkit is consumable from a ROS workspace as well as from pip.

Installing it, and the optional extras that decide your path

Supported Python versions are 3.8 to 3.12, and there are two installation routes.

With uv, the command is a single line.

uv sync

That sets up a local venv, which you activate with source .venv/bin/activate, installs the necessary dependencies and bop_toolkit_lib. Extra flags are available: --python 3.10 to set the venv interpreter version, and --extra eval_coco, --extra eval_gpu, --extra eval_hot3d and --extra scripts to pull in the dependencies for those areas.

With pip, the same extras are expressed as package extras.

pip install .  # bop_toolkit_lib with core dependencies only
# with additional dependencies
pip install .[eval_coco]  # install dependencies for coco evaluation
pip install .[eval_gpu]  # install dependencies for gpu evaluation
pip install .[eval_hot3d]  # install dependencies for hot3d evaluation
uv pip install .[scripts]  # install dependencies for utility scripts (e.g. `annotation_tools.py`)

The extras are defined in pyproject.toml and they are not decorative. eval_coco pulls pycocotools from a fork of the cocoapi repository, eval_gpu pulls torch, eval_hot3d pulls the Hand Tracking Toolkit, and scripts pulls open3d and trimesh.

The core dependency list is worth reading for what it implies about the environment. numpy is pinned below 2.0.0, and alongside it sit pillow, pypng, pytz, webdataset, pyopengl, imageio, scikit-image, scipy, vispy, opencv-python, matplotlib and tqdm.

The numpy ceiling is the kind of detail that saves an afternoon, because a scientific Python stack that has been upgraded to numpy 2 will not install cleanly against this package.

There is also a unit test suite, and the command in the README has a stray backtick at the end of it, which is worth knowing before you paste it.

python -m unittest discover bop_toolkit_lib/tests`

The presence of uv.lock at the repository root means the exact resolved dependency set is recorded, which is what you want for reproducing a published number.

Four environment variables decide where everything lives

Before any script runs, the toolkit needs to know where your data is, and it takes that from four environment variables, with config.py holding the full list.

BOP_PATH is the root of the BOP datasets on your machine. BOP_RESULTS_PATH is the folder with pose results to be evaluated. BOP_EVAL_PATH is the folder for the calculated pose errors and performance scores. And BOP_OUTPUT_PATH is the folder for outputs, for example visualisations.

The configuration file is bop_toolkit_lib/config.py, and the README says the paths may be specified as environment variables or modified in config.default_paths.

That two-level arrangement matters in practice. Environment variables are how you point the same installation at different dataset copies on a shared machine or in a container, and config.default_paths is where a site pins its own convention so that a person running a script does not have to remember four exports.

The four variables also encode the data flow of a whole evaluation, which is worth reading as a sequence. The datasets come in through BOP_PATH and are read-only inputs. Your method's output goes into BOP_RESULTS_PATH as one .csv file per dataset. The evaluation writes errors and scores into BOP_EVAL_PATH. And the visualisation scripts write images into BOP_OUTPUT_PATH.

Keeping those separate is what allows the same result files to be scored again with a different method version or re-rendered after a renderer change, without recomputing anything.

Evaluation scripts, and why there are several of them

The usage section walks through a five-step workflow, and the evaluation scripts differ by task rather than by year, which is a point worth making because the names look like a chronology.

Step one is getting the BOP datasets into the expected folder structure. Step two is running your own method and saving the estimated poses in one .csv file per dataset. Step three is configuring the toolkit with the paths above. Then there are the evaluation steps.

For the 6D detection task, the script is eval_bop24_pose.py.

python scripts/eval_bop24_pose.py --result_filenames=NAME_OF_CSV_WITH_RESULTS --use_gpu

The --use_gpu flag requires PyTorch and a GPU with CUDA support, and the README is specific about a constraint: the current implementation limits GPU memory usage to less than 2GB for BOP servers, with a max_batch_size parameter to raise it on bigger GPUs. Without a GPU the evaluation runs on CPU with 10 parallel processes, and --num_worker 1 changes that count.

For the 6D localization task the script is eval_bop19_pose.py, and it takes a renderer type.

python scripts/eval_bop19_pose.py --renderer_type=vispy --result_filenames=NAME_OF_CSV_WITH_RESULTS

The renderer_type accepts vispy, python or cpp, and the README recommends vispy because it is easy to install and works headlessly. The cpp option requires the separate C++ renderer.

For detections and instance segmentations there is a third script that takes COCO-format JSON rather than CSV.

python scripts/eval_bop22_coco.py --result_filenames=NAME_OF_JSON_WITH_COCO_RESULTS --ann_type='bbox'

The --ann_type flag chooses between bbox for amodal bounding boxes and segm for segmentation masks.

So the naming reflects which task the script was introduced for, and the practical question is which task you are being scored on, not which year.

Three renderers, and why the choice changes the numbers

The renderer is not a display detail here. It decides what the evaluation actually sees, and there are three implementations with different characteristics.

The default is the Vispy renderer, a Python based headless renderer with an EGL backend implemented using Vispy, and Vispy is installed by the pip command above. The README warns that the nvidia opengl driver might be required in case of any errors, which is the practical detail that catches people out on a headless GPU box.

The Python renderer is marked deprecated. It is implemented using Glumpy, which depends on freetype and GLFW, and the README notes it is similar to the vispy renderer since glumpy and vispy have similar apis, but that it does not support headless rendering. On Linux freetype and GLFW install with two apt commands.

apt-get install freetype
apt-get install libglfw3

The third is the C++ renderer, and for fast CPU-based rendering on a headless server the README recommends installing bop_renderer, an off-screen C++ renderer with Python bindings, from a separate repository.

That recommendation is the interesting one. The evaluation of a 6D pose involves rendering the object under the estimated pose and comparing images, and the renderer is in the loop for every frame of every object in every scene. A Python renderer that cannot run headless is a real constraint on a server, and an off-screen C++ renderer with Python bindings is the answer to it.

This is also where reproducing a published number gets delicate. If the published evaluation used the C++ renderer and you use Vispy, small differences in rasterisation can move the error metrics. The renderer type is part of the configuration, not an incidental detail.

Visualising results, and the HOT3D special case

The scripts directory also holds the visualisation tools, and they are what you use to see why a score is what it is.

vis_poses.py visualises estimated and groundtruth poses, and vis_poses_cli.py holds the CLI arguments for the gt, groundtruth, and est, predictions, visualisation modes. There is a separate vis_object_symmetries.py that renders object symmetries.

Two sample commands show the shape of the interface, where the mode is a positional argument and the dataset is a flag.

python scripts/vis_poses.py gt --dataset ycbv --scene_ids 48 --vis_orig_color

python scripts/vis_poses.py est --result_filename /bop/results/gt-equivalent_ycbv-test.csv --n_top 2

The --n_top flag limits how many predictions are drawn, which matters because a per-scene prediction file can contain many hypotheses and drawing all of them is unreadable.

There is also a fuller visual suite, since the README mentions passing --vis_types overlay depth_diff depth_heatmap contour bbox3d when calling vis_poses.py est to get a full suite of visuals for a given set of predicted poses. Those are not screenshots of a scene, they are diagnostic renderings: an overlay of the prediction on the image, a depth difference map, a depth heatmap, a contour and a 3D bounding box.

The one dataset with an extra requirement is HOT3D. The Hand Tracking Toolkit Fisheye camera implementation is necessary for evaluation on the HOT3D dataset, and it installs from a git URL.

The install command is `pip install git+https://github.com/facebookresearch/hand_tracking_toolkit`. That is why there is an eval_hot3d extra in the packaging: the dependency is a git install rather than a released package, so it cannot simply be listed as a version constraint.

The repository also has a notebooks/ directory, and the primary language being Jupyter Notebook reflects that there is demo work in there rather than only the evaluation scripts.

Where BOP Toolkit is the wrong tool

Three cases, and the first one is the most common misunderstanding.

It does not estimate poses. There is no pose estimator in this repository, no model weights and no inference. If you want accurate 6D poses for your objects, you need a method, and this toolkit only scores what your method produced. The usage sequence makes that explicit: run your method, save one .csv per dataset, then evaluate.

The second is reproducibility outside the benchmark's own conditions. The evaluation on the official servers is bounded to less than 2GB of GPU memory for the GPU path, and the CPU path uses 10 parallel processes. Those are the settings the published numbers were produced under, and reproducing them locally with different hardware or a different renderer can move the result. If your purpose is a scientific claim rather than a challenge submission, matching those conditions is part of the work, not an afterthought.

The third is the environment ceiling. Python support stops below 3.13 and numpy is pinned under 2.0.0, so on a machine with a modern scientific stack you are working around the environment rather than in it. The dependency list is also heavy, with scipy, scikit-image, opencv, vispy, matplotlib and imageio in the core set, and the optional extras add torch, open3d and trimesh on top.

There is one more practical gap worth naming. The README does not describe how the toolkit handles a dataset it does not know, or how it treats a result file with a missing object, and the answers are in bop_toolkit_lib/config.py and the conventions in the docs directory. For a benchmark with strict scoring rules, that is where you should read before submitting, and it is not in the file most people read first.

Editorial conclusion

Adopt bop_toolkit if you are entering or reproducing a BOP challenge result, because it is the scoring code the benchmark itself uses and the numbers are only comparable if you use it. Do not adopt it as a pose estimation library, since it contains no estimator, and note that reproducing the exact BOP server configuration means matching its renderer and GPU memory limits rather than just running the script. Verify first which evaluation task you need, the 6D detection script or the 6D localization one, since they take different inputs, and set BOP_PATH before anything else so the rest of the paths resolve against your dataset copy.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. thodan/bop_toolkit on GitHub