cam_lidar_calibration: estimating camera to lidar calibration with an uncertainty

The contribution is sample selection, not the solver

cam_lidar_calibration is the second version of the official code release for the ITSC 2021 paper Optimising the selection of samples for robust lidar camera calibration. The framing in the README is unusually precise about what the package contributes.

The stated aim is to simplify the calibration process by optimising the pose selection process, to take away the tedious trial-and-error of having to re-calibrate with different poses until a good calibration is found. And the objective is stated as obtaining calibration parameters as an estimate with uncertainty that fits the entire scene instead of solely fitting the target.

That second sentence is the thesis of the paper in miniature. Target-based calibration, where you put a chessboard in front of the sensors and solve from its corners, fits the target very well and the rest of the scene not at all, because the target occupies a small part of the field of view and the fit is overfitting that part. The README attributes this to two causes, user error and overfitting of the target.

The second version also reorganised the workflow around a feature extraction user interface where you create a region of interest where the scene is static, which is the input to the new sample selection.

What V2 changed in the workflow

The changelog lists what is new in V2 and it is worth reading as a description of the algorithm's data flow.

The feature extraction UI creates a region of interest where the scene is static. The tool then takes a sample of that static scene. The captured sample is subtracted from every consecutive frame, which is used to extract any other new item in the scene, and the purpose is to distinguish just the board in the frame.

Capture sample now takes five consecutive frames and gets the running average of the board's parameters, which is a noise reduction step. Take as many samples as you need, and the optimise step does what it used to do.

Read as a pipeline: you mark a static region, the tool captures a reference, it differences each new frame against that reference to find the target board, it averages five frames to estimate the board's parameters, and it accumulates samples. The optimise step then selects among those samples.

Two smaller changes are about output quality rather than algorithm. The visualise results script takes care of angle wrapping, to stop sudden angle sign fluctuations at plus or minus 2Pi, which is a real annoyance when a rotation estimate crosses the wrap boundary. And pre-commit was set up with Clang enforced to format all the files, with the README noting the code was hard to read and needed consistency very badly. There is a .clang-format file and a .pre-commit-config.yaml at the repository root for that.

The transform direction and the ROS dependencies

The package estimates the calibration parameters that transform the camera frame, described as the parent, into the lidar frame, described as the child. That direction matters, because getting a camera to lidar calibration backwards is a classic way to spend an afternoon confused, and the parent and child terminology here is the opposite of what a camera driver usually implies where the camera is the parent.

The implementation is C++ inside a ROS package, with a package.xml, CMakeLists.txt, a nodelet_definition.xml and a plugin_description.xml at the root, plus directories for src, include, launch, cfg, scripts, action, srv, cmake, data, docker, rviz and img. The presence of action and srv directories means the calibration runs as a ROS action with a service interface, which is the right pattern for a process that takes minutes and reports progress.

The topic requirements are specific. The lidar must publish a 3D point cloud of point type XYZIR as sensor_msgs::PointCloud2, and the README is explicit that this package relies on the ring value, so if you do not have that you need to modify your lidar driver to use this package. The camera side needs an image published as sensor_msgs::Image and the corresponding meta-information topic as sensor_msgs::CameraInfo.

The ring value requirement is the practical constraint to check first. The ring index tells the tool which laser produced each point, which is what makes it possible to separate the target board's points from the scene's points in the first place.

Installing locally with catkin

The package has only been tested in ROS Melodic, which the README states without hedging. The local install starts by cloning into the catkin workspace source folder, on the calib-v2 branch:

git clone -c http.sslverify=false -b calib-v2 https://github.com/acfr/cam_lidar_calibration

The ROS and Python dependencies come next:

sudo apt update && sudo apt-get install -y ros-melodic-pcl-conversions ros-melodic-pcl-ros ros-melodic-tf2-sensor-msgs
pip install pandas scipy

Then the package is built and the workspace sourced:

catkin build cam_lidar_calibration
source ~/catkin_ws/devel/setup.bash

The branch name matters and is easy to miss, because the default branch is not the one you want. Version 1.0 of the code, the original implementation from the paper, lives on the branch labelled master, and the README says to switch to that branch to access it.

The clone command includes -c http.sslverify=false, which disables SSL verification for that one clone. That is worth noticing rather than copying blindly, because it removes certificate checking on the fetch, and a cleaner approach in a locked-down environment is to configure the network rather than the repository.

The Docker route, and its file ownership wrinkle

The Docker path is shorter. Clone into catkin_ws/src as before, then run the image, which is pulled from ockerhub. If the machine has an Nvidia GPU, set the cuda flag to on, otherwise off:

cd cam_lidar_calibration/docker
./run.sh --cuda on

The script starts the container, which immediately builds the catkin workspace and sources the setup.bash file. A build.sh script is also provided if you want to build the image from scratch instead of pulling it.

The README then documents a real consequence of this arrangement. The run.sh script mounts the local cam_lidar_calibration folder to /catkin_ws/src/cam_lidar_calibration inside the container, so when you run the calibration it creates csv files inside the container under root ownership, which the README calls not ideal. The documented workaround is to run a command outside the container that changes ownership of all files in the current folder to match your $USER and $GROUP locally:

sudo chown $USER:$GROUP *

That is a blunt instrument, since it applies to every file in the current directory rather than to the generated csv files specifically, and it requires sudo. Anyone who has run a calibration under Docker and then found their data files unwritable will recognise the pattern, and the README is upfront about it rather than leaving it to be discovered.

Quick start: optimise then assess

The quick start uses the quick-start data and is two steps. The first runs the calibration process, taking the saved poses and computing the best sets with the lowest VOQ score:

roslaunch cam_lidar_calibration run_optimiser.launch import_samples:=true

After calibration the output is saved in the same directory as the imported samples, and for the quick start example that is cam_lidar_calibration/data/vlp/.

The second step obtains and assesses the calibration parameters by taking a filtered mean of the best sets and displaying the Gaussian fitted histogram of the estimated parameters. It also computes the reprojection error over all the data samples and a visualisation if specified. You pass the absolute path of the csv output file:

roslaunch cam_lidar_calibration assess_results.launch csv:="$(rospack find cam_lidar_calibration)/data/vlp/calibration_quickstart.csv" visualise:=true

You should see a terminal output with the reprojection errors, a Gaussian-fitted histogram and a visualisation.

The two-step split is the design worth noticing. The optimiser produces a set of candidate calibrations, and the assessment step aggregates them, which is why the result can carry an uncertainty rather than a single number. A histogram over the estimated parameters is a more honest artefact than a printed extrinsic matrix, because it tells you how tightly the data pins the estimate down. The reprojection error over all provided samples is what tells you whether the fit generalised across the scene rather than only the board.

For your own data, all data and output files are saved in a cam_lidar_calibration/data/YYYY-MM-DD_HH-MM-SS/ folder, timestamped per run.

The condition number correction and where the method stops

There is an erratum in the paper that anyone reproducing from the PDF needs to know about. Equation (2), which shows the equation for the condition number, has a typo. The correct equation for calculating the condition number is implemented in the repository, and the README includes the formula as an image.

This is worth flagging because the condition number is central to the method rather than incidental. Selecting samples so that the resulting geometry is well conditioned is what makes the estimate stable, so an error in the published equation propagates straight into the sample selection and the final calibration. The repository is the authority here, and the README says so explicitly.

As for the limits, the first is the platform. The package has only been tested in ROS Melodic, which is an Ubuntu 18.04 generation environment, and the dependencies are pinned to ros-melodic packages. The second is the ring value requirement on the point cloud, which means a lidar driver that does not publish ring indices has to be modified before this package can use it. The third is that the calibration is scene-specific by construction: the method optimises which samples to trust for one environment, and the README describes the goal as fitting the entire scene rather than solely fitting the target, so a calibration estimated in an empty car park is not automatically valid in a cluttered one.

For a comparison, a target-based approach with a chessboard and a PnP solver solves quickly and gives a small residual on the board itself, and it is the right choice when you need a quick extrinsic for a specific scene or when your lidar has no ring information. The difference in approach is that this package replaces the target-centre assumption with a whole-scene fit plus an uncertainty, at the cost of needing a static region, a scene with enough structure, and the ring values.

The licence is Apache 2.0, with a LICENCE file at the repository root, and there are no GitHub releases, so the calib-v2 branch is what you pin.

Editorial conclusion

cam_lidar_calibration suits a robotics team that has a camera and a lidar on the same rig and has found that target-based calibration results do not generalise to the rest of the scene, because the package selects samples to fit a static region and reports the result with an uncertainty estimate. It is the wrong tool if you are not on ROS Melodic with a bag file containing XYZIR point clouds and a CameraInfo topic, and it is the wrong tool if you need a calibration that holds across different scenes, since the estimate is scene-specific by construction. Verify first that your lidar driver publishes a ring value, because the package relies on it, and note that the repository has no GitHub releases with the last push on 2026-04-07.

Frequently asked questions

Official sources

1. acfr/cam_lidar_calibration on GitHub
2. Issues
3. License: Apache-2.0
4. README