Rerun: a multimodal data layer for robotics and physical AI

The problem Rerun targets: a robot that logs text but not the world

A robot that keeps hitting walls will happily log "Going through doorway". That string tells you nothing about why the planner decided a wall was a door. The README uses exactly this example, and it is the clearest statement of the project's purpose: you need a visual and temporal debugger that holds every representation the system has of the world at once, including the RGB camera feed, the depth image, the lidar scan and the segmentation mask.

The audience follows from that. Rerun is aimed at people building robots, simulation pipelines and computer-vision systems where several sensors produce data at different rates and the failure only makes sense when the streams are aligned in time. The README names robotics, simulation and computer vision, and adds anything that involves many sensors or other signals evolving over time. If your data is a single tabular stream, this is heavier than you need. If your data is a video file and a text log that never have to be compared frame by frame, a normal debugger plus print statements is still cheaper.

How Rerun stores and renders multi-rate data

The storage layer is the part worth understanding before you adopt anything. Rerun describes itself as built in Rust on column-chunk storage purpose-built for multi-rate physical data. The word multi-rate matters: a camera at 30 Hz, a lidar at 10 Hz and a joint-state stream at 1 kHz do not share a clock, and the store is designed to hold them without forcing a common sample rate.

Data enters through SDKs in Python, Rust and C++. Each log call targets an entity path and carries a timestamp on a named timeline. In the README example, rr.set_time("frame", sequence=42) associates everything logged afterwards with frame 42, and rr.log("path/to/points", rr.Points3D(positions, colors=colors)) writes colored 3D points to that entity. The viewer then renders all entities in sync and in realtime, so you can scrub episodes, compare sensors side by side and watch a CV pipeline run live.

The same data is described as queryable with dataframes or SQL, and streamable directly into training. That is the claim that separates Rerun from a pure visualizer: the recording is not a dead artifact you look at and discard. The repository layout backs the breadth of that claim, with crates grouped under store, viewer_support, views, panels and data_flow, plus separate rerun_py, rerun_cpp, rerun_js and rerun_notebook directories. It also means the surface area is large, and the README does not document how the streaming-to-training path handles backpressure or partial writes.

Installing Rerun and logging your first points

The README gives one command for the Python path, and it installs both the SDK and the viewer binary. The SDK is published on PyPI as rerun-sdk and is also available through conda.

pip install rerun-sdk

After that, rerun --help should work in any terminal, because the Python package bundles the Viewer. C++ and Rust do not: the README states that only the Python SDK comes bundled with the Viewer, while C++ and Rust always rely on a separate install. For Rust, the SDK is added with cargo add rerun, and the CLI binary is installed separately with cargo install rerun-cli --locked --features nasm. That nasm feature needs the nasm CLI on your path; skipping it is allowed, but the README warns it may result in inferior video decoding performance.

The smallest useful Python program is the README's own taste of the API. It initializes a recording, spawns a child viewer process, sets a timeline value and logs points.

import rerun as rr  # pip install rerun-sdk

rr.init("rerun_example_app")

rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk
# rr.connect_grpc()  # Connect to a remote viewer

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time("frame", sequence=42)

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors))

What you should see is a viewer window containing the points under the entity path you chose, with a frame timeline you can scrub. The two commented lines show the alternatives to spawning: rr.save writes everything to a .rrd file on disk, and rr.connect_grpc attaches to a viewer running elsewhere. The README does not document what happens when a gRPC connection drops mid-stream, so treat that mode as something to test yourself rather than assume.

Where Rerun slows down or is the wrong tool

The README is unusually direct about shortcomings, and both are worth taking at face value. The first is that the viewer slows down when there are too many entities, linked to a specific issue in the tracker. The second is that multi-million point clouds can be slow. Neither comes with a threshold in the README, so the honest position is that you have to find your own ceiling by logging at realistic scale before you build a workflow around it.

The status section adds a broader constraint: the project is in active development, many features are still wanted, and the API is still evolving, with the README stating plainly to expect breaking changes. For a debugging tool you run locally against your own logs, that is tolerable. For a long-lived internal platform that other teams write against, it means pinning versions and budgeting for migration work on each release.

There is also a structural mismatch to watch for. Rerun is built around spatial and temporal multimodal data. If your problem is a database query over structured records, or a dashboard over aggregate metrics, the column-chunk store and synchronized 3D viewer are machinery you will pay for and not use. The same applies if you only need to inspect one image at a time.

How Rerun differs from Foxglove and from plain MCAP files

The closest comparison is Foxglove, which also targets robotics visualization and also works with MCAP. The difference is in where the data lives and what you can do with it afterwards. Foxglove is primarily a viewer and bridge layer: you bring a live connection or a bag file and inspect it. Rerun positions itself as a data layer, with its own column-chunk storage and its own .rrd format, and the README claims the same data is queryable with dataframes or SQL and streams directly into training without export jobs or stale copies.

That is a real architectural difference, and it cuts both ways. If your team already standardizes on MCAP and only needs a viewer, Rerun's ingestion of MCAP is a convenience, not a reason to move your storage. If you are spending engineering time writing export scripts to get robot logs into a training pipeline, the query-and-stream path is the part that addresses your actual cost.

The second alternative is not a product but a practice: keeping MCAP files and writing your own plotting and slicing scripts. That is genuinely cheaper for small datasets and one-off investigations, and it keeps you on a format other tools read. It stops scaling when you need the camera, lidar and joint states aligned on a scrubable timeline, which is the specific thing the viewer exists to do.

Licence, release cadence and the cost of upgrading

Rerun is dual-licensed under MIT and Apache-2.0, with LICENSE-MIT and LICENSE-APACHE at the repository root and the Cargo workspace declaring license = "MIT OR Apache-2.0". That is a permissive combination and is the same choice many Rust projects make; it is not legal advice, and if you redistribute the viewer or embed the SDK in a product, have your own counsel read the two files rather than relying on a summary.

The release cadence is visible from the tags: 0.37.0 on 2026-09-01, 0.37.1 on 2026-09-04 and 0.37.2 on 2026-09-11. Patch releases within days of a minor release suggest active bug-fixing, and the last push to the default branch was on 2026-09-15. The cost of that cadence is the one the README already flags: an evolving API and expected breaking changes. The workspace Cargo.toml shows how the maintainers handle this internally, noting that alpha releases are treated as incompatible and that exact versions are pinned with = so that a 0.3.0-alpha.0 build does not silently pick up 0.3.0-alpha.4. If you depend on Rerun from Rust, the same pinning discipline is the practical upgrade strategy.

For Python, the pyproject.toml shows the development workspace requires Python >=3.10,<3.13, which is a useful signal about which interpreters the project actually tests against. The README does not document a rollback procedure for a bad upgrade, so keep a known-good version pinned in your own lockfile.

Editorial conclusion

Adopt Rerun if you debug multi-sensor robot or CV pipelines and want images, point clouds, transforms and time series on one timeline, or if you need to query and stream that data without an export job. Do not adopt it as a stable, frozen API surface: the README states the API is still evolving and to expect breaking changes. Before committing, verify the two shortcomings the README lists, viewer slowdown with too many entities and slow multi-million point clouds, against your own entity count and cloud size, and check whether you need the separate rerun binary for your language.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. rerun-io/rerun on GitHub