projectaria_tools: working with Project Aria data

A research toolkit for Aria recordings

projectaria_tools is described as a suite of C++ and Python utilities to help researchers work with Project Aria data, and the README is explicit about its purpose: to expand the horizons of augmented reality, machine perception and artificial intelligence research using the Aria device and its open datasets.

The problem it solves is not the camera. It is the container. An Aria recording is not a single video file. It is a VRS file holding several synchronised streams, each with its own configuration, plus calibration data describing the cameras and their relationship to the device body, plus, often, the output of on-device perception algorithms. A researcher who wants to, say, crop an image from the RGB camera using the pose of the head at that timestamp has to understand all of that.

This toolkit is the layer that handles it. The repository contains a core library, a Python package, tools, per project directories and examples for both device generations, and it is published both as source and as wheels.

The audience is therefore research groups working with Aria hardware or with the open datasets Meta publishes from it, in fields where egocentric video and head pose are the raw material: scene understanding, hand interaction, gaze estimation, and the evaluation of perception systems running on wearable devices.

One API across two device generations

The headline change is Aria Gen2 support, which arrived with full API support in the 2.0.0 release. The README states that the toolkit supports both Aria Gen1 and Aria Gen2 data, and the software section of the changelog calls out unified APIs, with the same Python and C++ interface for both generations of data.

That matters more than a compatibility checkbox. The Gen2 hardware is substantially different: a 12 megapixel RGB camera, four computer vision cameras with a wider field of view, HDR and front-facing stereo, two eye tracking cameras, and new sensors covering proximity, a contact microphone, PPG health, ambient light and GNSS. The form factor is foldable, with a battery life of six to eight hours and direct interaction with open-air speakers.

On-device machine perception is powered by a custom Meta co-processor and covers eye tracking, hand tracking with 21 keypoints, and VIO and SLAM producing a 20 Hz trajectory plus an 800 Hz high frequency trajectory.

The practical consequence of the unified interface is that a pipeline written for Gen1 mostly survives the move to Gen2. The alternative, which many research groups will recognise, is a branch in every script whenever the sensor layout changes. The README also lists new tools that come with Gen2: aria_rerun_viewer for interactive 3D visualisation, gen2_mp_csv_exporter, and an upgraded vrs_health_check, plus enhanced streaming over USB or wireless with on-device perception signals.

The tutorial path through the data

The README links a set of interactive Python tutorials hosted on Google Colab, and the sequence they form is the best available description of the library.

The first tutorial covers VrsDataProvider basics, loading an Aria VRS file and accessing its streams. The second is device calibration, with sensor intrinsics and extrinsics, projection and undistortion. The third covers sequential access to multi sensor data, described as the queued API for streaming several sensors in timestamp order. The fourth is device time alignment, covering time domains, timestamp queries and multi-device alignment over SubGHz. The fifth is machine perception services, with the MPS output layout and the MpsDataPathsProvider and MpsDataProvider classes. The sixth covers VIO and trajectory, contrasting the on-device VIO with the MPS trajectory and the semi-dense point cloud. The seventh is hand tracking, and the eighth is eye tracking, which the README describes as gathering gaze from all three sources: on-device geometric, on-device ML and MPS. The ninth covers the Meta Neural Band sEMG stream, its batch model and its two clocks.

That ordering is a curriculum. It starts with how the file is structured, moves to how the sensors relate to each other geometrically, then to how time works across streams and devices, and only then to the perception outputs on top of that.

One detail to notice: the Gen1 tutorials are pinned to version 1.6.0 of the toolkit while the Gen2 notebooks point at 2.3.0. The notebooks are version specific, and a notebook that fails to run is usually a version mismatch rather than a broken environment.

On-device signals versus machine perception services output

The tutorials make a distinction that is easy to miss and expensive to get wrong: several perception signals exist in two places, and they are not the same thing.

Eye tracking, for example, is available from three sources, as the README puts it: on-device geometric, on-device ML and MPS. The hand tracking tutorial similarly presents hands from both sources, the on-device stream and the MPS result. The VIO tutorial contrasts the device pose produced on-device with the MPS trajectory and the semi-dense point cloud.

The practical difference is that an on-device algorithm runs on the co-processor at capture time and produces a result alongside the recording, while MPS is a cloud or offline service that processes the recording afterwards and returns a result that you then have to load and align with the sensor data. A result that was computed at capture time is already in the file; a result from MPS has to be fetched, and its timing relative to the sensors has to be established by you.

That is why the time alignment tutorial sits before the MPS tutorials in the sequence, and why the sEMG tutorial spends attention on the stream having two clocks. When you build an analysis that fuses, say, gaze with hand position, you are mixing signals with different origins and different time bases, and the toolkit gives you the pieces to resolve that, but the resolution is your work, not the library's.

The Multi-device alignment over SubGHz material is the same idea one level up: when several Aria devices record together, their clocks differ and have to be reconciled before you can treat the recordings as one capture.

Building the wheels and the native library

The build is CMake based with a Python binding layer on top, and the repository shows the packaging work in detail.

The setup.py defines a CMakeExtension and a custom build_ext command, which is the standard way to compile a C++ library and expose it as a Python extension. The version is read from a version.txt file at the root rather than being hard coded.

def _get_version():
    path = os.path.join(ROOT_DIR, "version.txt")
    version = open(path, "r").read().strip()
    return version

The pyproject.toml documents how the wheels are produced, and the commands are given as comments for each platform.

#    python3 -m cibuildwheel --output-dir dist --platform linux .
#    python3 -m cibuildwheel --output-dir dist --platform macos --arch x86_64 .
#    python3 -m cibuildwheel --output-dir dist --platform windows --arch AMD64 .

The Windows configuration is where the complexity shows. The file explains that VCPKG is used to retrieve and compile static libraries, that only release artifacts are built because building all of boost is slow, and that a pinned VCPKG commit is used for a CMake 4.0 fix until a release tag carries it. The macOS configuration installs its dependencies with brew, including boost, fmt, glog, jpeg-turbo, libpng, lz4, xxhash, zstd and opus, and pins the deployment target and the clang compilers through CMAKE_ARGS.

There is also a pixi.toml with a matching lock file, which is an alternative environment definition for development, a build_third_party_libs directory, and a generate_stubs.py script at the root for producing the typing stubs used by the Python API.

Tools, datasets and the maintenance picture

Three command line tools are worth knowing by name. vrs_health_check inspects a recording and reports on its health, which is the first thing to run when a capture looks wrong. aria_rerun_viewer gives an interactive 3D visualisation for inspecting what a rerun produced, and gen2_mp_csv_exporter exports Gen2 machine perception data to CSV for analysis outside the toolkit.

The open datasets are the other half of the story. The Aria Gen2 Pilot Dataset is described as multi-participant indoor and outdoor recordings, with the full on-device outputs including eye tracking, hand tracking and VIO, SubGHz synchronised multi-device captures, and high quality MPS outputs covering SLAM, point clouds and trajectories. The Gen1 Everyday Activities dataset is listed alongside it, with its own interactive notebook.

On releases, the tags are 2.3.0 from 2026-08-27, 2.2.0 from 2026-08-13 and 2.1.4 from 2026-06-11, and the last push was on 2026-10-21. The cadence is a patch release every few weeks during active development of a feature area, which fits a research toolkit that has to follow new device firmware and new dataset formats.

The licence is the Apache License 2.0, with the Meta Platforms copyright header at the top of setup.py and a LICENSE file in the repository root. That permits use in commercial and research work with attribution and the Apache patent conditions, though whether it fits a given organisation is a question for your own legal review.

Editorial conclusion

projectaria_tools is the toolkit to adopt if your research data comes from an Aria device, because it removes the format work that otherwise dominates an AR dataset project, and because the Gen2 support arrived with a full API surface rather than as a reader bolted on afterwards. It is the wrong choice if you do not have Aria hardware or Aria open datasets, since there is little in it that applies to another sensor, and it is the wrong choice for a team that needs only video decoding. Verify first that the release you install matches the device generation of your recordings, read the version pinned by the notebook you follow so that the API matches, and check that the platform and Python version in the wheel configuration matches your machine before you build.

Frequently asked questions

Official sources

1. facebookresearch/projectaria_tools on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases