QuestCameraKit: six Unity samples for the Quest passthrough camera

What the kit actually is

The repository tagline is that it builds mixed reality experiences that can see the world around you, and the description calls QuestVisionKit a collection of template and reference projects demonstrating how to use Meta Quest's new Passthrough Camera API for advanced AR and VR vision, tracking and shader effects.

The README is more specific about what is inside: Unity samples for Meta Quest 3 and Quest 3S, to sample real-world colours, detect objects, track QR codes, create camera-based shader effects, ask questions about your surroundings, and stream video to another device. The primary language is C#, the licence is MIT, the default branch is main, and the recent releases are v81, marked 2.0, from 2025-11-09 and a release marked 1.0 from 2025-03-17. The last push was on 2026-09-23 and the project is not archived.

The repository layout matches that description. There is a Unity-QuestVisionKit/ directory holding the Unity project, a docs/ directory, a scripts/ directory, and the usual repository files including a CODE_OF_CONDUCT.md and a LICENSE. The scenes live under Unity-QuestVisionKit/Assets/Samples.

What this is not is a reusable library. It is a reference implementation set, which means the right way to read it is as working code that shows how the API behaves, particularly around permissions and the per-eye camera layout, rather than as a package you would add to your own project.

A headset build is the only honest test

The What you need section is where the project's practical constraints live, and there are four prerequisites.

A Quest 3 or Quest 3S, configured for development, with USB debugging authorized and headset software that supports the camera and spatial features you want to use. Unity 6000.3.12f1, installed through Unity Hub with Android Build Support, Android SDK and NDK Tools, and OpenJDK. Git and Git LFS, plus network access for Unity Package Manager. And touch controllers for the sample menu, with individual samples possibly also supporting hand interactions.

Then the sentence that decides whether your first hour was productive:

Use a headset build to test the camera workflows. An Editor preview does not establish on-device camera or tracking behaviour.

That is a stronger claim than the usual warning that the editor differs from the device, and it follows from what the API does. The passthrough camera is a device capability with device permission grants, and neither the camera feed nor the native spatial tracking exists in an editor preview. A developer who tries the samples in the editor will conclude that the camera is broken when it is not.

The repository pins the exact editor version, 6000.3.12f1, in a badge that points at the ProjectVersion.txt file, which is a good sign for reproducibility, and it links to Meta's Passthrough Camera API documentation for the platform requirements.

Cloning, and the LFS step that catches people out

The clone sequence is four commands:

git lfs install
git clone https://github.com/xrdevrob/QuestCameraKit.git
cd QuestCameraKit
git lfs pull

The git lfs install and git lfs pull steps are there for a reason. Unity projects routinely contain binary assets, and the Git LFS commands are what make the sample scenes, models and media actually arrive rather than as pointer files that Unity then fails to load. Running git lfs install before cloning configures the filter, and git lfs pull after the clone fetches the binaries for the checkout you just made.

After that the README gives five numbered steps. Open Unity-QuestVisionKit in Unity Hub with the pinned editor version. Let Package Manager finish importing the pinned dependencies, where QR tracking uses the included Meta MRUK package and WebRTC dependencies resolve automatically. Select Android in Build Profiles and connect your headset. Choose Build and Run. And keep the headset awake, accept the relevant camera and spatial-data permissions, and choose a sample from the menu.

The build includes all six headset samples, starting with ColorPicker. The desktop WebRTC receiver is separate, which is a detail that matters for the sixth sample, since that one needs two peers.

The control scheme is a small table worth reading before you put the headset on, because the sample menu is driven by controllers. Showing or hiding the menu is Y on the controller and M on the editor keyboard, choosing a sample is left thumbstick up and down or the arrow keys, and opening a sample is X or Enter.

Six samples, and what each one is really demonstrating

The sample table gives the one-line description of each, and the details below it explain what is interesting in each.

The color picker maps a point in the room to a camera pixel. The description is precise about the mechanism: you aim at a surface and press and release A to sample its colour, and the sample uses environment raycasting to locate a point in the room, projects it into the camera image, and applies the sampled colour to a virtual object. That three-step chain, raycast, project, apply, is the whole AR colour workflow, and it is the sample that explains how the camera image relates to world space.

Object detection runs the bundled YOLO model with Unity Inference Engine and projects detections into the room, running automatically. You configure label filters on ObjectRenderer to show selected classes, or leave the list empty to show all model classes, and confidence filtering applies when the model supplies scores. The README is candid that detection accuracy and performance depend on the scene, the model and the device, and there is a supported labels file in the sample's Scripts directory pointing at YOLOv9Labels.cs, which tells you the model generation is fixed.

Native QR tracking reads QR payloads through Meta MRUK with no additional decoder or NuGet setup, displaying the decoded payload and the bounds supplied by native tracking. You use the controller-mounted panel to request scene and spatial-data permission, check support and enable tracking, and present a clear, well-lit QR code. The limitation stated is important: native tracking updates spatial poses over time and is not intended for fast-moving object tracking.

Camera shaders explore stereo passthrough mapping, frosted glass and wavy portal effects in one scene, using both camera feeds and exposing per-eye UV offsets for calibration. The per-eye offsets are the detail to note, because they are what makes a shader effect look right or wrong on a stereo headset, and the sample exposes them explicitly for that purpose.

Image and voice AI records a question, transcribes it, sends the text and a camera frame to OpenAI, and plays the response with text-to-speech. You configure a private development API key on the scene's ImageOpenAIConnector or OpenAI Manager, choose the model and command mode, connect to the network and grant microphone permission, and press the Menu or Start button on the controller to start or stop recording, with recording also stopping at the configured maximum duration.

WebRTC streaming sends camera video to a receiving peer using SimpleWebRTC, Unity WebRTC and NativeWebSocket. You configure your own WebSocket signalling server on both peers, the connection sitting under BuildingBlock, Camera Rig, TrackingSpace, CenterEyeAnchor, Client-STUNConnection, and you enable WebSocket Connection Active after replacing the placeholder address, because signalling connections are disabled by default.

The API key advice, and the parts the samples cannot tell you

The image and voice sample carries the most useful warning in the README, and it is worth quoting in substance. Keep real keys out of commits and distributed APKs. For a shipped application, route authenticated requests through your own backend. The sample requires an OpenAI account with API access, and network requests may incur charges.

Three separate concerns are bundled there. The first is that a key baked into a build is a key you cannot rotate, and a distributed APK is a key you cannot revoke. The second is that putting a provider key in client code means anyone with the APK has it. The third is cost, because an agent that sends a camera frame per question is a metered API call, and a loop in the code is a bill.

The advice to route through your own backend is the correct pattern and also the reason this sample is the one most often a starting point rather than a copy-paste. A backend proxy lets you hold the key, rate-limit the caller, and log what was sent.

The other limitation the samples cannot resolve for you is the permission model. Each sample needs its own grants, spatial data or camera, and the pattern in the QR sample is the one to copy: request the permission through the controller-mounted panel, check support, enable tracking, and only then expect data. A developer who understands that sequence from the sample will avoid the more common failure of writing code that assumes the permission was already granted.

And the last gap is performance. The README says detection accuracy and performance depend on the scene, model and device, and the CI and CI, which is exactly the kind of statement that a benchmark cannot be written from. A YOLO model running through Unity Inference Engine on a standalone headset, at whatever frame rate the camera allows, is a design constraint you will have to measure yourself on the device you ship to.

Editorial conclusion

Adopt QuestCameraKit as a reference when you are building a passthrough camera feature on Meta Quest 3 or 3S and want working code for the permission flow, the per-eye camera layout and the raycast-to-pixel projection rather than reading the API documentation alone. Do not treat it as a library to depend on, since it is a set of sample scenes, and do not ship the image and voice sample with a provider key in the build, because the README is explicit that keys belong out of commits and that a shipped application should route requests through its own backend. Before you commit, build to the headset rather than previewing in the editor, pull the Git LFS assets, and measure detection performance on your target device, since the README says accuracy and speed depend on the scene, the model and the hardware.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. xrdevrob/QuestCameraKit on GitHub