OpenSeeFace: CPU face and landmark tracking that streams over UDP

A tracking library, not an avatar program

The README opens by drawing a line that most projects in this space blur: this is a tracking library, not a stand alone avatar puppeteering program. If you are looking for something that loads a model and shows a moving character, this is not it and the author points you elsewhere.

What sits on top of it are separate projects. VSeeFace animates VRM and VSFAvatar 3D models using OpenSeeFace tracking. VTube Studio uses OpenSeeFace for webcam based tracking of Live2D models. A renderer for the Godot engine exists as a separate repository. That ecosystem is the strongest argument for the design: three different consumers in three different engines, all fed by the same tracker, which is only possible because the tracker does not care what happens to the data after it produces it.

The technical core is a facial landmark detection model based on MobileNetV3. The README gives the reason for a specific technical choice: PyTorch 1.3 CPU inference speed on Windows was very low, so the model was converted to ONNX format and run through onnxruntime, where it reaches 30 to 60 fps tracking a single face. Four models ship, trading speed against tracking quality. The repository topics line up with that story: onnx, onnxruntime, mobilenetv3, pytorch, face-detection, face-landmarks, unity, virtual-youtuber.

Why the landmarks are deliberately not iBUG 68

The most technically interesting decision here is a refusal. OpenSeeFace's landmarks are close to iBUG 68, with two fewer points at the mouth corners and quasi-3D face contours instead of contours that follow the visible outline. That makes numerical comparison against the academic literature hard, and the README says so plainly rather than hiding it.

The reasoning is that the landmarks are optimised for animating an avatar, not for fitting the face image exactly. The worked example is the eyes: as long as the eye landmarks show whether the eyes are open or closed, even a somewhat misplaced position is still useful for the purpose of driving a mouth or an eye shape. Once you accept that goal, a large class of accuracy measurements stop being the right question.

On observed behaviour, the README claims the tracker holds up in low light, high noise and low resolution, keeps tracking across a wide range of head poses, and holds landmark positions steady. Compared with MediaPipe it says the landmarks stay more stable in challenging conditions and represent a wider range of mouth poses, while conceding that eye region tracking can be less accurate. That concession lines up with the iBUG comparison, since removing the mouth corners and going quasi-3D trades precision at the silhouette for stability in the interior.

The author also notes the name is a pun on the open seas and seeing faces, with no deeper meaning, which is worth knowing before you go looking for a philosophy behind it.

Running the tracker with uv

On Windows, a downloaded release ships `facetracker.exe` inside the `Binary` folder and runs without Python installed. There is also a `run.bat` in that folder for a basic demonstration. Anywhere else, the README recommends uv to manage dependencies and to pin a compatible Python version.

The environment comes up with one command:

uv sync --locked

`--locked` matters more than it looks. The `pyproject.toml` pins `requires-python` to `==3.9.*` and pins dependencies exactly, including `numpy==1.21.3`, `onnxruntime==1.9.0` and `pillow==10.2.0`, and the repository has both `.python-version` and a `uv.lock`. Older pins like numpy 1.21.3 are the sort of thing that fails to build on a new Python, which is exactly why the lockfile exists.

Options are discovered the usual way:

uv run facetracker.py --help

And a demonstration run against a video file, which also proves the UDP path works:

uv run facetracker.py --visualize 3 --pnp-points 1 --max-threads 4 -c video.mp4

The README's walkthrough for that last command is to create a new Unity scene, add an empty game object, attach both the `OpenSee` and `OpenSeeShowPoints` components, and run the tracker on a video file while the scene is playing. `--visualize 3` draws the tracker's own overlay so you can see whether tracking is working before you blame the Unity side.

The UDP seam and the field you must copy

The tracker sends tracking data over UDP to the Unity component, and this split is a feature with a stated purpose: tracking can run on a different machine from the one using the data, which helps performance and avoids accidentally revealing camera footage.

The receiving side has one rule that the README repeats because it causes real bugs. UDP packets arrive on a separate thread, so anything using the `trackingData` field on the `OpenSee` component must first copy the field and work from the copy, or the data may be overwritten during processing. The same design means `trackingData` keeps updating even when the `OpenSee` component is disabled.

Two components ship for inspection. `OpenSee` receives the packets and exposes `trackingData`. `OpenSeeShowPoints` visualises the landmark points of a detected face and doubles as a worked example of consuming the component correctly. The README tells you to read it for that reason. The `Examples/` folder holds more, and `Examples/OpenSeeVRMDriver.cs` and `Examples/OpenSeeVRMExpression.cs` are the interesting ones for anyone driving a VRM model, alongside `Examples/OpenSeeWebcamInfo.cs` for enumerating cameras.

Camera selection on the Unity side goes through `OpenSeeLauncher`, which exposes `ListCameras()`, `StartTracker()` and `StopTracker()`. Setting `cameraIndex` to `-1` disables webcam capture. `StartTracker()` shuts down a running instance and starts a fresh one with current settings, and on Windows the component uses WinAPI job objects to make sure the tracker child process dies if your application crashes.

Command line arguments are an array, not a string

One implementation detail in the README is worth repeating because it is an easy mistake. Additional custom command line arguments go into the `commandlineArguments` array one element at a time. The example given is `-v 1`, which must be added as two separate elements, one containing `-v` and one containing `1`, not one element containing both parts.

This is the usual consequence of passing arguments to a child process. Windows `CreateProcess` takes a single command line string that the receiving runtime re-splits using its own rules, and libraries that accept a pre-split array avoid the round trip through string parsing entirely. Getting it wrong produces an argument error that looks like a bug in the tracker rather than in the launcher.

`OpenSeeIKTarget` is mentioned as usable with FinalIK or other inverse kinematics setups, for driving a head target rather than raw landmarks. It was cut off in the README's description, so the specifics of its properties are not something you can learn from the repository alone.

The `facetracker.spec` file at the repository root is the PyInstaller spec, and `make_exe.bat` and `make_exe.sh` are the build scripts for producing the executable that `OpenSeeLauncher` is designed to launch. That is the packaging path the Windows release follows.

Platform notes and a real Linux gotcha

There is a specific Linux failure the README documents, which is the kind of thing that otherwise costs an afternoon. One user reported that Python failed to load the onnxruntime library on Linux. The fix is to run this from within the OpenSeeFace folder:

execstack -c .venv/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.cpython-39-x86_64-linux-gnu.so

`execstack` marks the stack as non executable, which is what the loader needs on distributions that keep it executable by default. The path encodes the Python 3.9 and x86_64 Linux specifics, so if your interpreter or architecture differs you substitute the matching path in `.venv`.

Release history is worth reading for context on the packaging. v1.20.5, published 2026-09-14, switched the build system from poetry to uv, which the release notes call out as making it easier to run the code with the correct Python and dependencies, and added `uv.lock` with dependency hashes as a guard against supply chain attacks. That release is why the uv instructions above are the current ones. v1.20.4, from 2021-09-17, changed the gaze tracking model to always run single threaded, and its release notes carry a warning that the code in the release archive is outdated and will not run on recent numpy versions, which is another argument for running from a checkout with uv rather than from a downloaded archive. The last push to the repository was on 2026-09-18.

Editorial conclusion

OpenSeeFace is the right choice when you need head and landmark tracking on CPU and want the data in your own application, and the wrong choice if you want a finished avatar program with a face. The README says this in its first line and the rest of the design follows from it: the Python process runs separately, the Unity side is a thin receiver, and the seam between them is a UDP packet. The `OpenSee` component's `trackingData` field must be copied before use, because the receive thread can overwrite it mid read, and that is the one piece of the integration that will bite you first. Start with `uv sync --locked` and `uv run facetracker.py --help`, then read `OpenSeeLauncher` before you write any glue code.

Frequently asked questions

Official sources

1. emilianavt/OpenSeeFace on GitHub
2. Issues
3. License: BSD-2-Clause
4. README
5. Releases