hysts/pytorch_mpiigaze_demo: running MPIIGaze, MPIIFaceGaze and ETH-XGaze from a webcam

What ptgaze actually solves, and for whom

Appearance-based gaze estimation has a reproducibility problem. The papers publish architectures and normalization schemes, but the code that produced the numbers is often a research repository with hardcoded dataset paths. hysts/pytorch_mpiigaze_demo packages the inference half of that work as a pip-installable tool named ptgaze. It takes an image, a video file, or a webcam stream and returns a gaze vector plus a visual overlay, using one of three pretrained models trained on MPIIGaze, MPIIFaceGaze or ETH-XGaze.

The audience is narrow and specific. It is for engineers who want to see whether gaze estimation is good enough for their use case before committing to a dataset download and a training run. It is also for people reproducing published results who need the exact normalization the authors used. The README points at two sibling repositories for training: hysts/pytorch_mpiigaze for MPIIGaze and MPIIFaceGaze, and hysts/pl_gaze_estimation for ETH-XGaze. This repository is the inference demo, not the training code, and that boundary matters when you plan a project.

The pipeline: face detection, normalization, then a ResNet

The data flow is a short chain. A face detector locates the face and its landmarks. Four detectors are selectable: dlib, face_alignment_dlib, face_alignment_sfd and mediapipe, with mediapipe as the default. From the landmarks the code estimates head pose and builds the normalized face image that the model expects. That normalized image goes through a ResNet backbone loaded from a safetensors checkpoint, and the output is a gaze vector in the normalized camera frame, which is then drawn back onto the original frame.

The normalization step is where this repository earns its keep, and the README is unusually candid about it. The camera parameters in ptgaze/data/normalized_camera_params/ do not look like the numbers in the papers, but the README states they are equivalent: the warp depends only on the ratio of focal length to normalized distance, so focal length 1600 with distance 1.0 m at 224x224 is the same warp as 960 with 300 mm at 448x448 after downscaling. The README says these values match OpenGaze, the reference C++ implementation by the paper authors. If you have ever had a reimplementation produce plausible but slightly wrong gaze angles, this is the paragraph to read.

One more detail is documented rather than hidden: the normalized head pose fed to the MPIIGaze model follows the dataset annotation convention, meaning the pitch and yaw of the z-axis of the normalized head rotation matrix. That is a convention choice, not a bug, but it means the MPIIGaze head pose output is not directly comparable to the ETH-XGaze one without checking the frame.

Installing ptgaze and running a first webcam session

The README states the program has only been tested on Ubuntu, so treat that as the supported platform. Python 3.10 or newer is required by pyproject.toml. The install is one command:

pip install ptgaze

On Linux, mediapipe needs the GLES system libraries, which the README gives as `sudo apt install libgles2` on Ubuntu. If you skip that, expect an import or context-creation failure rather than a helpful message. The default mediapipe detector downloads its model file, face_landmarker.task, into ~/.ptgaze/mediapipe/ on first run, so the first invocation needs network access.

The dlib detectors are optional and are not installed by default. dlib ships no prebuilt wheels, so the extra builds it from source and needs CMake and a C++ compiler:

pip install ptgaze[dlib]

With the base install in place, running the demo is a single command. The README's quick start uses the ETH-XGaze model:

ptgaze --mode eth-xgaze

With no --image, --video or --camera argument, the program opens the default webcam and displays the overlaid result in a window. While that window is focused, the keys l, h, t, b and n toggle intermediate visualizations: landmarks, head pose, projected points of the 3D face model, the face bounding box, and the normalized images fed to the model. The n key is the useful one when a prediction looks wrong, because it shows you exactly what the network saw.

To process a file instead, pass --video or --image, and add --output-dir to save the overlaid video. The --ext flag chooses avi or mp4. With --no-screen the video is not displayed and is written to the output directory, which is the mode you want on a headless machine. Note that the README does not document rollback or a way to resume an interrupted batch, so for long videos you are managing that yourself.

Where the demo stops being enough

The most important limitation is stated by the README itself: this program has only been tested on Ubuntu. There is no documented Windows or macOS path. On Linux the mediapipe detector also needs GLES libraries, which means a minimal container without graphics libraries will fail even though nothing about gaze estimation requires a GPU-accelerated GL context.

A second constraint is the checkpoint format. ptgaze loads safetensors checkpoints only. If you trained with hysts/pytorch_mpiigaze you have a .pth file, and you must convert it once with the ptgaze-convert entry point, then point gaze_estimator.checkpoint in your config at the converted file. The converter accepts both the {"model": state_dict} layout and a bare state dict, and with --mode it validates the state dict against the architecture for that mode and records the mode in the file metadata. That validation is a real safeguard, but it also means a checkpoint whose keys do not match the expected architecture is rejected rather than loaded partially.

A third issue is calibration. The --camera flag points at a camera calibration file, and the README links ptgaze/data/calib/sample_params.yaml as the example. That is intrinsic camera calibration, not per-person gaze calibration. The models are trained across many subjects, so they predict a generic gaze direction for a face. If your application needs to know where on a specific screen a specific person is looking, this pipeline gives you a direction vector, not a screen coordinate, and there is no documented personal calibration step to close that gap.

Finally, the config file is all-or-nothing. The README states that when using --config, all other command line arguments are ignored. That is convenient for reproducibility and annoying when you want to override one field for a quick experiment.

Choosing between ptgaze and a training framework

The obvious alternative is not another demo but the training repositories the README names: hysts/pytorch_mpiigaze and hysts/pl_gaze_estimation. The difference in approach is the whole point. ptgaze gives you frozen weights and a fixed preprocessing pipeline; you cannot fine-tune on your own subjects through this interface. The training repositories give you the dataset loaders, the normalization code and the training loop, which is what you need if your faces are not well represented by MPIIGaze or ETH-XGaze, for example if your users wear glasses with heavy frames or work in unusual lighting.

There is also a middle path worth knowing about. Because ptgaze-convert validates a state dict against a named mode's architecture, you can train with the sibling repository, convert the result, and run inference through the same CLI you already scripted. That keeps your evaluation harness identical between the pretrained baseline and your fine-tuned model, which is a real advantage over writing separate inference code for each.

If your goal is a production gaze service rather than an experiment, neither path is a drop-in answer. The demo's window-based interaction model, its dependence on a display for the key toggles, and the absence of any documented server mode mean you would be wrapping a research tool, not deploying a component.

Maintenance, licensing and upgrade cost

The repository is not archived, and the last push was on 2026-07-25, so it is being touched. The release history is uneven in a way worth noting: v0.2.3 and v0.2.4 landed in July and October 2021, and v0.3.0 arrived on 2026-07-05. That is a long gap between minor releases, so the dependency floor in pyproject.toml is the practical thing to check rather than the release cadence. torch is pinned at 2.7.0 or newer, torchvision at 0.22.0 or newer, and mediapipe at 0.10.30 or newer. A modern torch requirement means the install pulls a large wheel and, on CUDA, a matching toolkit.

Upgrade cost is dominated by two things. The safetensors-only policy means older .pth checkpoints need a one-time ptgaze-convert run, and the README does not describe a batch conversion mode, so a directory of checkpoints is a loop you write. The dlib extra adds build time on every clean environment because dlib has no wheels on PyPI, which the pyproject comment states explicitly. If you rely on the dlib or face_alignment_dlib detectors, budget for CMake and a C++ toolchain in your image.

The licence is MIT, declared in the LICENSE file and referenced from pyproject.toml. That is permissive for the code. The pretrained weights are a separate matter: they are hosted on the Hugging Face Hub under the hysts namespace, and the README does not state a licence for the model repositories. If you plan to ship something built on these weights, check the model repository pages rather than assuming the MIT code licence covers them. This is a factual gap in the documentation, not a legal opinion.

Editorial conclusion

Adopt it if you want a working gaze overlay on your own webcam footage within an hour and you are on Ubuntu with a CUDA device or a willingness to run on CPU. Skip it if you need calibrated per-user accuracy, a Windows deployment path, or a service you can run without a display. Verify first that your GPU driver and torch build agree, that libgles2 is present for mediapipe, and that the model weights you need are reachable from the Hugging Face Hub, because the first run downloads them.

Frequently asked questions

Official sources

1. hysts/pytorch_mpiigaze_demo on GitHub
2. Issues
3. License: MIT
4. README
5. Releases