DigiHuman: driving a Unity3D character from a webcam through MediaPipe landmarks

Two engines bolted together: MediaPipe feeds the Unity3D scene

DigiHuman is not a general purpose animation tool. It is a narrow pipeline that puts a computer vision model in front of a game engine and lets the engine do the animating. The README describes the arrangement in plain terms: MediaPipe generates 3D landmarks for the human whole body and face, and Unity3D is used to render the final animation after processing the generated landmarks from MediaPipe. The architecture diagram in the repository, stored as a dataFlow.png image, sits between those two halves.

The split matters because the two halves fail in different ways. MediaPipe is the part that estimates where a person is and what their face is doing. Unity is the part that has a character with a skeleton, a set of blendshape weights, and a scene that can be recorded. Digihuman does not attempt to infer animation curves or author keyframes. It relays landmark positions, and the Unity side decides what a given landmark means for a given rig.

The repository topics list the intended terrain: 3d-landmarks, blendshapes, character-animation, facemesh, facial-expression-recognition, mediapipe, mediapipe-hands, pose-estimation, metahumans, and rendering-3d-graphics. The primary language is C#, which matches the Unity project files at the repository root such as Assembly-CSharp-firstpass.csproj and Packages/.

The COCO checkpoint you have to place yourself

There is no single install command for DigiHuman, because the project spans a Python backend and a Unity project. The backend side is a pip install sequence, and the ordering matters because the project specific dependencies live in a requirements file inside the backend directory rather than at the repository root.

Start with the two packages the README names individually:

   pip install mediapipe

That is the landmark generator itself. OpenCV is the second named dependency, and it is installed the same way:

   pip install opencv-python

Then the rest of the backend requirements, installed from inside the backend directory:

   pip install -r requirements.txt

The remaining step is not a command. The README instructs you to download the pre-trained generator model for the COCO dataset and place it into backend/checkpoints/coco_pretrained/. That directory has to exist and hold the model file before the server will do anything useful, and the model is fetched from a Google Drive link rather than from a package registry. This is the part of the setup where people lose time, because nothing in the pip steps pulls it in for you.

Launching the backend and opening the main scene

Once the dependencies and the checkpoint are in place, the backend is started from the backend directory with a single script:

   python server.py

The Unity side has its own prerequisites, and they are not optional. The README asks you to install Unity3D through UnityHub, add and register a license, and install a Unity Editor inside UnityHub. LTS versions and a version higher than 2020.3.25f1 are recommended. One setting matters beyond the rest: in the Unity player settings you have to allow HTTP connections, since the Unity project talks to the backend over the network. The README includes a screenshot of that toggle.

Recording also pulls in two external Unity packages, FFmpegOut and FFmpegOutBinaries, downloaded as .unitypackage files and dragged into the project. The README notes their licences as MIT for the package and GPL for the binaries, which matters because those packages live outside the DigiHuman repository.

The test loop itself is straightforward. Run the backend, open the Unity project, load the main scene at Assets\Scenes\MainScene.unity, and then upload a video to the backend from the Unity project. The README also mentions that you can test the application by selecting provided animations from the right side menu, which is the fastest way to confirm the whole chain works before feeding it your own footage.

Your character needs a Humanoid rig and named blendshape weights

Adding a character is the second half of the work, and the constraints are specific. Characters should have a standard Humanoid rig to show kinematic animations, and for the face they need a facial rig expressed as blendshapes. The README points you at the Unity asset store and at Mixamo for models.

The procedure is a numbered list of Unity editor steps. Open the character setting and set the rig to humanoid. Drag the model into the CharacterChooser/CharacterSlideshow/Parent object in the main scene. Add BlendShapeController and QualityData components to the character object. Then, inside BlendShapeController, add the character's SkinnedMeshRenderer component and find each blendshape weight number under it, entering those numbers into the controller.

That last step is the manual part. Every blendshape index has to be read off the renderer and typed in, and the README's own text contains a typo when it says to set blnedShape weights, which suggests the list was edited in a hurry. A character with a rich facial rig means a long index mapping session before anything renders. DigiHuman is not reading blend shape names, and it is not matching them by convention, so the mapping is entirely on the person doing the setup.

What the sample GIFs leave out

The README presents its results as animated GIFs: hands animations, full body animation under the heading Full body animation, and a Face animation group that includes mouth deform, blinks, and directional mouth motion. These are convincing as a proof of concept, but they are also curated clips. There is no accompanying section on failure behaviour, and no documented discussion of what happens when a limb leaves the camera frame, when two people are in the shot, or when the lighting is poor.

The README also does not document the latency between the camera and the rendered result, does not state a frame rate, and does not list the resolution or format the backend expects for uploaded video. It does not document how recorded video is separated from live camera input beyond the mention of an upload feature. Anyone planning to use this for a livestream or a real time avatar will find no documented answer to the timing question, and the only public evidence of speed is a demo video link.

The repository also has no GitHub releases. There is no changelog, no versioned package, and no tag history to consult, so the state of the project is whatever the default branch happens to contain at the moment you clone it.

Where DigiHuman sits against ordinary Unity animation work

The alternative is the standard Unity route: animate the character by hand with keyframes, motion capture, or a human animator, and treat facial performance as a separate authored problem. The difference in approach is not quality, it is where the work happens. In conventional Unity animation, the artist defines the motion. In DigiHuman, the camera defines the motion and the artist defines the mapping from landmarks to the rig, which is a one time cost per character rather than a per shot cost.

That trade is only interesting when the same character performs many different takes, or when the person being recorded must be preserved exactly. For a single scripted scene, hand keyframing is usually less work than wiring a blendshape index table. The two approaches also differ in failure style. Hand animation degrades into something that looks stiff; automatic animation degrades into tracking error, jitter, and lost limbs, and the project does not offer smoothing, filtering, or cleanup controls to tune that.

MediaPipe itself is the other half of any comparison. It is a general landmark library with many more consumers than this one, and nothing in the DigiHuman repository is a MediaPipe modification. Reading DigiHuman gives you one worked integration, not a fork of the vision stack.

GPL-3.0, one author, and a last push on 2026-05-05

The repository is licensed GPL-3.0, which is a strong copyleft licence with obligations for derivative works that you distribute. The README does not discuss licensing exceptions or a separate licence for the sample assets, and the FFmpegOutBinaries package the setup pulls in carries a GPL notice of its own. Treat that as a flag to review, not a legal opinion.

The project is the B.Sc. thesis of Computer Engineering at Amirkabir University of Technology by a single author, and that context explains both the strengths and the gaps. The architecture is clean and the documentation covers the happy path end to end, but the work stops where a thesis stops. There is no test suite, no CI configuration, and no issue or contribution guidance in the README. The last push was on 2026-05-05, so the branch is not frozen in the sense of an archived repository, but there is no release cadence behind it either, and the absent release list means you cannot pin a known good version.

For a company, that shapes the upgrade cost. Any change in MediaPipe output, any Unity editor version change, and any blendshape index change lands on the person maintaining the integration, with no upstream changelog to read. For a student or a technical artist learning the pipeline, that is an acceptable price for a readable reference implementation.

Editorial conclusion

DigiHuman fits students and technical artists who want to see a MediaPipe to Unity3D landmark pipeline end to end, and it is the wrong tool for a production pipeline that needs pinned versions, recorded releases, or a maintained character rig library. Before adopting it, check that you can place the pre-trained COCO generator model at backend/checkpoints/coco_pretrained/, that your character exposes a Humanoid rig plus blendshapes, and that you are prepared to work inside the project as it stands, with its last push on 2026-05-05.

Frequently asked questions

Official sources

1. Danial-Kord/DigiHuman on GitHub
2. Issues
3. License: GPL-3.0
4. README