reze-mipo: turning a webcam into MMD bone rotations in the browser

One webcam pass covering body, both hands and face

Reze MiPo, previously known as MiKaPo, turns a webcam into a motion capture rig for MMD models. The pitch is stated in one line in the project README: dance in front of your webcam and watch your MMD character dance with you live in the browser, at 60 FPS, with every take saved as a .vmd file you can open in MMD.

The capture is not split across several models. A single MediaPipe HolisticLandmarker pass runs inside a Web Worker and produces body pose, both hands at 21 points each, and face blendshapes, while the renderer holds 60 FPS. Splitting the work this way matters because detection and rendering compete for the same frame budget otherwise.

The output targets are real MMD data rather than an approximation. The README states that 33 pose points become per-bone quaternions in each bone's parent-local frame, including finger and forearm twist, and that face blendshapes are mapped onto the morphs a model already ships with, including the blink, vowel and wink morphs, while gaze drives the left and right eye bones. So the result is a file MMD understands, not a format you need a converter to reach.

The project is one part of a wider family. The README positions Reze MiPo as the real-time motion capture piece, alongside reze-engine for the WebGPU character rendering and physics, reze-design for scene work, reze-studio for animation editing on a timeline, and reze-rig for retargeting FBX animations into MMD format. The interesting dependency for an implementer is the first one, since the renderer and the solver agree on the coordinate conventions.

The solver lives in the gap between MediaPipe points and MMD bones

The core problem is a coordinate mismatch, and the README describes it precisely. MediaPipe gives 3D points around a hip-centred origin. MMD wants a quaternion per bone, expressed in that bone's parent-local frame, on a skeleton whose rest pose belongs to whoever modelled the character. Everything the project does happens in that gap.

The implementation is a single generic pass over a bone table. Each row describes a bone, its parent, where its direction comes from in the pose data and which landmarks define that direction. A direction bone names a source joint and a target joint, and the solver computes the world-space segment between them, rotates it into the parent-local frame, then converts the rest reference direction into the live direction as a quaternion. The README notes that rotating into parent-local space is a quaternion conjugation rather than a matrix build, and that the frame allocates nothing.

Five steps run per frame. Calibration happens once per model, at rest, where every parent chain is identity and the world-space parent-to-child direction is the parent-local reference. Solving walks bones in hierarchy order so each parent chain is computed once. Smoothing applies a One-Euro filter on media time. The body is then cleared of the model's own collision capsules, and placement walks both legs from the hips and drops the body until the lower foot rests where it rests in the bind pose.

The hierarchy ordering is worth dwelling on, because it is what allows the same table to work on any model. Nothing in the bone table hard-codes a model's dimensions; the rest pose is read from the model at load time.

Calibration to any PMX model at load, with no config and no T-pose

Drop a PMX folder into the page and the solver calibrates itself to that model's rest pose on load. The README is blunt about the result: any model, no config, no T-pose.

This is the difference between the project and a script that maps landmarks onto a fixed bone list. Because the reference directions are read from the loaded model rather than assumed, a model with long arms, short arms or a different shoulder width produces correct rotations. A T-pose requirement would also make the tool unusable for most existing MMD models, since those are authored in whatever pose their creator preferred.

The rest of the model is used too, not just the skeleton. The model's own physics capsules define where an arm may go, so an arm that ends up inside the chest swings out at the shoulder. The README describes this as rotation only, so the arm still points where the landmarks put it while avoiding contact with the torso.

Two placement rules carry visible consequences. The centre bone carries height, so crouches and level changes read as the character moving vertically while holding its mark, which is the base an animator wants to key over. And the body rests on its lowest part, taking the lower foot, so a standing split measures against the standing foot, while a hip-clearance candidate in the same maximum means a body rolling on the floor rests on its hips.

Trunk motion is spread across three joints. Chest rotation splits evenly across the two upper-body bones so the spine curves, the lower-body bone takes a quarter of the mean thigh pitch, which the README credits with making sitting look like sitting, and shoulder-line tilt reaches the shoulder bones so a shrug actually shrugs.

Filtering is where the shimmer and the wobble go away

Three separate filtering decisions are described, each aimed at a different artefact.

The first is time-domain smoothing with a One-Euro filter running on media time, driven by true angular velocity and bounded by what a limb could physically have done since the last frame. Because the bound is physical rather than statistical, a fast movement is not flattened the way a fixed smoothing window would flatten it.

The second targets twist, not swing. Roll about a bone's own axis is where landmark noise concentrates, and the README makes the useful observation that roll moves no bone direction, so a landmark preview can never show it. Arms and thighs are therefore split into swing and twist, with the twist placed on a much calmer filter, which is what stops the character shimmering.

The third is the z channel. x and y come off the image, while z is inferred and carries the noise. Low-passing that single channel before solving took torso wobble from 6.7 degrees per frame down to 0.3, according to the README.

Roll itself is reconstructed rather than filtered. Every arm and thigh names a child segment as a witness and builds full rest and live orthonormal bases from it, weighted by how observable the roll is at that bend angle. Thighs keep a second witness in reserve: when the knee is straight, where the toes point indicates how the femur is rotated.

Detection gaps are handled by timing rather than by guessing. A bone holds its last measurement for 250ms, eases to rest over 500ms, and eases back in over 250ms, so an ordinary gap in detection costs nothing visually. Hands are gated more strictly, since hand landmarks collapse toward the origin when tracking fails while still reporting 21 structurally perfect points. The hand's own span and the pose model's wrist visibility gate them, and a hand only engages after a full second of continuous tracking.

A Web Worker for detection and WebGPU for rendering

The stack is three pieces. Detection is MediaPipe HolisticLandmarker running in a Web Worker. Rendering is reze-engine, a custom WebGPU MMD renderer with anime-character rendering and physics and no dependencies. The framework is Next.js 16 with Tailwind v4 and shadcn/ui.

The repository metadata shows the dependency set behind that. The MediaPipe package is @mediapipe/tasks-vision, the renderer is the npm package reze-engine, and the framework versions are Next 16 with React 19 and TypeScript 5. The project also depends on @radix-ui packages for labels, radio groups, scroll areas, slots and tooltips, plus lucide-react for icons and next-themes for theming.

The renderer is where the 60 FPS claim comes from, and it also means the browser requirement is real. WebGPU support is a precondition for running the app as intended, and the detection side needs a camera. On a machine without either, the project is not usable in the way its documentation describes.

Running the application locally takes two commands, and the development server then serves on port 4000:

npm install
npm run dev

There is a harness for offline analysis. The package scripts include a harness entry point that bundles harness/analyze.mts with esbuild and runs the result on Node, and a separate harness:record script that runs node harness/record.mjs. The harness directory at the repository root is separate from src, which suggests the analysis tooling is kept apart from the application code.

Output is a .vmd file, written frame by frame at 30 fps

The output format is the practical reason to use this project. The README states that a video steps frame by frame at 30 fps, producing a result that is identical on any machine, and that a photo saves as a one-frame pose. A .vmd file is what MMD consumes directly, so the capture loop ends in the editor rather than in another conversion step.

The determinism claim deserves attention. Recording at a fixed 30 fps means the exported file does not depend on how fast the browser happened to run during capture, so the same take exports the same way on a laptop and on a desktop. That is a design decision with a cost: a 60 FPS capture is resampled down to 30, so fast motion loses half its temporal samples. If you are animating at 60, the project does not currently export at 60.

The README also states that your model and your footage survive a refresh, which matters for long recording sessions where an accidental reload would otherwise cost the take.

The demo is hosted at mipo.reze.one, and the README points to it as the place to try the application without setting anything up locally. A camera permission prompt is required, and the calibration step runs against the model you load, so the very first thing worth watching on the hosted page is whether the character settles into its rest pose after load.

Releases moved fast, and the licence is GPL-3.0

The versioning history shows a project moving quickly under a changing name. v3.0.0, titled MiKaPo 3.0 with 60 FPS real-time MMD motion capture, was published on 2026-07-21. v4.0.0 followed on 2026-08-04, and v5.0.0, named Reze MiPo 5.0, was published on 2026-08-28. The last push to the repository was on 2026-10-12, and the repository is not archived.

Three major versions in about six weeks is a fast cadence for a project that depends on a renderer package versioned separately at 0.55.2. The practical consequence is that examples and integration code written against v3 or v4 may not line up with v5, and the rename from MiKaPo to Reze MiPo means older documentation and third-party references use the old name.

The licence is GPL-3.0, with the LICENSE file at the repository root. For a personal or research use case that constraint is rarely the deciding factor. If you intend to embed the solver in a closed source product, the copyleft terms of GPL-3.0 are the thing to examine first, since that is a distribution question rather than a technical one.

The repository itself is a private package. The package.json declares the name mikapo at version 5.0.0 with private set to true, which matches a Next.js application rather than a published library. The dev script is next dev -p 4000 and the build script is next build.

Editorial conclusion

reze-mipo suits MMD authors and motion hobbyists who record dance or pose footage and want a.vmd result without hardware, and for anyone building on top of a WebGPU MMD renderer since the solver runs entirely client side. It is the wrong tool when you need sub-centimetre positional accuracy, or perfectly calibrated articulation on models whose rest pose differs drastically from the poses the solver was validated on. Before adopting it, load your own PMX folder and watch the calibration pass, check that your browser supports the WebGPU path the renderer needs, and expect the hand tracking to engage only after a full second of continuous detection.

Frequently asked questions

Official sources

1. AmyangXYZ/reze-mipo on GitHub
2. License: GPL-3.0
3. Project website
4. README
5. Releases