# ComfyUI_Sonic: portrait animation driven by audio in ComfyUI

> ComfyUI_Sonic is a ComfyUI custom node that wraps the Sonic research method, generating a talking portrait video from an image and an audio clip using Stable Video Diffusion for the frames. It suits someone experimenting with audio-driven portrait animation inside ComfyUI, and the node carries a set of hardware workarounds for bf16, 12GB cards and Apple MPS.

**smthemex/ComfyUI_Sonic** — Sonic is a method about ' Shifting Focus to Global Audio Perception in Portrait Animation',you can use it in comfyUI

- Repository: https://github.com/smthemex/ComfyUI_Sonic
- Stars: 1,142 · Forks: 110
- Language: Python
- License: MIT
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/smthemex-comfyui-sonic

## What the node does

Sonic is a method about shifting focus to global audio perception in portrait animation, and this repository makes it usable inside ComfyUI. The description is that you can use it in ComfyUI, and the node is a custom node that takes a still portrait and an audio clip and produces an animated video of the portrait speaking. The underlying method is the research work published as an arXiv preprint, with a second paper on real-time audio-driven face generation cited alongside it, so the node is a wrapper around someone else's method rather than an independent implementation. The primary language is Python, the licence is MIT, the last push was on 2026-10-05, and the repository has no GitHub releases, so the version marker is the package version of 1.0.0 in the pyproject file. The pyproject also carries the Comfy Registry metadata, with a publisher id and a display name of ComfyUI_Sonic, which is how the node appears in the registry.

## The fork notice, and the dependency fix behind it

The README opens with a fork notice that explains a real dependency problem. The default branch is `fix/torchaudio-soundfile`, and the change replaces `torchaudio.save` with `soundfile.write` in order to sidestep the `torchcodec` ABI mismatch on torch 2.11 with the cu130 build. The `IMPROVEMENTS.md` file is said to hold the full list of downstream changes and the rationale for them. This is worth understanding rather than treating as noise, because an ABI mismatch between torch and a native codec library presents as a crash at the moment you write a file rather than at import time, and the symptom gives no hint that the audio library is at fault. A fork that changes one call for that reason is a fork doing exactly what a fork should, and the change is narrow enough to review.

## Installing the node and the requirements

The installation is the standard ComfyUI custom node procedure. In the `./ComfyUI/custom_node` directory you run the clone:

```bash
git clone https://github.com/smthemex/ComfyUI_Sonic.git
```

Then the requirements are installed with:

```bash
pip install -r requirements.txt
```

The requirements file lists diffusers 0.29.0 or newer, opencv-python, torch, torchaudio, torchvision, transformers, imageio, imageio-ffmpeg, omegaconf, tqdm, librosa and einops, with several pinned versions commented out at the bottom of the file, including torch 2.2.1, diffusers 0.29.0 and librosa 0.10.2.post1. The commented pins are a useful signal, since they record the versions the node was developed against even though the install leaves them loose.

## The model layout, and the SVD checkpoint

The models are not managed by ComfyUI's usual mechanism, and the README spells out the directory structure you have to create. The Sonic models go under `ComfyUI/models/sonic/` and consist of `audio2bucket.pth`, `audio2token.pth`, `unet.pth`, `yoloface_v5m.pt`, a `whisper-tiny/` directory with `config.json`, `model.safetensors` and `preprocessor_config.json`, and a `RIFE/` directory with `flownet.pkl`. The whisper-tiny model comes from openai on Hugging Face, and the rest from a Google Drive folder. Separately, the Stable Video Diffusion checkpoint goes into `ComfyUI/models/checkpoints` as either `svd_xt.safetensors` or `svd_xt_1_1.safetensors`, from the two stable-video-diffusion repos. The `RIFE` entry and the `yoloface` file tell you what the pipeline does, with a face detection model to locate the face and RIFE for frame interpolation between the generated frames.

## The hardware fixes, which are the most useful part of the README

The update section lists four fixes, and they are all about making the node run on real hardware rather than about the method. Some machines must use `cuda:0` and error otherwise, which is a device selection fix. The bf16 error is fixed, so the node runs in half precision where it previously failed. The 12GB VRAM out-of-memory on first run is fixed, which matters because the model loading is the peak memory moment. And the MPS device error is fixed, adding Mac support. The later update notes explain the memory knob. The `image_size` parameter controls the minimum size of the output image, and if you get an out-of-memory error you should reduce that value, and the same note says non-square image output is supported but is easier to run out of memory on. There is also a change replacing the frame number option with a duration option, measured in seconds, for controlling the length of the generated audio, with the caveat that the comparison is against the audio amplitude array and is not a precise percentage.

## The repository layout, and the separate Gradio app

The tree is a ComfyUI node plus the research code it wraps. There is `__init__.py` at the root, which is how ComfyUI discovers the node, and `sonic_node.py` as the node definition, with `sonic.py` holding the model and inference code and `node_utils.py` for the helpers. There is a `gradio_app.py` for running the same thing as a standalone web interface, and a `demo.py` which is a script entry point. The `svd_repo/` directory holds the Stable Video Diffusion code, `src/` holds the Sonic implementation, `config/` the configuration, `tests/` the tests, and `example_workflows/` the ComfyUI workflow JSON files that the README links to as examples, with a newer one and two older ones. The `examples/` directory holds sample inputs, with an `image/` and a `wav/` folder.

## Conclusion

Adopt ComfyUI_Sonic if you want to try audio-driven portrait animation inside ComfyUI rather than running the research code as a standalone script. Skip it if you need a production pipeline, since this is a custom node that expects a specific model layout under `ComfyUI/models/sonic/`, and expect to move the checkpoints yourself. Before you run it, note that the node is a fork with a documented change replacing `torchaudio.save` with `soundfile.write` to sidestep a `torchcodec` ABI mismatch on newer torch builds, and that the memory guidance in the README says to lower `image_size` if you hit an out-of-memory error.

## FAQ

### What does the ComfyUI_Sonic node do?

It applies the Sonic method, shifting focus to global audio perception in portrait animation, inside ComfyUI, generating an animated portrait video from an image and an audio clip. It uses Stable Video Diffusion for the video frames and RIFE for interpolation.

### How do I install the ComfyUI_Sonic node?

Clone it into the ./ComfyUI/custom_node directory with `git clone https://github.com/smthemex/ComfyUI_Sonic.git`, then run `pip install -r requirements.txt` for the dependencies, which include diffusers 0.29.0 or newer, torch, torchaudio, torchvision, transformers, imageio, omegaconf, librosa and einops.

### Where do the ComfyUI_Sonic models go?

The Sonic checkpoints go under ComfyUI/models/sonic/ as audio2bucket.pth, audio2token.pth, unet.pth, yoloface_v5m.pt, a whisper-tiny/ directory and RIFE/flownet.pkl. The Stable Video Diffusion checkpoint goes into ComfyUI/models/checkpoints as svd_xt.safetensors or svd_xt_1_1.safetensors.

### What is the difference in the ComfyUI_Sonic fork?

The default branch fix/torchaudio-soundfile replaces torchaudio.save with soundfile.write to sidestep the torchcodec ABI mismatch on torch 2.11 with cu130. The full list of downstream changes is in IMPROVEMENTS.md.

## Sources

- [Issues](https://github.com/smthemex/ComfyUI_Sonic/issues)
- [License: MIT](https://github.com/smthemex/ComfyUI_Sonic/blob/main/LICENSE)
- [README](https://github.com/smthemex/ComfyUI_Sonic/blob/main/README.md)
- [smthemex/ComfyUI_Sonic on GitHub](https://github.com/smthemex/ComfyUI_Sonic)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/smthemex-comfyui-sonic