FoleyCrafter generates sound effects that line up with the picture

Adding sound to footage that was shot without it

FoleyCrafter is a video-to-audio generation framework which can produce realistic sound effects semantically relevant and synchronized with videos. The framing in the project description is about the unsung work of cinema and gaming, the sound effects that enhance realism, impact and emotional depth for an audiovisual experience.

The problem it addresses is concrete. Video is often captured without usable sound, or with sound that has to be replaced, and recording foley for every scene is expensive and slow. FoleyCrafter generates the effect instead, from the visual content of the clip, so the sound is both appropriate to what is on screen and lined up with the moment it happens.

The project is part of the OpenMMLab organisation and the paper is titled FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds, with an arXiv identifier of 2407.01494 and an IJCV 2026 note in the repository description. The authors are Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu and Kai Chen, the last two and Yanhong Zeng being corresponding authors.

The public artefacts are a project page at foleycrafter.github.io, a model on Hugging Face, a Hugging Face Space for the demo, an OpenXLab application, and the code in this repository. The licence is Apache-2.0, the primary language is Python, and the last push was on 2026-06-15 with no GitHub releases published.

A diffusion base model plus four adapters

The mechanics are visible in the checkpoint layout, and they explain the dependency structure.

The audio generation is built on a text-to-audio base model, and the documentation names it: the checkpoints for the base model are cloned from the Auffusion repository, and the project says it uses Auffusion. On top of that base, the FoleyCrafter checkpoint set contains a semantic adapter, a vocoder with its own configuration, a temporal adapter, and a timestamp detector.

The division is easy to read. The semantic adapter is what ties the video content to the acoustic space, so a visible object maps to the right class of sound. The vocoder turns the model's latent output into a waveform. The temporal adapter is the part that makes the result synchronized rather than merely plausible, since it has to place the sound in time against the visual events. The timestamp detector is the mechanism that finds where in the clip something happens, which is what lets a footstep land on the frame where the foot lands rather than somewhere nearby.

The consequence for installation is that this is not a single self-contained checkpoint. You need the base model as well as the FoleyCrafter weights, and the two have to be placed in the expected directory structure under a checkpoints folder.

The repository itself is small and conventional for a research release: a foleycrafter/ package directory, app.py for the Gradio interface, inference.py for batch generation, an examples/ directory, assets/, a requirements/ directory, and a pyproject.toml that carries only ruff configuration for linting and formatting.

The conda environment and git-lfs

The setup is a conda environment, and the commands are given in full:

# install conda environment
conda env create -f requirements/environment.yaml
conda activate foleycrafter

# install GIT LFS for checkpoints download
conda install git-lfs
git lfs install

The git-lfs step is not optional decoration. The checkpoints are stored in the Hugging Face repositories as large files that git-lfs manages, and without it the clone produces pointer files rather than weights, and the inference script has nothing to load. The environment file itself is in the requirements/ directory of the repository, so the pinned dependency set comes from the project rather than from a list you assemble yourself.

After the environment is ready, the checkpoints are downloaded automatically by running inference.py. That is the shortest path, and the documentation presents the manual route as the alternative for people who want to control where the weights live, for example on a shared machine where the default path is not writable.

The manual route clones two repositories into the checkpoints directory. The base model first:

git clone https://huggingface.co/auffusion/auffusion-full-no-adapter checkpoints/auffusion

and then FoleyCrafter itself:

git clone https://huggingface.co/ymzhang319/FoleyCrafter checkpoints/

The documentation then spells out the expected directory layout, with a semantic subdirectory holding the semantic adapter, a vocoder subdirectory holding the vocoder weights and its config, and the temporal adapter and timestamp detector at the top level of the checkpoints folder. The naming has to match, since the code looks for these paths by name.

Running inference and the browser demo

Generation is a single script, and the output directory is a command line argument:

python inference.py --save_dir=output/sora/

The example in the documentation points the output at a directory named for one of the example sets, and the results table in the README pairs input videos with generated audio so you can hear the difference.

For people who do not want to touch a command line, there is a Gradio interface:

python app.py --share

The share flag is what produces a public link to the interface, which is convenient for a quick demonstration and worth understanding before you use it on anything private, since a shared Gradio session is reachable by anyone holding the URL.

The same interface is available without any of this setup through the Hugging Face Space and the OpenXLab application linked from the project page, which is the fastest way to judge whether the output quality is good enough for your footage before committing to a local environment.

The two installation routes, hosted space and local conda environment, differ in what you can change. The hosted space runs the released model as it is. The local setup lets you swap checkpoints, run the example sets yourself, and integrate the generation into a script, which is what a pipeline needs.

The examples directory shows what the model was tuned for

The examples/ directory is the most informative part of the repository, because the names describe the situations the model was prepared for.

There are five sets: PromptControl, avsync, gen3, sora and vggsound. Two of those names point directly at capabilities. PromptControl is about steering the generation with a prompt rather than letting the model decide alone, which is how you get a specific kind of sound when the visual cue is ambiguous. avsync refers to audio synchronisation, the property the model is named for.

The other three are more about data and provenance. gen3 and sora are named after video generation systems whose footage appears in the demonstrations, and vggsound refers to the VGGSound audio dataset, which is a large collection of video with sound effect labels and the natural training source for a system that has to learn which visual events map to which sounds.

That list also tells you where the model is likely to be weakest. A model trained to associate visible events with sound effects will handle clear cases well, a door closing, a dog barking, rain starting, and it will have less to go on when the sound that belongs to a scene is off screen or implied by context rather than shown.

The assets/ directory holds the project images, including the logo used at the top of the README, and the .pre-commit-config.yaml shows the project runs pre-commit hooks on contributions.

Inference is public, training is not

The roadmap in the README is short and one line matters a great deal.

The checklist has three items. Release the model and code of FoleyCrafter, which is marked as done on 2024/07/01. Release the training code, which is not ticked. And a more powerful model, also not ticked. The release note is dated 2024/07/01, which is when the model and code were published, and the last push to the repository was on 2026-06-15.

So the situation is: inference is public, the weights are public, the training code is not, and the model you get is the one that was released in July 2024. Anyone who wants to fine-tune the model on their own footage, or to train on a different domain of sound, cannot start from this repository.

There are no GitHub releases either, which is consistent with how a research release is distributed. The checkpoints come from Hugging Face through git-lfs clones, the model page is on Hugging Face, and the demos run in a Space. The package is licensed Apache-2.0, which is the permissive choice a company making its weights available would make, and the paper and arXiv entry are the formal record of the method.

Where this fits against a sound library

The alternative to generating sound effects is not to generate them, it is to use a recorded library, and the difference in approach is worth stating plainly.

A recorded library gives you a human performance of every sound you might need, cut and ready, with consistent loudness and no processing artefacts. FoleyCrafter gives you a sound synthesised for the specific frame, which is the point of the synchronisation claim. For a shot where a machine thumps once, a generated effect can land exactly on the impact, and no library recording can be trimmed to the millisecond of the visual event without hand work.

The cost of that is control. A library recording is finished audio, with an engineer having chosen the perspective, the decay and the room. A generated effect is the model's opinion of what that frame sounds like, and it may be right about the timing and wrong about the character, or the other way round. The documentation's own framing, that it produces realistic sound effects semantically relevant and synchronized, contains both claims, and a team that needs the second more than the first is the intended user.

The practical path for evaluating it is the hosted Space, then the local environment for the example sets, and only then a decision about whether a generative step belongs in an existing foley workflow. The tooling here is a research release, so a production pipeline should expect to wrap it rather than depend on it.

Editorial conclusion

FoleyCrafter suits a video team that needs placeholder or production sound effects for footage shot without audio, and it is the wrong tool for anyone who needs to train or fine-tune the model, because the training code is still listed as an unreleased item in the project roadmap. Before adopting it, check that the released checkpoints match the model you need, and plan for the fact that the weights arrive through git-lfs, which makes the first setup considerably larger than a normal pip install.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. open-mmlab/FoleyCrafter on GitHub
4. Project website
5. README