audio.cpp: a native C++ runtime for TTS, STT and voice conversion
An all-in-one, pure C++ inference engine for audio models, powered by ggml. Supports TTS, STT, VAD, voice conversion, music generation, and more, with highly optimized performance. No Python dependency.
At a glance
- What is it?
- audio.cpp puts TTS, ASR, VAD, voice conversion and music generation behind one ggml-based C++ runtime with no Python dependency. The trade-off is a fast-moving model surface and a licence file that needs reading before you ship.
- Who is it for?
- audio.cpp fits teams that already accept native build tooling and want one runtime serving several audio tasks, and it is a poor fit if you need one narrow, stable ASR binary with a settled API. Before adopting, read the LICENSE file at the repository root, which GitHub reports as NOASSERTION, and confirm that the model family you need appears in the supported model table rather than only in a release note.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 3 days ago.
- What is it written in?
- Mainly C++, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem audio.cpp addresses: one native runtime instead of many Python environments
The README opens with the complaint it is answering: juggling a dozen Conda environments, hundreds of Python packages, and dependency conflicts just to try a few audio models. That is a real pattern in audio work. A text-to-speech checkpoint, an ASR checkpoint and a voice-conversion checkpoint usually arrive as separate repositories, each pinning its own PyTorch build, its own audio I/O library and its own sampling code. The cost is not the model weights. It is the environment around them.
audio.cpp replaces those per-model paths with a shared native runtime built on ggml. The README describes it as a framework for real end-to-end execution rather than one-off model demos, and lists the same runtime serving TTS, voice cloning, voice conversion, ASR, diarization, VAD, source separation, alignment and codec-style models. The audience is therefore narrower than anyone who wants speech recognition. It is engineers who are willing to compile C++ and want one binary surface, one backend story and one packaging format across several audio tasks.
How the runtime is structured: ggml backends, GGUF packages and shared CLI and server entry points
The architecture visible in the repository is a layered one. ggml sits at the bottom as the tensor and compute layer. Above it, the README lists CUDA, HIP/ROCm, Vulkan, Metal and CPU backends behind shared CLI and server entry points, so the same task can move between an NVIDIA box, an AMD box, an Apple Silicon laptop and a CPU-only machine without a different Python stack per target. The top-level tree reflects that split: src/ and include/ for the engine, app/ and tools/ for the executables, webui/ for the browser interface, model_specs/ for model definitions, and external/ for the vendored ggml submodule.
Model weights arrive as GGUF packages. The README points to a Hugging Face repository at audio-cpp/audio.cpp-gguf and a ModelScope mirror, and states that all released model families support GGUF loading. It reports that tested Q8 packages can run up to 1.53x faster while reducing peak VRAM by up to about 37 percent on routes such as Higgs Audio, Fish Audio and Voxtral, with the measurements in docs/reports/gguf_q8_performance.md and support status in docs/gguf.md. Those numbers come from the project's own reports, not from an independent run.
One design choice worth flagging: the model surface is large and explicitly moving. Release 0.7 brought the count to 62 model families and 85+ variants; release 0.6 took it to 49 families. The README also carries a dev-branch note for Yue2 native song generation, described as available for community testing. A framework that adds families at that rate will have uneven maturity across them, and the supported model table is the document that tells you which side of that line your target sits on.
Building audio.cpp and running a first inference path
The repository root contains CMakeLists.txt, a flake.nix and flake.lock for Nix users, and Makefile.windows, which tells you the intended entry points. The README does not print a canonical build command in the text available here, so treat the CMake file as the source of truth for options rather than copying a flag from a blog post. The configure and build steps are the standard CMake pair:
cmake -B build
cmake --build buildWhat you get depends on which backends the configure step found. The README states that CUDA, HIP/ROCm, Vulkan, Metal and CPU are all supported behind the same entry points, so the build is where you decide whether this machine runs on GPU or falls back to CPU.
For container users, examples/docker/ is the place to look, and the repository ships a .dockerignore at the root. A Docker path matters here because the whole pitch is avoiding per-model Python environments; if you containerise the runtime, the image becomes the environment.
Once built, the first useful exercise is not a benchmark but a comparison. The README describes an Arena tab in the WebUI that takes one shared input, queues multiple models or GGUF variants for TTS, voice conversion and ASR, and lets you review outputs with metrics. That is the fastest way to see whether a given GGUF package behaves acceptably on your audio before you wire anything into a pipeline. The README also mentions experimental JSON pipeline support for multi-step workflows, and examples/voice_dir_example.json is the sample file to read if you want to see the shape of that format.
Where audio.cpp is the wrong tool
The licence is the first constraint. The repository has a LICENSE file, but the metadata GitHub reports is NOASSERTION, meaning the platform could not map it to a recognised licence identifier. Nothing in the README clarifies the terms. If you are shipping a product, read that file directly and get your own answer before you build a dependency on it.
The second constraint is scope. If your entire requirement is transcribing files with one Whisper-class model, audio.cpp is a large dependency for a small job. You would be compiling a framework whose value comes from covering TTS, voice conversion, diarization, source separation and alignment as well, and you would be tracking a project that added thirteen model families in a single release cycle. A narrower, single-purpose binary has a smaller surface to break.
The third is the dev branch. The README states that Yue2 native song generation is on the dev branch for community testing, with dev binaries from CI artifacts. Anything on that branch is explicitly not a stable target, and the README's own contribution guidance asks new model ports to start under the community models surface with reproducible validation. That is a reasonable policy, and it also tells you that parts of the model catalogue are less exercised than the headline families.
Finally, rollback, version pinning and upgrade compatibility between releases are not documented in the README. If your deployment needs a documented downgrade path, you will have to derive it from the release history yourself.
Alternatives and the real difference in approach
whisper.cpp is the closest point of comparison for speech recognition, and the difference is architectural rather than cosmetic. whisper.cpp is a single-model-family runtime: it exists to run Whisper efficiently in C++ on top of ggml. audio.cpp uses the same underlying tensor library but treats Whisper-class ASR as one path among many, sharing components across TTS, voice conversion, VAD and music generation. If you need only Whisper, whisper.cpp is the smaller thing to own. If you need Whisper plus a TTS model plus a VAD stage in the same process, audio.cpp is designed for that combination, and whisper.cpp is not.
The other alternative is the Python reference stack itself. The project's own positioning is that Python paths are the thing being replaced, and its performance claims are framed as comparisons against those Python reference paths: the README reports multiple TTS paths running 1.8x to up to 8x faster than their Python reference paths with end-to-end latency cut by 45 to 85 percent on CUDA. Those are the project's figures. The honest counterpoint is that the Python ecosystem still has more models, more tutorials and more people who can debug it at 2am. Choosing audio.cpp means trading that ecosystem for a native runtime and a GGUF packaging format.
Maintenance, upgrades and what the release cadence costs you
The last push to the default branch was on 2026-09-15, and the most recent releases are v0.7.4 on 2026-09-13, v0.7.3 on 2026-09-07 and v0.7.2 on 2026-09-04. Three patch releases inside ten days is a fast cadence, and the release notes describe substantive additions rather than fixes alone: v0.7.4 added VibeVoice ASR Streaming 7B, Irodori-TTS v4.1 Anime, Moonshine Streaming ASR and Kokoro 82M.
That pace has a cost that the README does not address. Nothing in the repository documentation describes a long-term support branch, a deprecation policy, or how a model spec changes between minor versions. If you pin to v0.7.4 and a model you depend on is revised in v0.8, the upgrade path is your problem to work out from the diff. The GGUF packaging helps here, because weights and runtime are separated, but the model_specs/ entries that describe how a package is loaded are part of the repository and move with it.
On licensing, the practical implication is narrow and worth stating plainly: because GitHub reports NOASSERTION, you cannot infer the terms from a familiar identifier in the repository metadata. Redistribution rights for a product that embeds this runtime, and the separate question of the licences attached to individual model weights in the GGUF repositories, both need to be checked against the actual files rather than assumed.
Editorial conclusion
audio.cpp fits teams that already accept native build tooling and want one runtime serving several audio tasks, and it is a poor fit if you need one narrow, stable ASR binary with a settled API. Before adopting, read the LICENSE file at the repository root, which GitHub reports as NOASSERTION, and confirm that the model family you need appears in the supported model table rather than only in a release note.
Frequently asked questions
What is audio.cpp?
It is a C++ audio inference framework built on ggml that runs TTS, voice cloning, voice conversion, ASR, diarization, VAD, source separation and alignment through a shared runtime. The README describes it as built for real end-to-end execution rather than one-off model demos, with no Python dependency.
What are some good audio libraries for C++?
audio.cpp is one option, and its distinguishing feature is that it wraps ggml for model inference rather than providing general audio I/O or DSP primitives. The README also lists built-in denoise, enhancement, resampling and STFT/ISTFT utilities, so some signal-processing work is covered alongside inference.
What is Whisper CPP used for?
This repository does not describe whisper.cpp, so this article cannot answer that. audio.cpp is a separate project that covers ASR alongside TTS, voice conversion and other audio tasks on top of ggml.
Community notes