modelscope/FunASR: a pipeline toolkit for ASR, VAD, diarization and OpenAI-compatible serving

The gap FunASR fills: chaining speech components without writing the glue

A production transcription pipeline is rarely one model. It is voice activity detection to cut silence, an acoustic model to transcribe, a punctuation model to restore sentence boundaries, and often speaker clustering to attribute each segment. Assembling that by hand means writing the segment bookkeeping, the batching logic and the handoff between components, and redoing it every time you swap a checkpoint.

FunASR's answer is the AutoModel object, which takes the components as named arguments and returns a single structured result. The README's CPU-first example passes a recognizer, a VAD model and a speaker model in one constructor call, and the toolkit runs them in sequence. The audience is engineers building transcription products, call analytics, subtitle generation or voice input, particularly those working in Chinese, where the project's checkpoints and tokenizers are strongest. The topics list also names multilingual ASR, emotion recognition and audio-event detection, so the scope is wider than plain speech-to-text.

How AutoModel wires VAD, SenseVoiceSmall and CAM++ together

The mechanism is visible in the README's second example. AutoModel is constructed with model="iic/SenseVoiceSmall", vad_model="fsmn-vad" and spk_model="cam++". On generate, FSMN-VAD produces speech segments, SenseVoiceSmall transcribes each one, and CAM++ extracts spk_embedding vectors. AutoModel then clusters those embeddings and assigns speaker indices to the VAD segments. The result is read from result[0]["sentence_info"], where each entry carries start, spk and sentence fields.

The README is explicit about two boundaries that matter. First, the speaker labels are local to a recording, not known-person identities, so this is diarization, not identification. Second, the speaker-aware output in that example is a property of the pipeline, not of the checkpoint: the README states these are not native speaker outputs of SenseVoiceSmall. Swap out spk_model and the sentence_info shape changes with it.

The serving layer is separate again. Fun-ASR-Nano can be accelerated with AutoModelVLLM from funasr.auto.auto_model_vllm, which takes a list of audio paths and a language argument. The README also points to an MCP server for Claude and Cursor, an OpenAI-compatible API example for LangChain, Dify and AutoGen, and an OpenClaw realtime plugin. Those are distinct entrypoints over the same checkpoints, and the README warns that support in one adapter does not imply support in another.

Installing FunASR and running a first speaker-aware transcription

The README gives a two-step CPU install from PyPI. torch and torchaudio come first, then the toolkit itself.

pip install torch torchaudio
pip install funasr

For GPU work the README does not pin a wheel for you. It says to install the PyTorch and torchaudio builds matching your NVIDIA driver from pytorch.org before installing FunASR, then confirm the device is visible.

python - <<'PY'
import torch
print(torch.cuda.is_available())
PY

Only use device="cuda" if that prints True. Otherwise stay on CPU or reinstall PyTorch with the correct CUDA wheel. This is the single most common setup failure, and the README treats it as a gate rather than a footnote.

The first real use is the CPU pipeline with five-language ASR plus emotion and audio-event tags. It downloads the checkpoints on first run, so expect a slow start.

from funasr import AutoModel
from funasr.utils.postprocess_utils import rich_transcription_postprocess

model = AutoModel(model="iic/SenseVoiceSmall", vad_model="fsmn-vad", spk_model="cam++", device="cpu")
result = model.generate(
    input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav",
    batch_size_s=300,
)

for seg in result[0]["sentence_info"]:
    print(f"[{seg['start']/1000:.1f}s] Speaker {seg['spk']}: {rich_transcription_postprocess(seg['sentence'])}")

What you should see is one line per VAD segment: a start time in seconds, an anonymous speaker index, and the text with SenseVoice tags stripped by rich_transcription_postprocess. The README states plainly that text and segment boundaries depend on the audio and the checkpoint, so do not compare your output against a fixed transcript.

If you need the larger model instead, the GPU example is a single checkpoint swap. Note that the language coverage is checkpoint-specific.

from funasr import AutoModel

model = AutoModel(model="FunAudioLLM/Fun-ASR-Nano-2512", device="cuda")
result = model.generate(input="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav")
print(result[0]["text"])

For batch throughput, the README shows the vLLM path, which takes a list of files and a language parameter defaulting to auto.

from funasr.auto.auto_model_vllm import AutoModelVLLM

model = AutoModelVLLM(model="FunAudioLLM/Fun-ASR-Nano-2512", tensor_parallel_size=1)
results = model.generate(["audio1.wav", "audio2.wav"], language="auto")

The README links a pinned vLLM setup guide in docs/vllm_guide.md rather than specifying versions inline, which is worth reading before you build a container.

Where FunASR stops being the right tool

The speaker output is the clearest limit. CAM++ produces embeddings and AutoModel clusters them per recording. You get Speaker 0 and Speaker 1, not names, and the indices are not comparable across files. Anything requiring a persistent speaker identity needs a separate enrolment and matching layer that FunASR does not provide.

Language coverage is checkpoint-bound, and the README makes this a warning rather than a detail. SenseVoiceSmall is described as a five-language checkpoint, and Fun-ASR-Nano is listed for Chinese, English, Japanese, and Chinese dialect groups and regional accents. A separate 31-language checkpoint, Fun-ASR-MLT-Nano-2512, exists, and the README says Nano and MLT-Nano should be treated as distinct model choices. If your audio is not covered by the checkpoint you picked, no amount of pipeline configuration fixes it.

The dependency list in setup.py is also worth reading before you commit. A base install pulls in modelscope, huggingface_hub, transformers, hydra-core, omegaconf, umap_learn, oss2 and more. That is a heavy environment for a project you only want for one transcription call. There is a kaldi-native-fbank extra described as a fallback backend used when torchaudio is absent, aimed at Ascend NPU and aarch64 servers with no matching torchaudio wheel, and a silero extra for silero-vad. Those exist because the default path assumes a working torchaudio.

Finally, the toolkit is not the only way to run these models. The README's own quick start opens with a native Transformers path for Fun-ASR-Nano transcription that needs no FunASR toolkit and no remote Python code. If a single model call is all you need, the toolkit is extra surface area.

FunASR versus Whisper-style single-model transcription

The natural comparison is with a single end-to-end model that takes audio and returns text. Whisper-family checkpoints are one model doing one job, and the surrounding work (silence trimming, speaker attribution, punctuation) is left to you or to other libraries.

FunASR's difference is architectural rather than accuracy-based. It treats the pipeline as the unit: VAD first, then recognition, then optional punctuation and speaker clustering, with each stage a replaceable checkpoint. That buys you the ability to run SenseVoiceSmall on CPU with speaker segments, or Fun-ASR-Nano on GPU behind vLLM, without rewriting your application code. It also buys you the constraint described above, that a capability present in one checkpoint or adapter may be absent in another.

A second difference is serving. FunASR ships examples for an OpenAI-compatible API and an MCP server, so the same checkpoints can back a LangChain, Dify or AutoGen integration, or a Claude and Cursor tool call. A bare model checkpoint gives you none of that; you write the server. The trade is that you inherit the toolkit's dependency tree and its runtime-specific setup guides.

Licence, maintenance and upgrade cost

The code is MIT-licensed, which is permissive for commercial use. The repository also carries a separate MODEL_LICENSE file, and the checkpoints are hosted on ModelScope and Hugging Face under their own terms. Code licence and weight licence are different questions, and the repository keeps them in different files for that reason. Read MODEL_LICENSE and the model card for whichever checkpoint you deploy; nothing here is legal advice.

On maintenance, the last push was on 2026-09-10, and the recent release list shows v1.4.15 on 2026-09-09, funasr-onnx 0.4.3 on 2026-09-09, and v1.4.14 on 2026-09-03. The repository is not archived. The presence of a separate funasr-onnx release stream matters for planning: ONNX export and the Python toolkit version independently, so pin both if you depend on exported graphs.

Upgrade cost concentrates in two places. The optional extras in setup.py (knf, train, silero, all) mean the install surface can change between releases, so a requirements lock is worth keeping. And the serving paths have their own guides, including the pinned vLLM setup in docs/vllm_guide.md, so a toolkit bump and a serving-stack bump are separate operations. The README does not document rollback or compatibility guarantees between toolkit and adapter versions, so treat the pinned guide as the contract you actually have.

Editorial conclusion

Adopt FunASR if you need a Chinese-first pipeline that chains VAD, ASR, punctuation and speaker clustering in one AutoModel call, or if you want an OpenAI-compatible endpoint over a self-hosted checkpoint. Do not adopt it if you need stable named-speaker identities, because CAM++ only clusters embeddings into per-recording indices, or if your audio is outside the five languages SenseVoiceSmall covers. Verify three things before committing: that the checkpoint you want is served by the runtime you intend to use, since the README states support in one model or adapter does not imply support in every serving backend; that your PyTorch and torchaudio wheels match your NVIDIA driver, checked with torch.cuda.is_available(); and that your use of the model weights is permitted, because the repository ships a MODEL_LICENSE separate from the MIT licence covering the code.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. modelscope/FunASR on GitHub
4. README
5. Releases