ABexit/ASR-LLM-TTS: A Script-Collection Voice Loop With Three TTS Backends

The problem: a local voice loop where the TTS choice decides whether it feels live

A speech assistant demo is easy to draw on a whiteboard and awkward to assemble. You need an ASR model that handles Chinese, a language model small enough to run on the same GPU, and a synthesiser, plus the glue that decides when the user has stopped talking. This repository assembles that glue around three existing open source models rather than training anything. SenseVoice, from the FunASR project, does recognition. Qwen2.5 in 0.5B and 1.5B variants does the conversation. Synthesis is where the project makes its actual argument: CosyVoice, Edge-TTS and pyttsx3 are all wired in, and the README is explicit about why. CosyVoice inference is described as slow enough to seriously affect conversational real-time behaviour, so Edge-TTS and pyttsx3 were added as faster alternatives. The intended audience is someone with a CUDA-capable machine, a conda install, and a willingness to edit line numbers in a script. There is no package to install, no CLI, and no configuration file. The unit of delivery is a numbered .py file.

How the pipeline is wired: SenseVoice, then Qwen2.5, then one of three synthesisers

The main code is derived from CosyVoice, and SenseVoice is added on top of it as the recognition stage, with Qwen2.5 inserted as the dialogue model. So the flow is audio in, text out of SenseVoice, text into Qwen2.5, generated reply into the synthesiser, audio out. Model selection happens by editing source lines rather than passing arguments: the README points at line 215 for the SenseVoice model directory, where setting model_dir = "iic/SenseVoiceSmall" triggers automatic download, and line 220 for the language model, where model_name = "Qwen/Qwen2.5-1.5B-Instruct" is the automatic-download path via Hugging Face. Manual download alternatives are given through ModelScope for both. The interactive scripts are named by their backend: 10, 11 and 12 follow the pattern 10_SenceVoice_QWen2.5_xxx.py, and the real-time variants are 13_SenceVoice_QWen2.5_edgeTTS_realTime.py and 14_SenceVoice_QWen2.5VL_edgeTTS_realTime.py. That line-number-based configuration is the single most important thing to understand about the repository. It is not a design flaw so much as a statement of intent: these are demonstration scripts, and the author expects you to open them.

The interruption logic is the most concrete engineering in the repository

The 241123 update describes single-modality free interruption, and it is the one place where the README gives actual numbers rather than a description. Interruption detection uses webrtcvad for real-time voice activity detection with a detection window of 0.5 seconds, an effective voice activation ratio of 40 percent, and a chunk size of 20 milliseconds. The arithmetic is spelled out: 500ms divided by 20ms gives 25 detection segments, and 25 multiplied by 0.4 gives 10, so if 10 of those segments are active the half-second counts as valid speech and is added to the buffer. The author also names the weakness directly, suggesting a model-based VAD would remove noise interference. That self-criticism is worth taking at face value. WebRTC VAD is a cheap energy-and-spectrum classifier, and a 40 percent activation threshold over a half-second window will fire on sustained background noise, on a television, or on a keyboard being used near the microphone. It is a reasonable default for a quiet desk and a poor one for an open office.

Speaker verification, wake words in pinyin, and a 512-token memory window

The 241130 update adds three features that move this from a demo toward something usable. First, voiceprint recognition: you set a directory for registration audio, and if that directory is empty the system enters registration mode automatically. Registration audio defaults to longer than 3 seconds and is configurable, with the README noting that longer samples give more stable voiceprints. The model is Alibaba's open source CAM++, trained on 3D-Speaker Chinese data, which the README justifies on the grounds that it suits Chinese conversation. Second, custom wake words: these reuse SenseVoice's recognition output by converting recognised Chinese characters to pinyin and matching against a configured string. The defaults are 'ni hao xiao qian' in 15.0_SenceVoice_kws_CAM++.py and 'zhan qi lai' in 15.1_SenceVoice_kws_CAM++.py. Third, dialogue memory: user and system history queues are maintained, and a new turn fetches the history and concatenates the new instruction with it. Maximum history length is configurable and defaults to 512. The split between 15.0 and 15.1 is deliberate: 15.0 has no history memory, 15.1 does. If you are debugging why the assistant forgets context, that file difference is the first place to look.

Getting it running: two install paths, and the one that fails

The README defines an environment as conda create -n chatAudio python=3.10 followed by conda activate chatAudio, then PyTorch with CUDA, tested locally on 2.0 and above, with torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pinned against the cu118 index. The simple path, which avoids CosyVoice, installs edge-tts==6.1.17 funasr==1.1.12 ffmpeg==1.4 opencv-python==4.10.0.84 transformers==4.45.2 webrtcvad==2.0.10 qwen-vl-utils==0.0.8 pygame==2.6.1 langid==1.1.6 langdetect==1.0.9 accelerate==0.33.0 PyAudio==0.2.14, and is verified by running python 13_SenceVoice_QWen2.5_edgeTTS_realTime.py. The CosyVoice path is where the friction lives. The README singles out pynini and wetext as the dependencies users report the most trouble with, and gives a specific workaround: conda install -c conda-forge pynini=2.1.6 followed by pip install WeTextProcessing --no-deps. The remaining CosyVoice dependencies are HyperPyYAML==1.2.2 modelscope==1.15.0 onnxruntime==1.19.2 openai-whisper==20231117 importlib_resources==6.4.5 sounddevice==0.5.1 matcha-tts==0.0.7.0, verified by python 10_SenceVoice_QWen2.5_cosyVoice.py. Note the advice to open the terminal as administrator if permission errors block installation. Also note that ffmpeg and anaconda installation are dismissed with 'search for a tutorial yourself', which is a fair signal of how much hand-holding to expect.

Where this is the wrong tool

The repository is a set of scripts, not a service. There is no daemon, no HTTP interface, no streaming protocol, and no packaging metadata described in the README beyond a requirements.txt. If you need to embed speech interaction in an application, you will be copying code out of numbered files and maintaining that copy yourself. There are no releases listed, so there is no version to pin against and no changelog beyond the dated update headings in the README. The configuration model compounds this: model paths and model names are edited at specific line numbers, which means any upstream refactor of those files silently invalidates the instructions. The wake-word mechanism is another boundary. It works by converting recognised Chinese characters to pinyin and matching a string, so it inherits SenseVoice's recognition errors. A misrecognised syllable means the wake word does not fire, and there is no phonetic fuzzy matching described. Finally, the project is Chinese-first by construction: the CAM++ justification cites Chinese training data, and the wake-word examples are Chinese phrases. If your users speak English, the ASR and wake-word layers are the parts you would have to replace, and at that point you are using the repository as a reference architecture rather than a dependency.

The alternative, and the actual difference in approach

The obvious comparison is the upstream CosyVoice repository, which this project states its main code comes from. The difference is not quality, it is scope. CosyVoice is a synthesis system with its own inference interface; this repository wraps it, prepends SenseVoice recognition and inserts a Qwen2.5 dialogue stage, and then adds two lighter synthesisers because CosyVoice's inference speed was hurting interactivity. So if your problem is text-to-speech specifically, going to CosyVoice directly gives you the maintained upstream and drops the ASR and LLM stages you did not ask for. If your problem is a full local voice loop, the value here is the assembled pipeline and the VAD interruption logic, neither of which CosyVoice provides. A second comparison worth naming is against hosted speech APIs. Edge-TTS is the middle position: it is a network synthesiser with no local model weights, and the README notes a connection error during experimentation that was resolved by upgrading to version 6.1.17, with no VPN required. That is the trade you are making across the three backends. pyttsx3 is local and fast but sounds like a system voice. Edge-TTS sounds better but needs a network round trip and has already shown one version-specific failure. CosyVoice is local and the most controllable, at the cost of the pynini and WeTextProcessing install and inference speed that the README itself calls too slow for real-time conversation.

Maintenance cost and licence position

The maintenance surface here is larger than the file count suggests, because the project is a thin layer over four moving upstreams: FunASR for SenseVoice, Qwen2.5, CosyVoice, and CAM++. Every pinned version in the install commands is a compatibility decision someone will eventually have to revisit, and the pins are tight (transformers==4.45.2, funasr==1.1.12, edge-tts==6.1.17, modelscope==1.15.0). The Edge-TTS connection failure that required a version bump is a concrete example of what that costs. There is also a hard platform assumption: conda, CUDA, and a PyTorch build matched to your driver, with the README pointing at the PyTorch previous-versions page rather than committing to one combination. On licensing, the repository itself is Apache-2.0, which is permissive and generally straightforward for commercial use. That does not settle the question for the whole system, because the models and libraries it loads carry their own terms, and the README does not enumerate them. Qwen2.5, SenseVoice via FunASR, CosyVoice and CAM++ each need checking independently against your intended deployment, and this is a question for your own legal review rather than something the repository answers.

Editorial conclusion

Adopt this if you want a working Chinese-language voice loop on a single machine and you are comfortable treating the numbered scripts as reference code to copy from rather than a library to depend on. Do not adopt it if you need a supported Python package with a stable API, or if your deployment requires English-first ASR and synthesis, since the documented model choices and the wake-word mechanism are built around Chinese. Before committing, run the simple path first (python 13_SenceVoice_QWen2.5_edgeTTS_realTime.py) to confirm your audio device and WebRTC VAD behave, then decide separately whether the CosyVoice dependencies are worth the pynini and WeTextProcessing install, because that step is the one the README flags as failing most often.

Official sources

1. ABexit/ASR-LLM-TTS on GitHub
2. Issues
3. License: Apache-2.0
4. README