CleanS2S: A Single-File Streaming Speech-to-Speech Agent

What CleanS2S is trying to fix in voice agent code

Most speech-to-speech demos arrive as a directory tree: a server module, a client module, a config layer, an abstraction for every component. That structure is useful once a project matures, but it makes the pipeline hard to read in one sitting. CleanS2S takes the opposite position. The README describes it as a Speech-to-Speech prototype agent that provides streaming interactions in a single-file implementation, and states the aim is to give users a Chinese interactive prototype agent in the GPT-4o style. The claimed audience is explicit: researchers who want to quickly explore or validate the potential of an S2S pipeline, and readers who want to glance at the whole pipeline and then modify it.

So the problem is not latency or model quality. The problem is comprehension cost. If you want to know where the transcript leaves the ASR stage and enters the LLM stage, a single file answers that question by scrolling. The README also notes that the project added what it calls Subjective Action Judgement, described as enhancing the agent's ability to proactively initiate actions during conversations. That is a design goal about turn-taking, not about recognition accuracy.

The scope is narrower than the topic list suggests. The demos linked in the README cover Chinese-language conversations on investment, mood, university application advice, and a stomach medicine topic. Nothing in the supplied material shows an English-language demo, and the README's own framing says the target is a Chinese interactive prototype. Treat the language coverage as an open question until you check the ASR and TTS handlers yourself.

The pipeline: ASR, LLM, TTS, and two WebSocket endpoints

The README names the components directly. The pipeline is composed of ASR (Automatic Speech Recognition, also called Speech to Text), LLM (Large Language Model), and TTS (Text to Speech), plus two WebSocket components: a Receiver that contains VAD, and a Sender. Audio and text both travel over the WebSocket connection.

The concurrency model is the part worth understanding before you read the file. According to the README, streaming is achieved with multi-threading and queueing, and every component is designed to be asynchronous and non-blocking, taking data from an input queue and putting results into an output queue. That is a producer-consumer arrangement. It is the reason the pipeline can keep the socket busy while a model is still generating: nothing waits on a synchronous call to return before the next chunk moves.

Full-duplex behaviour sits on top of that. The README attributes it to the WebSockets library and states that the user can speak and listen at the same time. Interruption handling is described as stopping the current processing and starting on the new input, while keeping the context of the previous conversation and the interruption. The README is candid about why this matters: it calls the assistant-style, turn-based response pattern one of the most important drawbacks for human-like conversation, and says the project adds other strategies to make the exchange more interactive.

What the README does not specify is the queue depth, the chunk size, or how partial TTS output is discarded when an interruption arrives. Those details live in the single file, and if you are evaluating barge-in quality, that is exactly where you should look first.

Web search and RAG: two separate helpers, two separate jobs

Two extensions are described in the README, and they are not the same mechanism. WebSearchHelper conducts online searches based on user queries or to gather additional information relevant to the conversation. The stated purpose is to let the agent reference up-to-date or external data. RAG, by contrast, implements retrieval-augmented generation: it first retrieves relevant information from a database, then uses that information to generate a response. The README calls this a two-step process and says it grounds replies in relevant, factual data.

The distinction matters operationally. Web search depends on a live external service and returns whatever that service returns. RAG depends on a database you populate, so its ceiling is set by your corpus, not by the model. If your prototype needs current prices or news, the search path is the relevant one. If it needs to answer from internal documents, the RAG path is the relevant one, and the work moves to ingestion and chunking, which the README does not cover.

The README includes output examples comparing LanguageModelHandler against RAGLanguageModelHelper on two cosmetic product cases. One comparison is striking: the plain handler produces a structured, list-like product script, while the RAG helper produces a first-person, exclamatory sales monologue. Both are in Chinese. The README does not explain whether that stylistic gap comes from retrieved context, from a different prompt template, or from sampling settings. It also carries a note that the maximum token output was limited to a small size because of the authors' computing resource constraints. Read those examples as illustrations of output shape, not as a quality benchmark.

Getting it running: backend first, then the client

The README's Get Started section splits into Backend (Server) and Frontend (Client), which is a two-process setup: the pipeline runs as a server and you connect a client to it. The supplied material does not include the literal install commands, the Python version, or the exact dependency list, so I cannot give you a verified command line here. What the README does establish is the shape of the work: install the Python dependencies for the backend, supply credentials and endpoints for whichever ASR, LLM, and TTS services the handlers expect, start the server, then start the client and point it at the WebSocket address.

Because the implementation is a single file, the authoritative configuration reference is that file. The component names to search for are the ones the README uses: the Receiver and Sender WebSocket components, the ASR, LLM, and TTS stages, WebSearchHelper, and RAG. The README also references a separate backend README (linked in Chinese) that documents the Subjective Action Judgement addition, so action-triggering behaviour is configured there rather than in the main README.

A practical warning before you start: the README's own output examples were produced under a small token limit, so a default configuration may truncate responses. If your first run produces clipped answers, check the generation limit before you debug the pipeline. And if you are outside the Chinese-language scenario the demos cover, verify the ASR model and TTS voice configuration first, because that is where a mismatch will surface immediately.

Where CleanS2S is the wrong tool

The single-file design is the project's selling point and its main constraint. Everything in one file means there is no package boundary to version, no plugin interface to implement against, and no stable import surface for another project to depend on. If you want to embed this pipeline as a library inside a larger service, you will be copying or forking the file, and your changes will diverge from upstream. The README frames the project as a reference implementation and a place to validate novel ideas, which is consistent with that constraint rather than in tension with it.

The repository has no releases. The supplied material shows no tags and no published version, so there is no versioned artifact to pin. Upgrading means tracking the main branch. That is fine for a research prototype and awkward for anything with a deployment schedule.

There is also a gap between what the README asserts and what it demonstrates. It states that all components are asynchronous and non-blocking, and that interruptions stop current processing. Those are architectural claims, and the supplied material contains no measurements of interruption latency, no queue-depth guidance, and no failure analysis for the case where a TTS chunk has already been sent when the user starts speaking. If your use case depends on barge-in feeling instant, that is the specific behaviour you must exercise yourself.

Finally, the demo set is Chinese-language and covers a handful of conversational topics. If your product speaks English, or needs multiple simultaneous sessions with isolation guarantees, nothing in the supplied material tells you how the single-file server handles that.

How it differs from assembling the same stack yourself

The obvious alternative is to build the same ASR, LLM, and TTS chain on a general-purpose agent framework or a real-time media server, wiring each component as a separate service. The difference is not capability, it is where the complexity sits. A framework-based build gives you versioned dependencies, plugin interfaces, and a community to ask; it also gives you a configuration surface and an abstraction layer between you and the audio path. CleanS2S inverts that: you read the audio path directly, and in exchange you own every integration detail yourself.

A second alternative is a hosted real-time voice API. That removes the pipeline from your responsibility entirely, including the interruption logic and the VAD tuning, and replaces it with a vendor endpoint and a per-minute cost. CleanS2S keeps the pipeline local and inspectable, which is the point if you are researching turn-taking strategies or swapping in your own model. The README explicitly says the user can quickly change the model they like, add new components, or customize the pipeline. A hosted API does not offer that.

The honest framing is that these are different jobs. If you need to ship a voice feature, a hosted API or a maintained framework is the shorter path. If you need to understand and modify the streaming mechanics, a single file you can read in full is worth more than an abstraction you have to reverse-engineer.

Maintenance cost and the Apache-2.0 terms

The repository is not archived, and the supplied material shows a last push in April 2026. There are no releases, so there is no changelog to read and no upgrade path other than the main branch. For a prototype that is acceptable; for a dependency it means every pull is an unreviewed change to the file you forked.

Licensing is Apache-2.0, which is a permissive licence that generally allows commercial use, modification, and redistribution provided you keep the required notices and state changes. Two obligations are worth flagging without giving legal advice: Apache-2.0 includes a patent grant and a corresponding termination clause, and it requires that modified files carry prominent notices of change. If you fork the single file and ship it, that notice requirement applies to your fork. The README also has a Citing CleanS2S section, which suggests academic citation is expected even though the licence does not require it.

One dependency note: the README credits WebSockets for the full-duplex behaviour. That library carries its own licence, and it is not covered by CleanS2S's Apache-2.0 grant. Check it separately if you redistribute.

Editorial conclusion

Adopt CleanS2S if you are a researcher or engineer who wants to read one Python file, swap the LLM, and validate an S2S pipeline idea in an afternoon. Do not adopt it if you need a maintained voice product with a stable API, versioned releases, or a support contract; the repository has no releases and the README itself flags that the demos ran under constrained compute. Before committing, verify that your ASR, LLM, and TTS credentials and endpoint formats match the handlers in the single file, and confirm the Apache-2.0 terms against your own distribution plan.

Official sources

1. Issues
2. License: Apache-2.0
3. opendilab/CleanS2S on GitHub
4. README