# Scenema Audio clones a voice you own and performs with it, and that distinction is the whole product > A diffusion-based speech generator where the prompt describes not just what to say but how, built by pulling an audio transformer out of a 22 billion parameter video model. Useful and legally narrow at the same time: cloning a voice you have the right to clone is the intended use, and everything else is a problem for someone else. **ScenemaAI/scenema-audio** — Zero-shot expressive voice cloning and speech generation. Generate anything from short clips to full-length audiobooks with realistic emotional delivery, pacing, and breath control. Clone any voice from a 10-second reference and perform emotions the original speaker never recorded. - Repository: https://github.com/ScenemaAI/scenema-audio - Website: https://scenema.ai/audio - Stars: 552 · Forks: 76 - Language: Python - License: MIT - Published: 2026-09-20 · Updated: 2026-09-20 · Language: en - Canonical page: https://hysenlabs.com/projects/scenemaai-scenema-audio ## The pitch is that speech should perform, not just pronounce The argument in the opening is a complaint about the standard text-to-speech category rather than a feature list. Speech that merely pronounces words correctly is called functionally useless for filmmaking, audiobooks, or any situation where the delivery carries as much meaning as the words. That is a real limitation of concatenation and of most neural vocoders driven by a flat embedding: they produce clean audio with no relationship between how a sentence sounds and what it is about. What this project generates instead is described as speech with intention, pacing, breath control, and emotional arcs that shift inside a single generation. All of that is steered from one text prompt that is supposed to describe not just what to say but how to say it. Nothing in that is a new algorithm. What is new is the control surface: making delivery direction something you write rather than something you hope for. The provenance is unusual and explains a lot. The audio component is described as extracted from an audiovisual model with twenty-two billion parameters, which means this is not a from-scratch speech project but a piece lifted out of a video generation model and repurposed. A model trained on moving pictures learned how people sound in real scenes, angry and laughing and whispering and exhausted, because that is what is in the footage. ## The prompt is markup, and the tags are stage directions The generation prompt is a small XML document, which is an odd choice for a text interface and a good one. Attributes carry identity and setting, and child elements carry direction and sound: ```xml Performance direction. Speech text here. Environmental audio event. More speech. ``` Two of those elements do the interesting work. An action element is described explicitly as stage directions that are not spoken aloud, covering emotional shifts, physical delivery, pacing and breath control. It sits between speech segments, which means the delivery can change mid-sentence without restarting the generation. A sound element generates environmental audio alongside the speech, and it only does anything in two of the three shot modes. That constraint is the whole reason the shot attribute exists: a close shot keeps sound effects minimal and speech forward, a wide shot balances environment against speech, and a scene shot reinforces the effects. It is a mixing decision exposed as a prompt parameter, which is a neat trick and also the fastest way to get an unexpectedly busy result if you leave it on the default. ## A voice description is graded, and the documentation says out loud how The voice attribute is called the primary control for the entire output, and the file is unusually direct about how to write it. Its example pair is the clearest writing advice in the repository: a weak version describes a man speaking, and a strong version describes a man in his sixties with a deep baritone and gravel, a specific regional inflection, worn but warm, a nostalgic firelit cadence, and a line about having seen too much and chosen kindness anyway. That contrast is the whole lesson. Specific and theatrical beats generic and neutral, and the file attributes the voice attribute to vocal quality, emotion, accent, age, timbre and delivery style at once, which is why one attribute can carry so much. Gender is a required second attribute and is described more mechanically than the first: it controls pronoun assignment in the prompt that reaches the diffusion model. Scene is optional and conditions the ambient environment, so rain, office hum or crowd noise can be requested without appearing in the text. Language defaults to English and is a language code, with major world languages claimed as natively sounding. All five attributes together are the minimum viable prompt, and two of them are mandatory, which is a good design choice for a system where an empty description produces nothing useful. ## Ten to twenty seconds of reference audio, and variability is the point Zero-shot cloning here means you supply reference audio and the model generates an expressive performance, then transfers the reference voice's identity onto it. The documented guidance is specific about what makes a reference work: ten to twenty seconds, containing some emotional variability, and containing a range of pitch and intonation, because flat monotone clips produce significantly worse identity transfer. That last point is the one to sit with. A reference that sounds good in isolation is a bad reference here, because identity transfer apparently depends on hearing the voice move rather than hearing it sit still. Practically it means you want a clip of someone talking naturally, not a clean studio read, and the emotion in the reference does not have to match the emotion in the output. The separation the documentation describes is worth stating plainly: the reference provides identity, and the performance comes from the prompt. Any voice can perform any emotion, including ones that speaker was never recorded in. There is a separate mode for previewing a voice design without any reference at all, which produces short previews for iterating on a description, and the web interface puts that mode on its own tab for exactly that reason. ## Sixteen gigabytes of memory and a thirty-eight gigabyte first run The hardware requirement is stated before anything else: any NVIDIA card with sixteen gigabytes or more of memory. The default configuration is an eight-bit audio transformer plus four-bit quantization on the language model, with automatic offloading for smaller cards, which is how a model descended from a twenty-two billion parameter setup fits at all. The install is a container, and the file recommends it: ```bash git clone https://github.com/ScenemaAI/scenema-audio.git cd scenema-audio # Set your HuggingFace token (Gemma 3 access required) export HF_TOKEN=your_huggingface_token # Build and run (models are downloaded on first start) docker compose up ``` Two details there cost real time and are stated rather than hidden. First, a token from the model hub is required, and the comment says why, because access to the language model family is gated. Second, the checkpoints are not in the image: the first start downloads roughly thirty-eight gigabytes and caches them in a volume, and only then do subsequent starts come up quickly. The compose file shows what that volume holds, and the names are a decent map of the pipeline: the quantised audio transformer, a separate audio autoencoder encoder, the pipeline itself, a twelve-billion parameter instruction-tuned language model in four-bit form, a neural vocoder, and a voice conversion component mounted from its own directory. One GPU is reserved explicitly through the container runtime rather than left to whatever happens to be attached. ## Generation is one request, and there are three ways to make it Once the container is up, the surface area is a single endpoint. The shortest path is the script that ships with the repository: ```bash # Using the included script python generate.py output.wav ``` Or a request to the generate endpoint with a prompt and optionally a seed, which is the shape you would want if you were wiring this into a script rather than experimenting by hand. Voice previews use the same endpoint with a different mode, and cloned output adds a reference voice URL to the same payload, so the three modes are one request with progressively more fields rather than three separate tools. The interface for experimenting is a bundled web application with four tabs: one that assembles a prompt from separate fields for voice, text, scene and action with preset examples, one for short voice previews, one for uploading reference audio, and one that takes raw prompt markup for full control. That last tab is the honest signal about how the thing is meant to be driven, since the field-based builder is a convenience over an interface you are expected to learn. The file also documents how to reach a remote GPU server, either by forwarding a port over SSH or by asking the web application to open an outbound tunnel. Both are reasonable. The tunnel option deserves a second look before you use it, because the documentation says the resulting link can be accessed by anyone, and an unauthenticated endpoint that generates speech from uploaded audio is a different proposition from one bound to your own machine. ## A small repository with no releases and an editor directory in it The tree is short enough to read in full: a licence, the readme, the container files, a small application module, the generation script, and folders for source, tests and documentation. There is no packaging configuration, no lock file, no continuous integration directory and no changelog. The image definition is where the real dependency list lives, since everything is installed inside the container, with pinned versions of the deep learning runtime, the transformers library and the acceleration library, plus a build of the media tooling fetched from a release URL. Two observations follow from that. The absence of releases means there is nothing to pin a deployment to, which is consistent with a project whose model weights change independently of its code anyway. And the presence of an editor configuration directory in the committed tree is a small sign of a project that has not yet separated the tools of its author from its deliverable. What the repository does have is the thing that matters most for this kind of system, which is documentation of the failure modes. It tells you what makes a reference recording bad, what a weak prompt looks like next to a strong one, which shot mode makes environmental audio audible, and what the first run will cost you in download time. For a tool whose output is hard to evaluate from the outside, that is the part worth reading before you start. ## Conclusion Read the consent line before the feature list. A tool that takes ten seconds of someone's voice and attaches any performance to it is a legitimate instrument for audiobooks, film and accessibility work, and it is equally a fraud kit aimed at anyone else's voice, and several jurisdictions now treat synthetic voice as biometric data with real consent requirements attached. With that settled, the engineering is worth a look: the audio transformer is a component lifted out of a much larger video model, the prompt carries delivery direction as first-class markup rather than as a paraphrase, and the hardware cost is stated plainly at 16 gigabytes of memory and a first-run download of roughly 38 gigabytes. Expect to spend your first afternoon on setup rather than on generation, and read the public sharing option carefully before you turn it on. ## FAQ ### How much hardware does Scenema Audio need? Any NVIDIA GPU with 16 GB or more of video memory. The default configuration uses an eight-bit audio transformer with four-bit quantization on the language model and offloads to slower memory on smaller cards. The first start downloads roughly 38 GB of checkpoints into a volume, and later starts reuse that cache. ### What does a good reference recording for voice cloning look like? Ten to twenty seconds that contains real emotional variability and a range of pitch and intonation. The documentation is explicit that flat, monotone clips transfer identity much worse, so a clip of someone talking naturally works better than a clean studio read, and the emotion in the reference does not need to match the emotion in the output. ### How do I control emotion and pacing in the output? With action tags placed between speech segments inside the prompt markup. They are stage directions rather than spoken words, and they cover emotional shifts, physical delivery, pacing cues and breath control. A separate optional element generates environmental audio alongside the speech, and it only takes effect in the wide and scene shot modes. ### Can I use a voice I do not own? Do not. The reference audio is what supplies the voice identity, so using someone else's recording means producing speech in a voice they never agreed to speak in, which is at best a breach of the terms of whatever platform the recording came from and at worst a fraud or impersonation offence. Clone your own voice, or get written permission from the speaker. ### Is there a hosted version I can try before installing anything? The project links a hosted page on its own domain for hearing demos and trying the system, and a video walkthrough. The open source side is a container you run yourself, which needs a gated model token before the first run because language model access has to be approved. ### Is the public sharing link in the web interface safe to use? It depends on what you are exposing. The documentation describes an option that opens an outbound tunnel and produces a link that anyone with it can reach, which is fine for a demo on a sample voice and a poor idea for a machine where you have uploaded reference audio or hold credentials. Forwarding a port over SSH keeps the interface bound to your own machine instead. ## Sources - [Issues](https://github.com/ScenemaAI/scenema-audio/issues) - [License: MIT](https://github.com/ScenemaAI/scenema-audio/blob/main/LICENSE) - [Project website](https://scenema.ai/audio) - [README](https://github.com/ScenemaAI/scenema-audio/blob/main/README.md) - [ScenemaAI/scenema-audio on GitHub](https://github.com/ScenemaAI/scenema-audio) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/scenemaai-scenema-audio