# video-ai-talking keeps credentials in localStorage and caches the lip-sync stage

> A local web tool that turns a clip of a person and a script into a vertical talking-head MP4, using Alibaba Bailian VideoRetalk for lip sync, one of two Chinese text-to-speech providers for the voice, and FFmpeg for the composition. The keys never touch the .env file, and the pipeline caches the expensive stage.

**yizhi-chengzi/video-ai-talking** — For talking-head videos without reading to camera yourself: upload a real person's video, write the subtitles, and it does local voiceover and lip-sync to produce a vertical MP4. API keys stay in your browser only.

- Repository: https://github.com/yizhi-chengzi/video-ai-talking
- Stars: 471 · Forks: 70
- Language: TypeScript
- License: MIT
- Published: 2026-09-18 · Updated: 2026-09-18 · Language: en
- Canonical page: https://hysenlabs.com/projects/yizhi-chengzi-video-ai-talking

## Keys live in the browser, and the .env file holds two port numbers

The credential story is the first thing to understand, because it decides what you have to trust.

API keys and text-to-speech tokens are stored in the browser under the localStorage entry vat.config. The .env file on disk is for ports only, and the template makes that explicit by shipping just PORT=8789 and WEB_PORT=5175 with an instruction not to put keys or tokens in it.

So the local server reads credentials that the browser sends it, rather than holding a credential of its own. The task JSON files written under the project data/ directory never contain a key or a token.

What does land on disk is the working material: task JSON, the generated audio, intermediate lip-sync clips and the finished MP4. Source material you reference is not copied into the project at all, only its path is recorded, so the tool works against files that stay where you put them.

Two consequences follow. Stopping the page or stopping the local service stops a task in progress, and the next start does not pick it up. And anyone with access to the browser profile on that machine can read the keys, so this is a single-user local arrangement rather than a shared deployment.

The README is explicit that the project is not an official product of Douyin, ByteDance, Volcengine, Alibaba Cloud or DeepSeek.

## Four credentials, three reachable domains, and only one is mandatory

The key table is the most useful page in the README, and it separates what is required from what is a choice.

Mandatory for any output is an Alibaba Bailian VideoRetalk API key for the North China 2 or Beijing region, starting with sk-, entered in the lip-sync configuration panel.

Voice is a choice of two. You can use a Bailian CosyVoice API key, also starting with sk-, or a Volcengine speech synthesis App ID together with an Access Token. Either one is enough.

The script is optional. A DeepSeek API key enables AI generation of the script text, and leaving it empty means you type the text yourself.

Two details about the credentials are easy to get wrong. The Bailian voice key and the Bailian lip-sync key go into separate fields, and you may use the same key in both or a different one; the choice of provider is made at render time rather than in the configuration. And for Volcengine, the App ID and the Access Token have to come from the same speech synthesis application, which is the specific thing to check when a voice test fails.

Three hosts have to be reachable from your machine: dashscope.aliyuncs.com for voice, temporary upload and VideoRetalk, openspeech.bytedance.com for Volcengine voice, and api.deepseek.com for the script.

There is also a test button on each card, which is the fastest way to find out which credential is wrong.

## The reference video and the voice are uploaded to object storage for 48 hours

The privacy position is more nuanced than the local framing suggests, and it is documented rather than buried.

The local claim is accurate in one sense. There is no cloud server for this tool, and it does not call any interface on your behalf beyond the providers you configure. Tasks and outputs are written to the local data/ directory.

But at the moment of rendering, the reference video and the generated voice audio are uploaded to Alibaba Bailian's official temporary storage, described as object storage with a retention of about 48 hours, so that VideoRetalk can process them. The details are in docs/privacy.md.

So a render means your face and your generated audio leave the machine, temporarily, through a provider you have an account with. That is inherent to using a hosted lip-sync model and not something the local architecture could avoid.

A separate and equally careful note covers the screenshots. The imagery in the interface shots comes from free commercially usable stock libraries such as Pexels and is used only to demonstrate the interface. It is stated to be not real users of the tool, and the person on camera does not endorse the project. The repository ships its own talking-head demo clip with its provenance recorded in demos/README.md.

The developer guidance is equally direct: do not import other products' source from this directory, and do not commit local data, keys, or footage of faces.

## One task at a time, and a regenerate that skips lip sync when nothing changed

The render pipeline is three stages in a fixed order: voice synthesis, then lip sync, then composition of the output.

Concurrency is limited by design. A run produces one output, and only one task can be running at a time. There is no queue and no batch mode in the documentation, so producing ten clips is ten sequential runs.

The interesting behaviour is the regenerate path. If the reference video, the script text and the voice are all unchanged, the lip-sync stage is skipped and only the composition step is re-run with FFmpeg.

That is the correct cache key, and it is worth understanding what it implies. It is the lip-sync stage that costs money and time and involves an upload, so a change confined to the composition layer, a different skin, or a subtitle style is cheap to iterate on. But editing one line of the script invalidates the whole thing, because the script text is part of the key.

The script is written in the fourth step, where you choose either AI-generated or manually typed text. AI generation lets you pick a finished duration first, with presets at 15, 30, 45 and 60 seconds and a custom range of 10 to 60, and then supply a topic.

Finished clips accumulate in an output library where they can be previewed, downloaded or deleted.

## Development runs on two ports, and the built mode needs a build first

There are two ways to run it and they serve on different addresses.

Development mode is the documented daily path:

```bash
git clone https://github.com/yizhi-chengzi/video-ai-talking.git
cd video-ai-talking
cp .env.example .env
npm install
npm run dev
```

That starts two processes at once: the Vite development server for the interface and a TypeScript-watched server for the API, run side by side with labelled output so you can tell them apart. The browser then opens on port 5175 while the API listens on 8789.

The second path is for after a build:

```bash
npm run build
npm start
```

A single address on 8789 then serves both the page and the API. The README notes that opening the page without building first gives you a 404, which is the failure you get from starting the server with no compiled frontend.

One implementation detail is worth knowing if you deploy this yourself. The start script runs the server through tsx, the TypeScript loader, rather than from compiled JavaScript, while the build step compiles the frontend with tsc and Vite. So the production path still needs the TypeScript execution dependency present at runtime.

The stack behind it is Hono on the Node server adapter, React 19 with Tailwind 4, Vite 8, Vitest 4 and TypeScript 6.

## An output with no Chinese font still renders, with boxes where the text should be

The troubleshooting section is unusually honest about a failure that produces a file rather than an error.

If the finished video has no Chinese title, the cause is that FFmpeg needs a usable Chinese font on the machine. The tool searches common paths for one. If it does not find one, the output is still generated and the title and subtitles may appear as boxes.

That is a design choice with a trade-off. Failing the render would be safer for quality, but producing a silent artefact with tofu glyphs is a common enough outcome that documenting it beats raising an error nobody can act on inside a headless run.

The other three entries are more conventional. If FFmpeg cannot be found, run the version commands directly in the terminal to confirm, and on macOS install it with the package manager. If lip sync fails, check that the VideoRetalk key is filled in, that dashscope is reachable, and that the reference clip is a close-up with a visible face. If the voice test fails, verify the credentials for the provider you actually selected, including that a Volcengine App ID and Access Token come from the same application.

There is one platform-specific entry too. On Linux the file picker does nothing unless Zenity is installed, since that is what the system picker shells out to. Dragging a file into the page works instead.

The other documented constraint is on the reference footage itself: a clear front-facing close-up, not a side profile, not a wide shot, not multiple people, and not an occluded face.

## One environment variable makes the whole chain runnable offline

Setting VAT_MOCK=1 replaces every external call with a mock, which is the single most useful thing in the configuration.

With it set, voice synthesis, the temporary upload, the VideoRetalk call, the FFmpeg step and the DeepSeek request all run against mocks, and nothing reaches the network.

The test suite takes the same approach by default. Running npm test and npm run typecheck makes no real calls to DeepSeek, Volcengine or Bailian, does not run a full VideoRetalk job, and does not require FFmpeg to be installed on the machine at all.

That combination means a contributor can work on the interface, the prompts, the composition layer or the caching logic on a laptop with no provider accounts and no media tooling. It is a real design investment rather than a convenience, and it is the reason the project can accept contributions from people who do not have an Alibaba account.

The governance files around it are conventional and present: a licence, a code of conduct, a contributing guide, a disclaimer, a security policy, a changelog and separate documentation for privacy and for fonts. Node 20 or newer is the stated floor.

The only unusual packaging detail is that the package is marked as not private while having no entry point, no exports field and no files list, so there is nothing there to publish.

## Conclusion

This fits someone making short talking-head clips on their own machine who has a clear close-up of a person and is willing to hold two or three Chinese provider accounts. It does not fit anyone expecting a hosted service, a queue, or a batch job. Three things to check before your first render. The reference video and the generated audio are uploaded to Alibaba's temporary object storage for roughly 48 hours, which is a different privacy posture from the local-only framing. Only one task runs at a time and an interrupted task does not resume. And a machine with no usable Chinese font still produces an output, but the titles and subtitles render as boxes. The last push is dated 2026-09-03.

## FAQ

### What does video-ai-talking need before it can produce a video?

Node.js 20 or newer, FFmpeg and ffprobe runnable in the terminal, and a Bailian VideoRetalk API key for the North China 2 or Beijing region, which is the only mandatory credential. Voice is one of a Bailian CosyVoice key or a Volcengine App ID with Access Token, and a DeepSeek key is optional because the script can be typed by hand.

### Where does video-ai-talking store my API keys?

In the browser, under the localStorage entry vat.config. The .env file holds only the two port numbers and is explicitly not the place for keys or tokens. Task JSON files under data/ never contain a key, and referenced source footage is recorded by path rather than copied into the project.

### Is video-ai-talking fully local and private?

There is no cloud server for the tool and outputs stay on your machine, but at render time the reference video and the generated voice audio are uploaded to Alibaba Bailian's official temporary object storage, described as about 48 hours retention, for VideoRetalk to process. Details are in docs/privacy.md.

### How do I run video-ai-talking locally?

Clone the repository, copy .env.example to .env, run npm install and npm run dev, then open the page on port 5175 while the API listens on 8789. After npm run build you can use npm start and both the page and the API are served from 8789, though opening the page without building first returns a 404.

### Why does my video have boxes instead of Chinese titles?

FFmpeg needs a usable Chinese font installed on the machine. The tool searches common paths, and if it finds nothing the output is still generated but titles and subtitles render as boxes. The fonts note is in templates/fonts.md.

### Can I test video-ai-talking without provider accounts?

Set VAT_MOCK=1, which mocks voice synthesis, upload, VideoRetalk, FFmpeg and DeepSeek and avoids the network entirely. The test suite works the same way by default: npm test and npm run typecheck make no real provider calls, do not run a full VideoRetalk job and do not require FFmpeg installed.

## Sources

- [Issues](https://github.com/yizhi-chengzi/video-ai-talking/issues)
- [License: MIT](https://github.com/yizhi-chengzi/video-ai-talking/blob/master/LICENSE)
- [README](https://github.com/yizhi-chengzi/video-ai-talking/blob/master/README.md)
- [yizhi-chengzi/video-ai-talking on GitHub](https://github.com/yizhi-chengzi/video-ai-talking)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/yizhi-chengzi-video-ai-talking