Interactive LLM Powered NPCs: a voice-driven dialogue overlay for single-player games

An overlay, not a mod: the game is never modified

The premise of this project is that you can hold a spoken conversation with any non-player character in an already released game, and it does it without touching the game's source code or doing complex modding.

The mechanism for that is stated plainly in the readme. The system works by replacing the facial pixels generated by the game, integrating the generated facial animation into the gaming environment. In other words, the game renders as usual, the tool captures the screen, replaces the NPC's face with a synthesised talking head, and puts the result back. The game has no idea.

That is a genuinely different design from a mod, and it is why the tool claims to work with any game, without the need for game modifications or altering the game's source code.

The target titles are named in the readme: Cyberpunk 2077, Assassin's Creed, GTA 5, and other popular open-world titles. The reasoning given is that these games have plentiful NPCs and beautiful environments, and have long lacked the ability to talk to any NPC, and that the developers of already released titles are unlikely to add it. The scope of interest is single-player immersive dialogue, not competitive play.

The audience is therefore a developer or hobbyist who wants to build an immersive dialogue layer on top of a game, and the entry point is a microphone rather than a keyboard.

The pipeline from microphone to replaced pixels

The readme walks through the pipeline in eight numbered stages, and each stage names the technology that does the work.

First, microphone input: you speak and your voice is converted to text. Second, facial recognition: the system identifies which NPC you are interacting with, whether a background character or a side character. Third, character identification: based on the dialogue and the character recognition, a unique personality and name are generated for background NPCs, while for side characters their specific character traits and knowledge are accessed.

Fourth, the LLM integration step passes the transcribed text and the character information to a language model, which also uses vector stores containing character-specific and world information. Fifth, a pre-conversation.json file holds the character's iconic lines and lines reflecting their talking style, and random lines from it are passed to the model while generating a response, so the NPC sounds like itself. Sixth, the response is converted to speech with text to speech, and a facial animation video is generated using the extracted face image of the NPC together with the audio, with SadTalker doing the lip synchronisation.

Seventh, the animation video and audio are integrated into the game by replacing the displayed face pixels. Eighth, emotion recognition: the system captures your facial emotions from your webcam so the NPC can adjust its response, which the readme presents as an additional layer of depth.

The vector store is what stops the character from forgetting everything, and the pre-conversation file is what stops it from sounding like a chatbot. Both are cheap, unglamorous mechanisms, and both are the difference between a demo and something you can talk to for ten minutes.

Talking to a character whose face you cannot see

One feature is worth separating out, because it is the one that changes the design of the whole thing.

The readme describes non-visual interactions, where the project goes beyond face-to-face conversation by enabling interactions when the NPC's face is not visible, such as during horseback riding or intense combat sequences. To engage, you say the NPC's name first, then your dialogue, and the system follows the same process of transcribing, generating a response, and converting it to speech, providing an uninterrupted conversation even in action-packed moments.

That is a hard problem. If the tool can only work by finding a face on screen and replacing it, then the entire feature set collapses the moment the camera is elsewhere, which in a third-person action game is most of the time. Supporting the name-then-speak path means the system has to identify the NPC from the spoken name rather than from pixels, and it has to keep the conversation state without the visual anchor.

It also means the tool is doing a lot of work in the background at any moment: audio capture, speech recognition, a model call and a speech synthesis round trip have to complete fast enough to be usable while the player is also fighting. The repository's later work on a simulation harness and a runtime host, described in the package scripts, is consistent with that latency problem being taken seriously.

The install steps in the readme describe the old Python layout

The published installation instructions are a numbered list, and they describe a Python environment with a vendored SadTalker directory.

The first steps are a clone, a directory change, and a virtual environment:

git clone https://github.com/AkshitIreddy/Interactive-LLM-Powered-NPCs.git

cd Interactive-LLM-Powered-NPCs

python -m venv .venv

.venv\scripts\activate

Then the dependencies, and the SadTalker models:

pip install -r requirements.txt

cd sadtalker

bash scripts/download_models.sh

The prerequisites list in the readme names Python 3.10.6, Git, wget, Microsoft Build Tools and Visual Studio for compiling on Windows, and FFmpeg for audio and video processing.

Here is the catch, and it matters if you are trying to run this today. The repository no longer matches those instructions. The top-level tree is now a Rust workspace with crates/, apps/, workers/, tools/, a native/ directory, a catalog/, schemas/ and profiles/, alongside a pnpm workspace, a Tauri application, and a set of planning documents including README-2.0.md, ROADMAP.md, HANDOFF.md, IMPLEMENTATION_STATUS.md and several dated handoff notes. The package.json describes itself as a Windows-first desktop runtime for low-latency, game-aware NPC conversations, at version 2.0.0-alpha.1.

So the readme documents the 1.0 Python arrangement and the repository is in the middle of a rewrite. The install steps are still a reasonable description of the original approach, but they are not a description of what is in the tree today.

A Rust workspace and a Tauri app, with an agent-driven workflow

The current state of the repository is visible in its manifests, and it is a lot more structure than the readme suggests.

The Cargo workspace has members crates/* and apps/runtime-host, edition 2021 with rust-version 1.96, and MIT licensed. The lint configuration is strict in a way that tells you about the code's standards: unsafe_op_in_unsafe_fn and unused_must_use are denied, and the clippy block denies await_holding_lock, dbg_macro and todo, warns on print_stderr and print_stdout, and warns on unwrap_used. The release profile is tuned for a shipped binary, with codegen-units = 1, thin LTO, opt-level 3, panic = abort and symbols stripped, and the dev profile deliberately disables debug info and incremental compilation with a comment about avoiding multi-gigabyte caches.

The JavaScript side is a pnpm workspace with Node 20.20.2 and pnpm 10.28.2, where the app scripts are filtered by package name, for example dev, tauri, build and test targeting @npc2/control, and there are separate scripts for a simulation harness under @npc2/simulation-harness, including sim and sim:verify. There is a visual:qa script running node scripts/visual-qa.cjs, with playwright-core as the only dev dependency, and a Rust test command that runs cargo test across the workspace with all targets and features.

The file names at the root are the other story. AGENTS.md, HANDOFF.md, NEXT_AGENT_HANDOFF_2026-09-16.md, NEXT_AGENT_HANDOFF_2026-09-17.md, PAUSED_PROGRESS_2026-09-07.md and REVIEW_READY_2026-09-08.md indicate a development process run by successive agents rather than a single maintainer, with progress state written to disk and handed over. That is unusual to see in a public repository and it explains the gap between a cheerful readme and a half-finished rewrite.

What the licence allows, and what the project does not promise

The licence is MIT, and the project is not archived, with the last push on 2026-10-16. The only tagged release is v1.0.0 from 2023-09-17, which is a useful way to see the gap between what has been published and what is in the tree.

The MIT terms mean you can use, modify and redistribute the code, including commercially, and that is the straightforward part. The unresolved parts are about the software this project depends on. SadTalker is a separate project with its own licence, the language model you point it at is a third-party service with its own terms and cost, and the games it is designed to overlay are commercial products whose anti-cheat and terms of service are a separate matter entirely.

That last point is where a reasonable person should slow down. The readme frames the tool as adding a feature the original developers are unlikely to ship, and describes the approach as replacing facial pixels on screen rather than modifying the game. Both the design and the documentation are careful to stay on the right side of the line, but a tool that reads a game's video output and writes back over it is the kind of thing that anti-cheat systems can flag, and whether that happens depends on the game and the mode you are playing in.

The honest summary of the state: the idea is well worked out and documented, the original version was installable, and the rewrite is a large, in-progress restructuring. The README-2.0.md, ROADMAP.md and the handoff notes are where the current status actually lives.

Editorial conclusion

Interactive LLM Powered NPCs fits a solo developer who wants to prototype spoken NPC dialogue on top of a game they already own, and it does not fit anyone who needs a documented, supported tool with predictable setup, since the published installation steps still describe the older Python and SadTalker layout while the repository has moved to a Rust and Tauri runtime. Before you start, read README-2.0.md and the handoff notes in the repository root, and confirm the Windows build tools and FFmpeg are in place, because the pixel-replacement approach depends on both the capture and the encoder.

Frequently asked questions

Official sources

1. AkshitIreddy/Interactive-LLM-Powered-NPCs on GitHub
2. License: MIT
3. Project website
4. README
5. Releases