VoiceMem

The dual-brain design

VoiceMem is a memory system for voice assistants built on what the authors call a streaming dual-brain architecture. The left brain manages information directly and keeps Mem0's performance under a Top-3 limit, which the README describes as maintaining full load without blowing up the context. The right brain manages emotional intelligence through long-term and short-term emotion attribution, with cross nodes that link feelings to the left brain's facts and maintain them together. The split is meant to give the assistant both accurate recall and a sense of the user's state, so replies feel informed rather than flat. A single query is about 300 tokens, which keeps each memory lookup cheap. The architecture is fully decoupled, and every component including the underlying memory engine can be swapped, so you are not locked into one embedding or one model. That design choice matters for researchers who want to experiment with different backends without rewriting the whole system. The authors also stress that the whole project will permanently stay open source, which is a commitment aimed at building trust with users who hand the system their voice data. The decoupling plus the open-source promise together make VoiceMem look less like a black box and more like a toolkit you can inspect and reshape.

Latency and openness

Low latency is a stated goal. The README says VoiceMem compresses information, stores it in layers, and uses streaming queries with speculative prefetch in the zero to 300 millisecond range, so memory lookups add almost no delay to a voice turn. The project also stresses that it will permanently stay fully open source, and the README links a homepage, a technical report on arXiv, and Hugging Face repositories for utility code, model families, and a ChatMem-400K dataset. There are official memory models, including a Qwen-based reply model the authors fine-tuned, and the components include ASR, speaker recognition, scene detection, emotion sensing, and local embedding, all bundled so a single install covers the pipeline. For a voice assistant, the benefit is that the system understands the user better over time while staying fast enough to use in a live conversation rather than only in batch processing. The bundled components mean you do not assemble five separate services before the first test, which lowers the barrier compared with most memory systems that assume you already run each piece. The Hugging Face links also give you prebuilt weights, so a researcher can reproduce results without training from scratch. The prebuilt weights mean a researcher can start without a long training run.

Running it

The quick start covers installation, model download, and basic usage. The install includes the full set of local components, ASR, speaker recognition, scene detection, emotion perception, and local embedding, so you do not assemble them separately. A note tells you to preload local models before the first call so the user does not wait on loading mid-turn. The base usage shows storing an audio file, after which the system runs ASR, speaker recognition, scene detection, emotion perception, and embedding extraction internally. Querying is described as decoupled from writing: writes are slower because they extract facts, assign labels, and build a graph, while queries use pure vector retrieval and are unaffected by write cost. A streaming interface is presented as a continuously running VAD-style endpoint that processes audio as it arrives. The repository also ships an interactive demo. VoiceMem is Apache-2.0 licensed and written in Python, and the authors commit to keeping the codebase open. The split between slow writes and fast queries is the part that makes real-time use feasible, because the expensive graph building happens off the critical path while the user is still talking, and the prefetch keeps the answer ready before the question ends. That design keeps the assistant responsive even as memory grows large.

Editorial conclusion

VoiceMem is published under the Apache-2.0 license, written in Python, and the project states it will stay fully open source.