UniVA: a video agent you direct in conversation

What the system claims to do, and the two words in its subtitle

UniVA is an open-source, next-generation video generalist system that enables you to plan, compose and produce videos through natural language instructions. The README puts it plainly: UniVA acts as an intelligent video director, iterating shots and stories with you through an agentic, proactive workflow.

The title of the project is Universal Video Agent, and the subtitle is Towards Open-Source Next-Generation Video Generalist. Both halves matter. Generalist is a claim about scope, that this is not a model tuned for one narrow kind of edit, and the accompanying artefacts back that up: there is a paper on arXiv, a benchmark dataset on Hugging Face, a leaderboard space, a live demo, a project page and a Discord for discussions.

The feature list is grouped in three. The first is agentic creation, with multi-round co-creation where you talk like a director and UniVA iterates shots and stories with you, deep memory and context split into global and user memory to keep preferences, lore and styles consistent, implicit intent reading for vague and evolving instructions with less prompt hacking, a proactive agent that plans, checks and suggests better shots rather than only obeying, and an end-to-end workspace in which UniVA plans, calls tools and delivers full videos.

The second group is the production engine: a universal video fabric taking text, images, entities and video into controllable video in one framework, an any-conditioned pipeline that claims super HD and consistent, cinematic quality with stable identity and objects, complex narratives of multi-scene, multi-role, multi-shot stories under structured control, ultra-long and fine-grained editing from long-form cuts down to per-shot and per-object refinement, and generation and edits guided by long-video comprehension and segmentation.

The third is extensibility, described as MCP-native with a modular design that is easy to extend with new models and tools, and production-ready video generation capabilities.

It is worth reading those claims as a research agenda rather than a product specification, given that the repository has no tagged releases and the last push was on 2026-05-09.

The two-agent split and the FastAPI surface

The architecture section describes two components, a Python backend and a Next.js frontend, and the backend is where the interesting structure is.

The backend has four parts. The Plan Agent does high-level planning and task decomposition. The Act Agent executes specific video generation tasks. The MCP Tools are modular tools for video processing, generation and editing. And a FastAPI Server provides the RESTful API for client communication.

Separating planning from acting is the pattern that makes the memory feature possible. The plan agent is the one that holds the shape of the story across rounds, while the act agent is the one that calls a tool, waits for a result and comes back. If both lived in one loop with one prompt, the context would grow without bound and the system would lose track of the thread.

The frontend has four parts to match: a chat interface for talking to the system, a timeline-based video editor, project management to save and manage video projects, and authentication with user management and access control.

The dependency list in the pyproject tells you what the backend is built from. It needs Python 3.10 or later, and it pulls in httpx for HTTP, prompt-toolkit for the terminal interface, click for the CLI, toml, tabulate and wcwidth for the console output, rich and pyperclip for terminal handling, the mcp package, aiofiles, loguru for logging, fastapi and uvicorn for the server, pydantic for validation and apscheduler for scheduling.

The two requirements files are worth distinguishing. requirements_simple.txt is what the README tells you to install for a quick start, and requirements.txt is the full pinned set, which runs to hundreds of entries and includes the heavyweights: accelerate, agno, the langchain and langgraph stack, librosa and numba for audio, moviepy for video handling, matplotlib, decord and imageio for decoding, huggingface-hub, plus a set of nvidia CUDA 12 wheels including cudnn, cufft, curand, cusolver, cusparse, nccl and nvjitlink. Those CUDA packages tell you the project expects a GPU environment for the local model path.

MCP as the extension seam, and how the servers are wired

The MCP-native claim is concrete, and the configuration file shows it.

The MCP servers are configured in a JSON file, univa/config/mcp_configs.json, and the shape is the standard one:

{
  "mcpServers": {
    "video-tools": {
      "command": "python",
      "args": ["-m", "univa.mcp_tools.video_server"],
      "env": {}
    }
  }
}

The server entry launches a Python module, univa.mcp_tools.video_server, which is one of the MCP tools the backend talks to. The env object is where per-server credentials would go, which is the natural place for a key that should not sit in the main configuration file.

This is the design decision that carries the most weight in the whole project. The generation and editing capabilities are not built into the agent loop, they are MCP tools that the act agent invokes, and a tool is a process with its own configuration. That means adding a new video model or a new editing backend does not mean changing the agent, it means adding an MCP server and pointing the config at it.

The cost is the same thing. Every capability you want has to exist as a tool, and the tools that call hosted services need credentials, which is why the environment file has a dedicated section for tool API keys.

The tool configuration in the environment file names two providers. There is an OpenAI key for tools that use LLMs, such as a query_llm tool, and a Wavespeed key for generation tools covering image, video and audio. The comment in the file is explicit about the split, and it is a useful one, because a query_llm tool and a video generation tool are different kinds of call with different cost and different latency characteristics.

Configuration, environment overrides, and the local model path

Configuration is layered, and the layering is documented in a way that saves debugging time.

You copy the example environment file to create your local configuration, and the variables in it override the defaults in univa/config/config.py and univa/config/mcp_tools_config/config.yaml. That note appears in both the README and the example file itself, which is a good sign for a project where configuration sprawl is the usual failure mode.

The core agent models are configured as provider, model id and API key triples, separately for the plan agent and the act agent:

# Plan Agent (High-level reasoning)
PLAN_MODEL_PROVIDER=openai
PLAN_MODEL_ID=gpt-5
PLAN_MODEL_API_KEY=your-api-key

# Act Agent (Execution & Tool use)
ACT_MODEL_PROVIDER=openai
ACT_MODEL_ID=gpt-5
ACT_MODEL_API_KEY=your-api-key

The provider field is not fixed to one vendor. The README names OpenAI, DeepSeek, Qwen and local models as options, and the example file also carries a base URL and an extra parameters field per agent, so an OpenAI-compatible endpoint works as well as a named provider.

Local model paths are the optional third option, and they are absolute paths to checkpoints for video editing and video understanding, with Wan2.1 and Qwen2.5-VL named as the examples. The README says these override the default settings without needing to modify code, which is the difference between a local model and a hard-coded one.

The system settings section is short: authentication can be enabled with an admin access code, session timeout is in minutes with a default of 60, and there is a maximum concurrent sessions per user set to 10. There is also a commented proxy configuration, which matters for anyone behind a network where the provider endpoints are not directly reachable.

The example file is worth reading for its own structure, because it shows what the project thinks is a first-class setting: models, system, MCP tools with shared keys, and local model paths, each in its own labelled block.

Running it, and the two ways to talk to it

The installation starts with prerequisites: Python 3.10 or higher, Node.js 18.0 or higher and Bun 1.2.18 or higher if you are using the web frontend, and CUDA recommended for GPU acceleration, described as optional but recommended.

Then the repository is cloned and the dependencies installed:

git clone https://github.com/univa-agent/univa
cd univa

pip install -r requirements_simple.txt

Or, using the project configuration:

pip install -e .

The frontend side is separate and optional. The README is explicit that if you only need the backend API you can skip the frontend section, which is the sensible read for anyone embedding the API. The frontend uses bun install, and the root package.json confirms it, with the package manager pinned to [email protected] and workspaces over apps/ and packages/.

There are two ways to use the backend. The first is a command line interface, with no web interface:

python univa/univa_agent.py

That starts an interactive session in the terminal, and it is the fastest way to evaluate the agent, since it removes the frontend, the API and the authentication from the loop.

The second is the server, for the web interface or API access:

cd univa
python univa_server.py

And the backend API is then available at http://localhost:8000. There is a health endpoint to check it, and the README says you should receive a response indicating the server is healthy:

curl http://localhost:8000/health

The health check being a documented step rather than an afterthought is worth noticing, because with a video agent the failure modes are not obvious from the outside. The model calls can fail, the MCP tool processes can fail to start, and a health check that only proves the process is up is a weak signal. What the health endpoint actually verifies is not described, so it is worth knowing before you rely on it in a deployment.

The frontend stack, and what this repository is still becoming

The JavaScript side tells you what the web interface is built from. Next.js at 15.3.4 or later is the framework, with undici for HTTP, formidable for file uploads, wavesurfer.js for audio waveform display and editing, and uuid for identifiers. The devDependencies are turbo for the task pipeline, TypeScript 5.8.3, Biome through the ultracite wrapper for linting and formatting, husky for hooks, and the formidable and uuid type packages.

Two details are worth a note. The trustedDependencies list contains only the Tailwind CSS oxide package, which is the mechanism for allowing a postinstall script to run, and it is scoped to exactly one dependency. And the lint and format scripts invoke npx ultracite@latest rather than a pinned local binary, so the linter version is whatever is current at the time you run it. For a repository that is a work in progress, that is a reasonable trade, but it does mean a formatting change can appear without a commit that touched the code.

The repository layout matches the two-component architecture. There is an univa/ directory for the Python package, apps/ and packages/ for the JavaScript workspaces, a pyproject.toml, the two requirements files, a turbo.json for the pipeline, a biome.jsonc, a tsconfig.json, an .env.example, and a bun.lock. The top level also has .npmrc, which is the file that keeps npm from interfering with the bun workspace.

On the state of the project, the honest reading is of a research release. The repository description calls it the official code repo for UniVA, the licence is MIT, and the artefacts around it, a paper, a benchmark, a leaderboard and a demo, are the kind of evidence a research project publishes alongside its code. There are no GitHub releases, so the code is the artefact rather than a versioned distribution, and the last push was on 2026-05-09.

For a team evaluating it, the practical entry point is the command line interface with hosted models, because that is the configuration with the fewest moving parts, and the MCP configuration is the file to read first if you intend to add a capability of your own.

Editorial conclusion

Adopt UniVA if you are building an application that generates or edits video from natural language and want the planning, tooling and memory layer already in place rather than writing an agent loop yourself. Do not adopt it expecting a hosted API, since the whole thing is a repository you clone, configure with your own API keys and run, and the model calls cost money at whatever provider you point it at. Before you start, read the configuration notes, because the environment file overrides the defaults in the Python config and the MCP YAML, and decide which model providers and generation tools you are willing to pay for before you wire the MCP servers.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. UniVA-Verse/univa on GitHub