claude-video-vision: a perception layer that hands Claude frames and transcripts

The gap this fills: Claude Code has no eyes for video files

Claude Code reads text files, diffs, and terminal output. Hand it an mp4 and there is nothing to read. The README frames the plugin as giving Claude the ability to watch and understand videos, and the mechanism is deliberately narrow: ffmpeg pulls frames, an audio backend produces a timestamped transcript, and Claude receives both. The README calls the plugin a perception layer, not an interpretation layer. That sentence is the whole design argument. The project does not try to summarize a video for you; it converts a video into something Claude can already reason over, and lets the existing model do the rest. The audience is people who debug from screen recordings, review a tutorial someone sent them, or want to ask a question about a clip without leaving their editor. If you are building a video pipeline that runs unattended, this is the wrong shape, because it is built around a human asking Claude a question.

Frames and audio travel on separate paths and meet in Claude

The architecture diagram in the README shows a single MCP tool, video_watch, reached from a skill called video-perception, which is in turn reached from the /watch-video slash command. Inside the Node.js MCP server, frame extraction and audio processing run on parallel branches. Frames come out of ffmpeg as base64 images with timestamps. Audio comes out as a transcription plus audio events. Both are handed back to Claude together. The README is explicit that all backends extract frames via ffmpeg and that Claude always has direct visual access, so switching audio backends never removes the visual channel. Six MCP tools are exposed in total: video_watch for the main path, video_analyze to probe structure with ffmpeg filters before extraction, video_detail to drill into a cached or freshly extracted moment, video_info for metadata without processing, video_configure for settings, and video_setup to check dependencies. The split between video_analyze and video_watch matters. Probing structure first is how the system avoids extracting a hundred frames from a one hour lecture when the question is about a ten second segment.

Adaptive extraction is a prompt-shaped contract, not a config file

The README's usage examples show Claude adjusting fps, time range, and resolution from the phrasing of your question. The first second of a file means original fps between 00:00:00 and 00:00:01. Summarizing a one hour lecture means low fps across the full duration. Asking what text appears on screen at 1:30 means high resolution over a narrow window. This is a real mechanism and also a real risk. The parameters are chosen by the model, so the same question phrased two ways can produce two different extraction plans, and nothing in the README describes a deterministic override at the slash command level. The config file sets the boundaries: max_frames caps the count at 100 by default, default_fps is auto, frame_resolution defaults to 512, and frame_format accepts jpeg, png, or webp. The README notes jpeg is the default for backwards compatibility and that png is useful for screen recordings where text and sharp UI edges should stay lossless. If you are reading code off a screen recording, that png note is the single most useful line in the configuration section.

Install and configure: the commands the README gives

Installation happens inside Claude Code, one command at a time. First /plugin marketplace add https://github.com/jordanrendric/claude-video-vision, then /plugin install claude-video-vision. The README states the MCP server auto-installs via npx from npm on first use, so there is no build step. For local development the alternative is git clone followed by claude --plugin-dir /path/to/claude-video-vision. Configuration runs through /claude-video-vision:setup-video-vision, an interactive wizard covering backend selection, whisper configuration, frame options, and dependency verification. Settings land in ~/.claude-video-vision/config.json. The keys that matter most are backend, whisper_engine, whisper_model, frame_mode, frame_format, frame_resolution, default_fps, max_frames, frame_describer_model, enable_index, session_max_age_days, and downloads_max_age_days. Whisper models download automatically to ~/.claude-video-vision/models/ on first use, with tiny, base, small, medium, large-v3-turbo, large-v3, and auto available, where auto picks the best fit for your RAM. Requirements are Node.js 20+, ffmpeg, and yt-dlp only if you pass YouTube URLs. Cache and download directories are pruned by the two max_age_days keys, both defaulting to 7.

Three audio backends with genuinely different trade-offs

The backend table separates them cleanly. Gemini API processes audio natively, including non-speech events, with a documented free tier of 1500 requests per day, and needs GEMINI_API_KEY. Local Whisper runs whisper.cpp or Python openai-whisper, is free and fully offline, and needs brew install whisper-cpp on macOS plus the automatic model download. OpenAI's API transcribes through its Whisper endpoint, is paid per usage, and needs OPENAI_API_KEY. The distinction that matters is the non-speech events column. Gemini is the only backend the README credits with hearing things that are not speech, which is exactly what you want when the question is whether an error sound plays or whether a door slams. Local Whisper is the only option that keeps the file on your machine, which for a bug report from a customer machine may be the deciding factor rather than a preference. The README's status section says v1.0.0 was tested on macOS with Apple Silicon using the local backend, so the local path is the one with documented mileage and the API paths are the ones you would be validating yourself.

YouTube transcripts carry a provenance label, and it is load-bearing

For YouTube URLs the server downloads with yt-dlp and follows a stated transcript order: manual YouTube subtitles when an English track exists, then YouTube automatic captions when manual subtitles do not, then the configured audio backend when captions are missing, empty, or cover too little of a longer video. Results are tagged with transcription_source, for example youtube_subtitles or youtube_auto_captions. The README says this lets Claude treat manual subtitles as stronger evidence than auto-captions. That is a sensible design and a narrow one. Auto-captions on technical speech mangle product names, function names, and numbers, and no amount of provenance labelling repairs the underlying text. If your question depends on an exact identifier spoken in a video, the caption path is the weak link and the audio backend fallback is the one you want to force. The README does not document a config key that forces the backend over captions, so that choice appears to be the server's.

Where the perception layer stops being enough

The honest limitation is the one the project names itself. A perception layer returns images and a transcript, and everything downstream is the model reading those images. Frame extraction samples; it does not observe continuously. Ask what happens between two extracted frames and the answer is inference, not measurement. There is no bounding box, no motion vector, no per-frame timestamp you can cite as a fact about the file. max_frames defaults to 100, which bounds how much of a long video can be seen at all, and the README gives no guidance on what a 100 frame sample of a two hour recording actually covers. The status section is equally plain: v1.0.0, initial release, tested on macOS with Apple Silicon and the local backend. Windows and Linux are not mentioned. There are no retrieved releases beyond that initial version, so there is no upgrade history to read. For frame-accurate work such as measuring a render time or verifying a specific frame index, this tool gives you a model's description of a sampled image, and that is a different kind of artifact.

Against a transcription-first pipeline such as Whisper plus a summarizer

The obvious alternative is the one most teams already have: run Whisper over the file, feed the transcript to a language model, skip video entirely. That pipeline is cheaper, faster, and produces a text artifact you can grep. It also cannot see anything. A screen recording of a broken UI transcribes to silence or to someone saying it looks wrong, and the actual failure is a red banner that never gets mentioned. claude-video-vision's difference is that frames travel alongside audio on a parallel branch and reach Claude as images regardless of which audio backend is configured, so the visual channel survives even when the audio is uninformative. The cost is the reverse: a video with no speech still burns frame extraction, and a pure audio question pays for images it does not need. A second alternative is the hosted multimodal video APIs, which accept a video file directly and handle sampling internally. Those remove the ffmpeg dependency and the local model download, but they also remove the choice of backend, the local-only path, and the ability to point Claude at a file on your own disk through a plugin you control.

Maintenance cost and the MIT licence in practice

The dependency surface is what you are actually maintaining: Node.js 20 or newer, ffmpeg on PATH, optionally yt-dlp for YouTube, optionally whisper-cpp with a downloaded model, and an API key if you chose a cloud backend. ffmpeg and yt-dlp both break against upstream changes, and yt-dlp in particular tends to need updating when sites change their delivery. The plugin itself installs through npx from npm, so version pinning is not described in the README and you should check what npx resolves to if reproducibility matters. Whisper models live in ~/.claude-video-vision/models/ and sessions and downloads are pruned after 7 days by default, which keeps disk use bounded without any action from you. The project is MIT licensed, which permits commercial use, modification, and redistribution provided the copyright notice and permission notice are retained. That is a summary of the licence text, not legal advice; if you are redistributing the plugin inside a product, read LICENSE and check how it interacts with the licences of the models and APIs you pair it with, since MIT covers this repository and not the Whisper weights or any vendor API terms.

Editorial conclusion

Adopt this if you already work inside Claude Code and want video treated as another readable input, and if you are willing to run ffmpeg plus a Whisper model or hold an API key. Skip it if you need frame-accurate measurement, long-video coverage, or anything you can cite as evidence rather than as a model's reading of an image. Before installing, run /claude-video-vision:setup-video-vision and check that ffmpeg and, for YouTube URLs, yt-dlp resolve on your PATH, then confirm which transcription_source the audio step reports on a sample clip, because that label decides how much weight the transcript deserves.

Official sources

1. Issues
2. jordanrendric/claude-video-vision on GitHub
3. License: MIT
4. Project website
5. README