video-analyzer: Frame-by-Frame Video Description with Whisper and a Vision LLM
Analyze videos using LLMs, Computer Vision and Automatic Speech Recognition
At a glance
- What is it?
- byjlw/video-analyzer turns a video file into a JSON description by extracting key frames, transcribing audio with Whisper, and captioning each frame with a vision model. It runs locally through Ollama or against any OpenAI-compatible endpoint, and the design is simple enough to read in one sitting.
- Who is it for?
- Adopt video-analyzer if you want a local, no-API-key pipeline that produces a JSON description of a video and you are willing to accept a frame-by-frame captioning model as the source of truth. Do not adopt it if you need per-object tracking, timestamps tied to specific events, or anything that looks like structured scene segmentation; the output is prose, not an index.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 149 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap video-analyzer fills: turning a video into text you can grep
Most video tooling produces more video. Cutting, encoding, transcoding: the output is another media file. video-analyzer produces text. It takes a video file and emits a JSON document containing metadata, an audio transcript when one can be produced, a frame-by-frame analysis, and a final natural-language description of what happens in the clip.
The intended user is someone who needs a searchable or summarizable record of video content without watching it. The README frames the tool around combining a vision model (Llama 3.2 11B Vision by default) with Whisper, where frame details and the transcript are merged into a single description. That is the whole pitch, and it is narrow on purpose. This is not a video editor, not a detection framework, and not a timestamped annotation tool. It is a captioning pipeline with a transcript bolted on.
If your problem is 'I have forty screen recordings and I need to know which ones mention the deployment step', this is aimed at you. If your problem is 'I need bounding boxes for every car in this traffic footage', it is not.
Three stages: OpenCV frames, Whisper transcript, vision model captions
The design documentation describes three stages, and the ordering matters.
Stage one is frame extraction and audio processing. OpenCV pulls key frames from the video. Whisper transcribes the audio, and the README mentions confidence checks to handle poor quality audio, with automatic handling of poor audio listed as a feature. Stage two analyzes each frame with the vision LLM. The detail worth noting here is that each frame analysis includes context from previous frames, so the model is not captioning in isolation; it is building a chronological progression. The prompt template driving this is frame_analysis.txt.
Stage three reconstructs the video. Frame analyses are combined in order, the audio transcript is integrated, and the first frame is used to set the scene. The output is the final description.
This sequential, context-carrying approach is the interesting design decision. Feeding prior frame context into each subsequent caption is what lets the output say 'the person remains facing away from the camera' rather than describing every frame as if it were the first. The cost is that errors compound: if frame three is misread, frames four through N inherit that misreading. There is no mechanism described in the material for correcting a bad early caption.
Getting it running: pip, FFmpeg, and one ollama pull
Installation is conventional Python. Clone the repository, create a virtual environment with python3 -m venv .venv, activate it, then pip install . for a regular install or pip install -e . for development. FFmpeg is a hard requirement for audio processing, installed via apt-get, brew, or choco depending on platform.
For local inference you need Ollama. The documented sequence is to install Ollama, run ollama pull llama3.2-vision, then ollama serve. After that, the quick start is a single command: video-analyzer video.mp4.
The system requirements are the real constraint. Python 3.11 or higher. When running LLMs locally, at least 16GB RAM with 32GB recommended, and a GPU with at least 12GB VRAM or an Apple M Series machine with at least 32GB. Those numbers are in the README, and they are the first thing to check against your machine before installing anything.
If local inference is not viable, the tool accepts any OpenAI-compatible endpoint. The documented flags are --client openai_api, --api-key, --api-url, and --model. For OpenRouter the example uses --api-url https://openrouter.ai/api/v1; for OpenAI, https://api.openai.com/v1. The README notes that on OpenRouter you can append :free to a model name, with meta-llama/llama-3.2-11b-vision-instruct:free given as the example.
Configuration cascades, and that is where surprises live
The README states the tool uses a cascading configuration system: command line arguments take highest priority, then user config in config/config.json, then the default config. The same settings can be expressed either as flags or as JSON. The OpenAI-compatible example shows the JSON shape, with a clients object containing a default key and a nested openai_api object holding api_key and api_url.
Cascading config is convenient until it is not. The failure mode is a config/config.json that sets one client while a flag on the command line sets another, or an api_url left pointing at OpenRouter while the model name is an OpenAI one. Nothing in the supplied material describes validation or a conflict warning, so the practical advice is to pick one layer and stay in it. If you use config/config.json, do not also pass --client and --api-url.
There is also a --prompt flag for custom instructions, shown in the quick start with --prompt "What activities are happening in this video?", and a --whisper-model flag with large given as the example value. Whisper model size is a direct trade between transcription accuracy and runtime, and the material does not quantify that trade, so treat it as something to measure on your own clips.
The output is prose, and that is a real limit
The tool writes output/analysis.json containing metadata, the audio transcript if available, frame-by-frame analysis, and the final description. The sample output quoted in the README is a paragraph of flowing description: a person with long blonde hair in a pink t-shirt and yellow shorts standing in front of a black plastic tub on wheels, ground covered in wood chips, and so on. A fuller sample lives at docs/sample_analysis.json.
That format is the limitation. There are no timestamps attached to described events in the sample, no object list, no confidence scores per claim. If you need to jump to the moment a specific thing happens, the description tells you it happened but not when. The frame-by-frame section may carry more structure, but the material does not show its schema, so you cannot plan an integration around it without reading the actual file.
The second limitation is cost and latency scaling. Every extracted key frame is a vision model call, and each call carries context from prior frames. A long video means many calls with growing context. The README does not state how key frames are selected or how many are extracted, which is the single most important number for predicting runtime and spend on a cloud endpoint. That omission is worth flagging: 'intelligent key frame extraction' is asserted as a feature, but the selection criteria are not described in the material provided.
The third is audio. Whisper transcription is described as handling poor quality audio with confidence checks, but a video with no usable speech yields no transcript, and the description then rests entirely on visual captions.
video-analyzer-tune, and how it differs from captioning frameworks
The project ships a companion package, video-analyzer-tune, published separately and installed with pip install video-analyzer-tune. The workflow is specific: run video-analyzer on a few representative videos, edit the outputs by hand to show what ideal results look like, then let DSPy MIPROv2 search for better prompt instructions. Tuned prompts are written as new files that you point to through your config, so the main package is untouched. This is a sensible separation, and it acknowledges something honest: the default prompts are a starting point, not a finished product. If you care about output quality on a specific genre of video, the tuning path is where that work happens, and it costs you a labeling pass over sample outputs.
As an alternative, consider a general vision-language pipeline you assemble yourself, for example OpenCV for frame sampling plus a direct call to the same Llama 3.2 Vision or GPT-4o model. The difference is not the model, it is the orchestration. video-analyzer gives you the sequential context passing, the Whisper integration, the prompt templates, and the JSON envelope already wired together. Building that yourself is perhaps a day of work, and you would control frame selection and output schema exactly. The reason to take the dependency instead is that you get a working baseline and a documented tuning path without designing the pipeline first. The reason to skip it is that you want a schema other than prose, in which case you are fighting the tool rather than using it.
Maintenance, licensing, and what to check before you commit
The project is Apache-2.0 licensed, which permits commercial use and modification with the usual attribution and notice obligations. It is not archived. Recent releases include v0.1.2 and a separate tune-v0.1.0, both dated 2026-03-20, with an earlier release in 2025. The version numbers are still in 0.1.x territory, which tells you the API surface and output schema should be treated as movable. If you build an integration on output/analysis.json, pin the version and expect to re-read the schema on upgrade.
Upgrade cost is dominated by two moving parts that are outside the project's control: the vision model you point it at, and Whisper. Swapping llama3.2-vision for a different model changes output style and may require re-tuning prompts through video-analyzer-tune. Swapping the Whisper model size changes runtime. Neither is a code change, but both are re-validation work.
The dependency footprint is ordinary for this kind of tool: Python 3.11+, FFmpeg, and either a local Ollama service or network access to an OpenAI-compatible API. Uninstalling is pip uninstall video-analyzer, though that leaves your config/config.json and any output files behind.
Before adopting, verify the frame extraction behavior on your own footage, because the material does not document how many frames are pulled or by what criteria, and that number drives both latency and cloud API cost more than any other setting.
Editorial conclusion
Adopt video-analyzer if you want a local, no-API-key pipeline that produces a JSON description of a video and you are willing to accept a frame-by-frame captioning model as the source of truth. Do not adopt it if you need per-object tracking, timestamps tied to specific events, or anything that looks like structured scene segmentation; the output is prose, not an index. Before committing, verify two things on your own hardware: that your GPU or Apple Silicon memory actually fits the vision model you intend to pull, and that the cascading config resolves the way you expect when both config/config.json and command line flags are present. Then run it once on a representative clip and read docs/sample_analysis.json next to your own output to see how far the description drifts.
Community notes