BiliSum: local-first video summarization and RAG for Bilibili, YouTube and local files

The problem BiliSum addresses, and who actually has it

Long video is a poor medium for retrieval. A two-hour lecture or a multi-part Bilibili series contains material you will want to quote, cross-reference or search later, and none of that is possible from a video timeline. BiliSum's answer is a pipeline that converts video into text and then into structured artifacts: a transcript, a text note, an illustrated note, a mind map, and an entry in a local knowledge base that supports semantic search and RAG question answering. The README frames the target audience clearly enough: people who watch Bilibili heavily, with YouTube and local files supported alongside it, and who care that, in the project's own words, the data all lands locally. That last constraint is the real audience filter. If you are indifferent to where your transcripts live, there are hosted transcription services that require less setup. BiliSum exists for the case where the transcript, the index and the configuration are files on your own disk. The topics list on the repository also names Obsidian, which tells you the intended destination for the output is an existing personal knowledge base rather than a proprietary viewer.

How the pipeline is put together

The README's own diagram is the clearest description of the data flow: Bilibili, YouTube or a local video goes in, then transcription, then a text note, then an illustrated note, then a mind map, then the knowledge base. Transcription is the first stage and is pluggable. The README lists SiliconFlow ASR, multimodal ASR, local Whisper and FunASR (QwenASR) as options, which means the project does not ship a single bundled recognizer; you choose one and supply credentials or local compute. The illustrated note stage is the most distinctive part. In the VLM understanding mode, a vision model reads both the original note and extracted video frames, and the documentation states that images follow their corresponding knowledge paragraph rather than being collected at the end. The README also describes extracting objective facts from candidate frames and filtering low-quality images before selecting which ones to keep. Underneath, the stack is FastAPI and SQLite for the backend, Electron, React, TypeScript and Vite for the desktop client, and yt-dlp with ffmpeg for video acquisition and processing. The knowledge base layer is embedding retrieval plus an LLM agent, with the README noting that local embedding and local LLM are supported. Multi-part videos can be processed in batch or summarized as a whole series.

Three ways to run it, and what each one asks of you

The desktop application is the path the README recommends first. You download a Windows or macOS installer from GitHub Releases, and on first launch you configure one transcription service and one LLM in the settings page. The README adds a specific operational note here: when you hit Bilibili's risk controls, prefer the built-in QR code login. That is a concrete hint about where the friction lives. The CLI is a separate npm package, installed globally with npm install -g bilisum, after which commands like bilisum summarize "https://www.bilibili.com/video/BV1xxxx", bilisum brief and bilisum transcribe ./demo.mp4 --output transcript.txt are available. The CLI's default behaviour is worth understanding before you install it: it first tries to connect to a running desktop instance and share that instance's tasks, configuration and knowledge base. Only if no desktop app is running do you initialize a standalone environment with bilisum env setup, which requires Python 3.12, and then open settings with bilisum --setting. The Docker route is a single image, lycohana/bilisum:latest, run with port 3838 published, a named volume at /data, and an access token passed as VIDEO_SUM_ACCESS_TOKEN. The README points at docs/configuration.md for the LLM, ASR, knowledge base and vision model settings rather than listing environment variables inline, so that file is the one you need open while deploying. Building from source requires Python 3.12, Node.js 20 or newer and uv, then uv sync --python 3.12 --all-packages, npm install --prefix .\apps\desktop and npm run dev.

The agent skill is the least obvious integration point

BiliSum ships a skill named bilisum-video-understanding, installed with npx skills add https://github.com/lycohana/BiliSum --skill bilisum-video-understanding. The stated purpose is to let an agent choose between a full summary, a quick brief, a transcription or a task status query depending on the request. This is a different consumption model from the desktop app. Instead of a person clicking through a UI, an agent running in Codex, Claude or Cursor decides which BiliSum operation to invoke. The README directs readers to docs/skills/README.md for installation, verification and update steps written for the agent to read, and to docs/cli.md for the command set, environment handling and output formats. If you are evaluating BiliSum for an agent workflow rather than personal note-taking, the CLI output format documented in docs/cli.md is the thing to inspect first, because that is the contract your agent will parse. The README does not describe the output schema in the repository description itself, so this is a case where the linked document carries information the top-level README does not.

Where the design imposes real costs

The first cost is configuration. BiliSum does not include a hosted model. Every deployment path ends at the same requirement: you supply a transcription service and an LLM, and optionally a separate vision model and embedding model. The README says the vision model can be configured independently and supports OpenAI, Anthropic, compatible interfaces and custom endpoints, which is flexibility, but flexibility here means decisions. A reader who wants to paste a URL and receive a summary in one step will find the first-run experience longer than expected. The second cost is platform scope. The badges and the desktop section name Windows and macOS only. The Docker image and CLI widen that somewhat, but the README does not claim Linux desktop support, and Docker on port 3838 with a token is a server deployment, not a desktop replacement. The third cost is dependency on external video platforms. yt-dlp is listed as the acquisition layer, and the README's own advice about Bilibili risk controls and QR login indicates that authentication against Bilibili is a recurring operational concern rather than a solved problem. Finally, the knowledge base is only as good as its embeddings. The README states that local embedding is supported, which is the right choice for privacy, but it does not describe how retrieval quality is evaluated, and no evaluation harness is mentioned. Treat semantic search quality as something you will judge yourself after ingesting your own library.

How it differs from a hosted transcription and summary service

The obvious alternative is a hosted service that accepts a video URL and returns a transcript and summary, with the vendor managing the models and the storage. The difference is not the feature list; it is where the artifacts live and who holds the keys. A hosted service typically owns the transcript store and charges per minute or per request, and your ability to search across your whole history depends on that vendor's search interface. BiliSum inverts this. The FastAPI and SQLite backend writes to your machine, the README states that notes, indexes and configuration are stored locally, and the knowledge base is built from embeddings you generate with either a hosted API or a local model. The trade is setup effort and operational responsibility for a self-hosted store. There is also a narrower comparison inside the project itself: the README mentions that local videos can optionally connect to Twelve Labs Pegasus to supplement visual information beyond subtitles. That is a third-party hosted model, which is a deliberate exception to the local-first posture and worth noting if you are choosing BiliSum specifically for data locality. Using it means sending frames or video to an external service, so the local-first claim applies to the pipeline as a whole, not to every optional component.

Maintenance, release cadence and licence

The repository is not archived, the default branch is master, and the last push recorded is 2026-09-10. Releases v1.21.0 and v1.21.1 landed within about three weeks of each other in August and September 2026, with a v1.21.0-alpha.1 preceding the stable build. That pattern suggests an active project with a pre-release channel, though the supplied material does not include a changelog, so what changed between versions is not something this review can state. The practical maintenance burden for an operator is the moving parts rather than the code: yt-dlp needs updating as platforms change their delivery, ffmpeg needs to be present, and the ASR and LLM endpoints you configured are external dependencies with their own availability and pricing. On licence, BiliSum is MIT, copyright 2026 Lycohana. MIT is permissive and permits commercial use and modification, but it ships without warranty, and this review is not legal advice. Two licence-adjacent points deserve attention if you plan to redistribute or deploy this: the Docker image bundles third-party components whose own licences apply, and any hosted model you connect (Twelve Labs, or a commercial ASR and LLM endpoint) is governed by that vendor's terms, not by BiliSum's MIT grant.

Editorial conclusion

Adopt BiliSum if you already have API keys or local models for ASR and an LLM, you watch long-form Chinese or English video regularly, and you want the resulting notes and embeddings to stay on your machine. Do not adopt it if you want a hosted service that works without any model configuration, or if your videos are short enough that reading the transcript yourself is faster than the setup. Before committing, verify two things: that your chosen ASR path covers the languages in your library, and that the export format matches your note tool. The README documents Markdown and Obsidian export, so check that against whatever you actually use.

Official sources

1. Issues
2. License: MIT
3. lycohana/BiliSum on GitHub
4. README
5. Releases