BookWith: a conversational ePub reader with a three-tier memory stack

The gap BookWith targets: readers that store text but not the thinking around it

Most ePub readers treat a book as a file to be paginated. Highlights land in a local store, notes sit next to them, and nothing connects the two to anything else you have read. BookWith's README lists four pain points in that model: you look things up yourself when a passage is unclear, you have no way to check whether you understood it, you cannot see links to earlier reading, and your annotations stay scattered. The project's answer is to put a chat panel beside the page and index the conversation, the highlights and the book text into the same retrieval layer. The intended user is not the casual reader. The README names four groups: researchers and graduate students working through papers, novel readers who want character relationships and historical context explained, students integrating multiple textbooks, and business readers turning a book's argument into an action plan. All four share one trait. They read more than one book on a subject and want the second book to know about the first.

The three-tier memory model is the actual design decision

The feature that separates BookWith from a chat box bolted onto a reader is its memory hierarchy, and the README is unusually specific about the boundaries. Short-term memory holds the latest 5 entries and is what keeps the conversation coherent when you turn a page. Mid-term memory summarizes every 20 entries, which is the mechanism meant to carry understanding across chapter boundaries without replaying the whole transcript. Long-term memory is vector search over all past dialogue, so retrieval is similarity-based rather than recency-based. A fourth layer, the user profile, is described as learning interests and preferences, though the README does not say what feeds it or where it is stored. The example the README gives is concrete: content from an economics book read a week earlier is linked to a marketing book being read now. That only works if dialogue from the first book was embedded and indexed at the time. The 5 and 20 thresholds are fixed numbers, not configurable values in the documentation, which means the summarization cadence is baked into the design rather than tuned per user.

What the stack looks like from the repository metadata

The topic list is the most reliable architecture signal available: TypeScript and React on the front end, Python and FastAPI on the back end, LangChain as the LLM orchestration layer, Supabase for storage and vector search, and epub-reader, rag, llm and openai as the remaining tags. That combination implies a specific data flow. The React client renders the ePub and sends the current page context plus the user's question to a FastAPI endpoint. LangChain assembles a prompt from the page text, the short-term buffer, the mid-term summary and whatever the vector search returns from Supabase. The response streams back into the chat panel and the exchange is written to the store, where it becomes eligible for the next summarization pass. Note the language split: the repository is tagged TypeScript as its primary language, but the backend and the LangChain work are Python. Anyone evaluating this needs competence in both, or at least the willingness to run a Python service alongside a Node build. The README also states that Supabase is the vector store, which means cross-book semantic search depends on Supabase's pgvector support rather than a dedicated vector database.

The podcast pipeline is a separate product with a separate dependency

AI podcast generation takes book content, extracts key points, and writes a host-guest dialogue script aimed at 5 to 10 minutes of listening. Audio synthesis uses Google Cloud Text-to-Speech multi-speaker output, which the README says supports Japanese and English and, more vaguely, multiple languages. This is the one feature with a hard third-party dependency outside the LLM provider. If you self-host BookWith and do not configure Google Cloud credentials, the reader and chat should still function but the podcast button will not. The README does not describe a fallback TTS path. It also does not say whether script generation and audio synthesis happen synchronously on upload or as a background job. That matters operationally: a 5 to 10 minute audio file takes real time to synthesize, and the README's phrasing that a generated podcast is immediately playable suggests the flow is designed to feel synchronous even if it is not.

Getting it running: what the README tells you and what it does not

This is where the documentation is thin, and it is worth being blunt about it. The README describes usage from the interface, not from the terminal. Step 1 is drag-and-drop an ePub file. Step 2 is select text, pick one of five highlight colors, add a Markdown note. Step 3 is ask questions in the right-hand chat panel or select text and choose Ask AI. Step 4 is semantic search across books, dialogue history and annotations. There is no install section in the supplied material. No clone command, no dependency install, no environment variable list, no docker compose file, no migration instructions for the Supabase schema. Given the stack, you can infer the shape of setup: a Node package manager for the React client, a Python environment for the FastAPI service, a Supabase project with the tables and vector index the code expects, and API keys for whichever LLM provider LangChain is pointed at plus Google Cloud TTS. Every one of those is an inference from the topic list, not a quote from the README. Treat the repository itself as the installation guide and budget time for it.

Where BookWith is the wrong tool

Three cases stand out. First, offline or air-gapped reading. The chat, semantic search and podcast features all depend on remote model providers, and the README describes no local model path. If your reading happens on a plane or inside a restricted network, you get an ePub renderer and nothing else. Second, privacy-sensitive material. Uploading a book pushes its text into prompts sent to the configured LLM, and the README does not discuss what is retained by the provider. For unpublished manuscripts or confidential documents, that is a decision you have to make before the first upload, not after. Third, anyone who wants a predictable dependency surface. There are no retrieved releases, which means no tagged version to pin. You would be tracking the main branch of a project whose backend, frontend and prompt logic all move together. The fixed 5-entry and 20-entry memory thresholds are a fourth, smaller concern: if a book's argument needs a longer conversational window than five exchanges, the summarization step is the only thing holding the thread, and the README does not describe how faithfully it preserves detail.

How it differs from Calibre plus an LLM plugin

The obvious alternative for a self-hoster is Calibre for library management with a chat plugin or a separate RAG script over exported text. The difference is in where state lives. Calibre's model is a library database of books and metadata, with annotations attached to a book. BookWith's model is a store of dialogue, annotations and embeddings that spans books, and the reader is one view onto it. That is why cross-book retrieval is a first-class feature here and an integration project there. The trade is control. With Calibre plus your own script you choose the embedding model, the chunking strategy and the prompt, and you can point it at a local model. With BookWith you inherit LangChain's abstractions, Supabase as the vector store, and the 5/20 memory cadence. If your goal is a reading environment that accumulates a searchable record of your thinking across a whole library, BookWith's architecture is aimed directly at that and Calibre's is not. If your goal is to keep every component swappable, the opposite is true.

Licence, maintenance and what to check before committing

BookWith is AGPL-3.0. For individual self-hosting that changes little. For anyone considering running a modified version as a network service, the AGPL's source-availability condition applies to users interacting with it over a network, which is a materially different obligation from a permissive licence. This is not legal advice; if you plan to host it for others, get your own. On maintenance, the metadata shows activity as of May 2026 and no retrieved releases. A project with no tagged versions means upgrades are a git pull plus whatever schema or prompt changes landed in between, and the README offers no migration notes. The practical cost is not the code, it is the operational surface: a Supabase instance, an LLM provider account with usage-based billing, and a Google Cloud project with Text-to-Speech enabled. Each of those is a separate thing to keep working. The sensible first step is to stand up the reader and chat against a single book with one highlight and one cross-book retrieval query, and confirm the vector search actually returns the earlier passage before you move a library into it.

Editorial conclusion

Adopt BookWith if you want a self-hosted reader where chat, highlights and cross-book retrieval share one store, and if you are willing to read the source to work out the deployment. Do not adopt it if you need a stable release, a documented API surface or a reader that runs without an LLM key. Before installing, verify three things in the repository: which Python entry point starts the FastAPI service, where the Supabase schema and vector index are defined, and which LLM and Google Cloud Text-to-Speech credentials the code reads at runtime.

Official sources

1. Issues
2. License: AGPL-3.0
3. README
4. shutootaki/bookwith on GitHub