wandbot is a W&B documentation bot that publishes its own accuracy table, regressions included

The accuracy table records its own regressions

The evaluation table is the most substantive thing on the page, and it does not flatter the project.

The English results start at 53.8% for the 1.0.0 baseline, move to 72.5% for 1.1.0, described as the version that spent the longest time in production, and 81.6% for 1.2.0. The 1.3.0 release candidates and releases then sit between 88.8% and 91.2%, before the last two rows fall.

The final two rows are the interesting ones. Version 1.3.2 scores 90.4% on index v54. Version 1.3.3 scores 82% on index v61 with 500-token thinking enabled on the flash model and the lite model held stable.

Every recent row carries five trials, while the three earliest carry one, which matters when reading the trend: the early numbers are single samples.

The rows are not bare numbers. Each links to a Weave evaluation holding the traces, and from 1.3.0 onward each also links to a data ingestion report for the specific index version used. So a reader can go from a percentage to the individual calls that produced it, which is the difference between a claim and a measurement.

The honest summary is that the pipeline improved substantially over its first three versions and has since wobbled, with the newest configuration 9 points below the best.

The same system scored 71.3% and 88.8% depending on the judge

Two rows in the table describe the same 1.3.0 release candidate with different judges, and the gap is 17.5 points.

With a gpt-4-preview judge the score is 71.3%. With a gpt-4o judge on the identical system it is 88.8%. The table says so explicitly: these are the same wandbot system evaluated with different judges.

That single fact bounds how much any of the other numbers mean. A judge model is part of the measurement apparatus, and moving from one frontier model to a newer one changed the score by more than the entire gap between the baseline and the 1.2.0 rewrite.

The notes below the table are where the project does its own quality control. The roughly 2.5% improvement between the release candidate and production is attributed mostly to the newer reranker and the newer flash model, and then qualified: evaluations before the production run had ten to twelve errors out of four hundred and ninety total calls, so there might be noise in the results.

Publishing the error count alongside the score is the detail that makes the rest credible. A table of percentages with no denominator discipline is marketing; a table that tells you four hundred and ninety calls contained a dozen failures is reporting.

A 91.2% score is held across a model swap and a smaller corpus

The 1.3.0 production row and the 1.3.1 row both report 91.2%, and the notes describe what changed between them.

Version 1.3.1 switched the response synthesizer to Claude Sonnet 3.7, updating from a dated GPT-4o build. So the number did not move even though the model composing the answer was replaced outright.

It also changed the corpus, and not by adding to it. The updated index excludes the Korean and Japanese versions of the documentation, and also excludes the blog posts from Fully Connected.

Both facts belong in the same sentence as the score, because a fixed number across a model swap and a smaller corpus could otherwise be read as robustness or as stagnation, and the two have different meanings. Removing pages makes the retrieval task easier, so part of the flat result is a smaller problem rather than a better answer.

The Japanese table gives the other side of that trade. Version 1.2.0 scored 56.3% response accuracy, and adding a translation process took 1.2.1 to 71.9%.

So the project's answer to non-English documentation is a translation stage, and the accuracy difference is large enough to explain why the index later dropped those language variants entirely.

Three models do the work, with a fallback for selection

The feature list is short and specific about which model does what.

A hosted Chroma vector store holds the corpus, with OpenAI's third-generation embeddings. Gemini flash-2.0 handles query enhancement, meaning the rewriting of a user's question before retrieval. GPT-4o performs response synthesis, meaning the writing of the answer. Cohere's reranking model sits between retrieval and generation.

That is four model calls in a chain with three vendors, and the chain has a documented history: query expansion moved from GPT-4o to the flash model, the reranker moved from version 2.0 to version 3.5, and the vector store moved from locally hosted Chroma to hosted.

The feature list also names a fallback mechanism for model selection, which is the part with the most operational value and the least documentation in the visible text. If one vendor in the chain is unavailable, something decides what happens instead, and a reader would want to know what.

One older note is worth recording because it changes the reading of the numbers above: web search was turned off in the 1.3.0 release, listed as for now. So the bot answers from its index alone, and a question outside the corpus has no second source to fall back on.

The RAG pipeline was split into three named modules

The 1.2.0 release is the architectural one, and its notes describe a rewrite rather than an addition.

The pipeline was divided into three modules: query enhancement, retrieval, and response synthesis. The stated reason is clarity and maintenance, which is the right reason for a refactor nobody asked for.

Several capabilities arrived with the split. Parent document retrieval was added inside the retrieval module, which returns the containing document rather than only the matching fragment. Sub-query answering was added to response synthesis, so a complex question can be broken into parts, answered, and recombined.

The retrieval stack changed twice in that release. FAISS was replaced with ChromaDB, for metadata filtering and speed. And the llama-index dependency was replaced with what the notes call the LECL, which enabled parallel LLM calls, described as a direct efficiency win since query expansion no longer blocks on a single call.

The query enhancer was also simplified to operate with a single LLM call, which is the opposite direction from the parallel calls and suggests the pipeline was tuned rather than simply maximised.

The API was restructured at the same time into separate routers for retrieval, database, and chat operations, which is what makes the later removal of the retriever endpoint a router-level change rather than a redesign.

The index has its own version lineage, and the corpus is re-ingested on a schedule

The index version is tracked separately from the release version, and it moves faster.

The evaluated configurations use index v50, then v52, then v54, then v61. Three of those correspond to releases in the table, and the last one is paired with a configuration change rather than a code change.

The update mechanism is periodic data ingestion with generated reports, and each report is linked from the evaluation row that used it. The reports are named after the index version and the ingestion run, so a specific corpus state can be named in a bug report.

That separation is the practical point. A support bot's answers depend on two things that move independently: the code and the corpus. Publishing both identifiers against every accuracy figure means you can tell whether a drop came from a pipeline change or from a documentation change.

The release notes reinforce it, marking two rows as involving a knowledge base update rather than only a code change.

For anyone running a similar bot, the pattern is worth copying regardless of the specific tools: name the corpus build, publish a report for it, and attach both to the quality number.

The 1.3.0 release removed things as well as adding them

Several entries in the 1.3.0 notes are subtractions, and they are the ones that describe the current shape.

Web search was turned off, marked as for now. The retriever endpoint was removed, also marked as for now, and a new endpoint was added in its place. Most LangChain dependencies were removed. All configuration was moved into a configs folder, which is a small change that makes a multi-file config reviewable.

In the other direction, the release added explicit LLM and EmbeddingModel classes, meaning the model clients stopped being generic calls and became typed interfaces. That is what a fallback mechanism for model selection needs in order to exist at all.

The dependency list tells you how far the LangChain removal actually went. LangChain and LangChain core are still declared as runtime dependencies, along with the community package. So the release note describes an intention and the manifest describes the current state, and for anyone reproducing the environment the manifest is authoritative.

Two more changes are worth naming. Package management moved to uv, and the project moved to Python 3.12.

The Python range is one minor version and four packages are exact

The packaging is where the operational constraints are.

The supported interpreter range is greater than or equal to 3.12 and less than 3.13. That is a single minor version, which is stricter than most projects and means a 3.13 upgrade will need a deliberate release rather than an incidental resolution.

Four dependencies are pinned exactly rather than with a floor. ChromaDB is pinned to 1.0.3, which matters because the vector store client has had breaking interface changes across its 1.x line. Tree-sitter is pinned to 0.21.3 and protobuf to 4.25.3, a pair that is usually constrained by generated code rather than by preference. And the fasttext wheel is pinned to a build named for numpy 2, which is a compiled artefact and therefore has to match the numpy major version in the environment.

Everything else uses a lower bound, and the list is long: FastAPI and uvicorn for the service, the Discord and Slack SDKs, four model clients for OpenAI, Anthropic, Google, and Cohere, Weights and Biases itself with the workspaces extra, and the Weave client.

Deployment targets appear in the repository as well as the dependencies: a Modal package directory and version, a container definition and build script for E2B, and a separate requirements file for evaluation runs.

The bot channel dependencies are also declared as an optional extra, so a deployment that serves only the HTTP endpoint does not install Discord and Slack.

Editorial conclusion

wandbot fits a documentation team that wants a support bot whose accuracy is measurable rather than asserted, since every version carries a correctness figure, a judge model, a trial count, and a link to the underlying evaluation. It does not fit someone who needs a stable answer rate, because the published trend is not monotonic and the newest configuration is the weakest of the recent ones. Before deploying, read the index version column rather than the version column, because the corpus is re-ingested independently of the code, and note that the table's newest row is newer than the newest release tag.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. wandb/wandbot on GitHub