Potato (davidjurgens/potato): a self-hosted annotation server configured in YAML

What Potato is for, and who ends up running it

Potato is a free, self-hosted annotation platform for NLP, agent evaluation, generative AI work and qualitative research. The README describes it as configured entirely through YAML, with no coding required. That claim is about the task definition, not the deployment: someone still has to install the package, start the server and hand annotators a URL.

The audience is therefore a research group, an evaluation team or an internal data team that already has a place to run a Python service. The data types listed in the README are broad: text classification and span labeling, agent traces from frameworks such as OpenAI, Anthropic, ReAct, LangChain, LangFuse, WebArena, SWE-bench, OpenTelemetry, CrewAI, AutoGen, LangGraph, MCP, Aider, Claude Code, ATIF, SWE-Agent and Web Agent, plus audio with 21 transcript and subtitle formats, video, images, dialogue and documents including PDF, Word, Markdown, code and spreadsheets.

Potato also targets qualitative data analysis, which the README describes as a workflow with a living codebook, memos and cases. That is a different buyer from the one who wants a quick labeling pass, and the configuration surface reflects it.

How the YAML config, Flask server and trace converter fit together

The architecture visible in the repository is a Flask application. setup.py lists Flask, Werkzeug, Jinja2, pandas, scikit-learn and pydantic among the core dependencies that flask_server.py and routes.py import at module level. A task is a directory containing a config.yaml plus its data, and the server renders the annotation interface from that config.

The entry point is the potato command, which takes a config path and a port. The README shows both the installed-package form and the run-from-source form, and the source form calls python potato/flask_server.py directly.

Agent traces do not go straight into the interface. The README documents a converter that reads a trace file and writes JSONL:

python -m potato.trace_converter --input traces.json --input-format openai --output data.jsonl

The supported input formats are the list of frameworks above, and the README notes that auto-detection is available with --auto-detect. Evaluation can happen at four levels: trajectory, step, span and comparison. The web agent viewer is a separate mode that steps through screenshots with SVG overlays for clicks, bounding boxes, mouse paths and scrolls, with a filmstrip bar to jump between steps.

One design point worth naming: all AI SDKs are imported lazily through an endpoint registry in potato/ai/ai_endpoint.py, so a task that does not select an endpoint_type does not need openai, anthropic, ollama or google-genai installed at all. That keeps the base install smaller than the requirements.txt file suggests.

Installing Potato and running a first classification task

The README gives the PyPI route first. Installing the package gets you the server, but not the examples directory, which ships with the source repository. The README is explicit about this: after a PyPI install you either clone the repo for the examples or point potato start at your own config.

pip install potato-annotation
potato start examples/classification/single-choice/config.yaml -p 8000

If you have not cloned the repository, that second command will fail on the missing path. The alternative the README recommends, because it brings the examples along, is running from source:

git clone https://github.com/davidjurgens/potato.git
cd potato && pip install -r requirements.txt
python potato/flask_server.py start examples/classification/single-choice/config.yaml -p 8000

Note that requirements.txt is a development superset. Its own header says it installs core plus optional dependencies, and points at setup.py for the package metadata. For a leaner environment the header gives pip install . for core only and pip install ".[ai,formats,viz,export,auth]" for the extras you want. The Dockerfile takes the same approach with a POTATO_EXTRAS build argument, and its comments note that all deliberately excludes vision because that pulls multi-gigabyte torch wheels.

Once the server is up, the README says to open http://localhost:8000 and start annotating. The examples directory is the place to look for a template close to your task before writing a config from scratch. For agent work, the converter command above produces the JSONL that the config then points at.

Where Potato gets in the way

The examples problem is the first real friction. The README's own quick start acknowledges it: a PyPI install leaves you without examples/, and the recommended path is cloning the repository. For a tool whose pitch is YAML configuration rather than code, the fastest route to a working task still involves git.

Dependencies are the second. The core list in setup.py is not small: pandas, numpy, scipy and scikit-learn all come in unconditionally. The Dockerfile comments are candid about the consequence, stating that on a 1 GB droplet pip install is an out-of-memory kill and on 2 GB it takes five to ten minutes on every redeploy. That is why the image exists, and it is a fair signal that Potato is not a lightweight process you tuck onto a small shared box alongside other services.

Third, the repository layout describes a local server, not a distributed system. There is no mention of a database backend, object storage or multi-node coordination. The Dockerfile mounts the project directory at /app and states that one image serves every task, which is a clean model but also means your data lives wherever you mounted it. Teams expecting managed storage or horizontal scaling should treat that as out of scope rather than a gap that will be filled by configuration.

Finally, the interface is the product. If your annotation scheme cannot be expressed in the listed schemes (radio, checkbox, Likert, span, pairwise comparison, per-step ratings, free text, triage, conditional logic) or through custom layouts and raw HTML, you are writing schema types rather than configuring them. The README says new schema types can be added, which is an invitation to code.

Potato compared with Label Studio

Label Studio is the obvious alternative for a self-hosted, browser-based labeling tool, and the difference is in where the configuration lives and what the project optimizes for.

Label Studio's model centers on a labeling config expressed in XML, with a large template library and a backend that supports multiple storage connectors and a database. Potato's model centers on YAML task configs served by a single Flask application, with the project directory mounted into the container. If your workflow depends on pulling data from cloud buckets or keeping annotations in a database that other services query, Label Studio's connector-and-backend approach is the more natural fit, and Potato's mounted-directory model will feel like a step back.

Where Potato differs is the evaluation surface. The README devotes substantial space to agent traces: a converter for more than a dozen trace formats, four evaluation levels (trajectory, step, span, comparison), and a web agent viewer that steps through screenshots with SVG overlays for clicks and scrolls. It also documents audio handling in unusual depth, including reading 21 transcript and subtitle formats and running Whisper and speaker diarization locally when no transcript exists. If your task is annotating agent behavior or audio, that is the reason to pick Potato over a general-purpose tool. If your task is bounding boxes on a few thousand images, both will work and the choice comes down to which configuration format your team will maintain.

Licence, maintenance and what an upgrade actually costs

Potato is GPL-3.0. The README badge and the LICENSE file agree on this. GPL-3.0 is a copyleft licence, so if you distribute a modified Potato or a product built on it, the obligations that attach to derivative works apply. Self-hosting it internally for annotation does not raise the same question as shipping it inside a commercial product, but that distinction is a legal judgement, not a technical one, and this article is not legal advice. Note also that package.json declares MIT for the npm side while the repository licence is GPL-3.0; if you care about the JavaScript assets, that discrepancy is worth resolving with the maintainers rather than assuming.

On maintenance, the last push to the default branch was on 2026-09-15, and the most recent release listed is v2.9.1 on 2026-09-14, following v2.9.0 on 2026-09-13. The project is not archived. Release cadence in the recent record is high, with three releases inside roughly three weeks, and the release titles are candid about their contents (v2.8.2 is titled "The Fixes 2.8.1 Said It Had").

Upgrade cost depends on which extras you installed. Core-only installs move with pip install . or a rebuilt image. If you enabled ai, formats, viz, export or auth, an upgrade can pull new versions of openai, anthropic, google-genai, pdfplumber, python-docx, mammoth, mistune, pygments, openpyxl, umap-learn, pyarrow or Authlib, any of which can change behavior independently of Potato. The repository also ships MIGRATION.md, which is the file to read before moving between major versions.

Editorial conclusion

Adopt Potato if your annotation task needs a browser UI you can stand up on your own hardware and define in YAML rather than code, particularly for agent traces, audio with transcripts, or documents. Do not adopt it if you want a hosted service with built-in worker recruitment, or if you need cloud storage backends, since the README and repository layout describe a local Flask server with a mounted project directory. Verify first that the extras you need are installed, because setup.py splits core dependencies from ai, formats, viz, export and auth groups, and that the Docker image's internal port 7860 is the one you map.

Frequently asked questions

Official sources

1. davidjurgens/potato on GitHub
2. License: GPL-3.0
3. Project website
4. README
5. Releases