# Kiln AI: a local workbench for evals, prompts, RAG and fine-tuning

> Kiln AI pairs a desktop app with an MIT-licensed Python library so one dataset can flow through evals, prompt optimization, RAG, agents and fine-tuning. The trade-off is a young project with an unusual licence file and a documentation set that leans on the app.

**Kiln-AI/Kiln** — Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

- Repository: https://github.com/Kiln-AI/Kiln
- Website: https://kiln.tech
- Stars: 5,071 · Forks: 380
- Language: Python
- License: NOASSERTION
- Published: 2026-09-10 · Updated: 2026-09-10 · Language: en
- Canonical page: https://hysenlabs.com/projects/kiln-ai-kiln

## The problem Kiln AI targets: one dataset, many techniques

Most teams working on an AI feature end up with a pile of disconnected tools. One library for orchestration, a spreadsheet or notebook for evaluation, a separate script for synthetic data, another service for fine-tuning. The README frames Kiln AI as an answer to that split: a workbench where a single task and dataset flow through evals, prompt optimization, fine-tuning, RAG, agents and synthetic data generation. The stated claim is that results compound across stages because nothing is re-exported between them.

The intended audience is wider than the usual Python-only crowd. The README says the desktop app lets product managers, subject experts and QA rate outputs and add data without writing code, while the MIT-licensed Python library ships the same tasks to production. That is the real pitch: the same task definition serves the person clicking through a rating UI and the person writing the deployment script. If your evaluation loop currently lives in a notebook that only one engineer can run, that gap is what Kiln AI is aimed at.

## How Kiln AI splits the work between app, server and library

The repository is a uv workspace, and pyproject.toml names three members: libs/core, libs/server and app/desktop. The root project, kiln-root, is not the library people install. It depends on kiln-ai, which the project description points to as the PyPI package for the library in libs/core, plus kiln-server and kiln-studio-desktop from the workspace. So the repository builds a desktop product and a server, and the pip-installable surface is the core library.

The Makefile shows how the pieces run during development. The dev target starts the desktop development API server with hot reload on port 8757. The ui target runs a Vite dev server, which the Makefile says listens on http://localhost:5173 by default. The schema target regenerates api_schema.d.ts from the running server's OpenAPI spec, and it requires that server to be up on port 8757. That tells you the web UI is generated against the server contract rather than hand-maintained, which is a reasonable choice for a project moving this fast, though it means the UI and the API schema are coupled at build time.

Deployment is the part worth thinking about. The README says agents built in the app can be deployed to production through the Python library, and that everything runs locally with your own API keys or fully offline through Ollama. The library being MIT-licensed is what makes that story workable for commercial products.

## Installing Kiln AI and running a first task

The README offers two entry points. The primary one is the desktop app: download Kiln Desktop for macOS, Windows or Linux from kiln.tech/download, then follow the 5-minute quickstart. There is no package manager step for that path, and the README does not document an unattended or headless install for the app.

The code-first path is the Python library. The README points to the Python library quickstart for this, and the root pyproject.toml pins kiln-ai==0.5.3, which is the version the repository itself resolves against. Installing from PyPI is the documented route:

```bash
pip install kiln-ai
```

If you want to work inside the repository rather than against the published package, the Makefile exposes a development server. This is what the project's own contributors run, and it listens on port 8757:

```bash
uv run python -m app.desktop.dev_server
```

The web UI is started separately, and the Makefile notes it uses the Node version recorded in app/web_ui/.nvmrc:

```bash
cd app/web_ui && npm run dev --
```

For moving work out of the app, the Makefile has a packaging target that wraps kiln_ai package_project. The example in the Makefile exports a project file with all tasks:

```bash
make package ARGS='~/KilnProjects/demo/project.kiln --all-tasks -o ./kiln_export.zip'
```

What you should see after the pip install is the kiln-ai library available for import; the README does not print a sample script, so the quickstart page is where the first task definition is written. Treat the app as the place to define and rate, and the library as the place to run.

## Where Kiln AI gets awkward

The licence is the first thing to check, and it is not a formality. The repository metadata reports NOASSERTION for the licence, while the README repeatedly describes the Python library as MIT-licensed and the repository ships a LICENSE.txt. Those two signals can be reconciled, but you should read LICENSE.txt yourself rather than take the README's word for it, especially if you plan to ship the library inside a commercial product. The project's own description of its licence is not the same as the licence file.

Version drift is the second issue. The root pyproject.toml pins kiln-ai==0.5.3, while the recent releases listed for the repository are desktop builds: v1.1.1, v1.0.4 and v1.0.3. The library version and the app version do not move in lockstep, and the README does not explain how they map to each other. If your team rates data in the app and deploys through the library, that mapping is something you have to establish yourself.

The third limitation is structural. The README's own framing of the problem is that code-only frameworks cover one slice, and Kiln AI's answer is to cover all of them. That breadth has a cost: the repository is a uv workspace with a desktop app, a server and a core library, and the developer workflow assumes uv, Node with nvm, and a running server on port 8757 before schema generation works. If your team wants a small dependency you can vendor and forget, this is the wrong shape. Kiln AI is also not a good fit if you cannot run a desktop application at all, since the collaboration and rating workflow the README describes lives there.

## Kiln AI against a code-only framework like LangChain

The honest comparison is with a code-only framework such as LangChain. The difference is not features, it is where the state lives. In a code-only framework, the task definition, the prompt, the evaluation set and the results are all artifacts in your repository, produced by scripts you write and review. Kiln AI puts a desktop app in front of that state and syncs it to Git, which the README describes as Git-native collaboration that works even for teammates who do not know Git. The app is the source of truth for tasks and datasets; the library is the execution path.

That inversion is the whole argument. It buys you non-engineers in the loop and a single dataset reused across evals, fine-tuning and RAG. It costs you the ability to define everything in a pull request, and it adds a GUI, a server and a Node toolchain to your development environment. The Makefile's schema and annotations targets both require the server to be running on port 8757, which is a concrete example of that added surface. If your team is three engineers who all live in the terminal, a code-only framework will feel lighter. If your team includes people who will never open a terminal but whose judgement you need in the evaluation data, the trade goes the other way.

## Maintenance, releases and what upgrading Kiln AI costs

The repository is not archived, and the most recent push recorded is 2026-09-10. The latest release listed is Kiln Desktop v1.1.1 from 2026-08-20, with v1.0.4 in July and v1.0.3 in June. Those are desktop releases, roughly a monthly cadence across the summer, which is fast enough that you should expect to re-read release notes before upgrading rather than assume compatibility.

Upgrade cost splits by component. The library is a normal PyPI dependency, so pinning kiln-ai in your own project is straightforward, and the root pyproject.toml demonstrates that pattern by pinning an exact version. The desktop app is a one-click install with no documented pinned channel, so the app and the library can move independently. The workspace also sets exclude-newer = "7 days" for uv, meaning the project deliberately holds back very fresh transitive dependencies, and it overrides starlette to a minimum version. Both are signs of a maintainer managing churn rather than ignoring it.

On licensing, the README's MIT claim covers the Python library, which is the part most teams embed. The repository-level licence resolves to NOASSERTION in the metadata. That is not legal advice, and the only reliable step is to read LICENSE.txt and confirm it matches the MIT text before you depend on the permission.

## Conclusion

Adopt Kiln AI if you want one place where a single task and dataset move through evals, prompt optimization, RAG, agents and fine-tuning, and you are willing to work from the app rather than a CLI. Skip it if you need a mature, narrowly scoped framework you can pin for years, or if your team cannot run a desktop application. Before committing, verify two things yourself: what LICENSE.txt actually grants, since the repository metadata reports NOASSERTION rather than MIT, and whether the Python library version you install matches the app version you export from.

## FAQ

### What is Kiln AI?

Kiln AI is a free desktop app plus an open-source Python library for building and evaluating AI products. The README describes it as a workbench covering evals, prompts, RAG, agents, fine-tuning and synthetic data generation, running locally with your own API keys or offline through Ollama.

### How do I install Kiln AI?

The README gives two paths: download Kiln Desktop for macOS, Windows or Linux from kiln.tech/download for the one-click app, or install the Python library from PyPI with pip install kiln-ai for the code-first route.

### Does Kiln AI run locally or in the cloud?

The README states that Kiln runs locally, with your own API keys, or fully offline using Ollama. Fine-tuning is the exception in the description, since it is offered across 60+ models on Fireworks, Together and Vertex.

## Sources

- [Issues](https://github.com/Kiln-AI/Kiln/issues)
- [Kiln-AI/Kiln on GitHub](https://github.com/Kiln-AI/Kiln)
- [Project website](https://kiln.tech)
- [README](https://github.com/Kiln-AI/Kiln/blob/main/README.md)
- [Releases](https://github.com/Kiln-AI/Kiln/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/kiln-ai-kiln