kani: a hackable Python microframework for tool-calling language models
kani (カニ) is a highly hackable microframework for tool-calling language models. (NLP-OSS @ EMNLP 2023)
At a glance
- What is it?
- kani wraps chat models in a small async loop where Python methods become tools via a decorator. It is aimed at researchers and developers who want to control the prompt and the call loop, not inherit someone else's agent architecture.
- Who is it for?
- Adopt kani if you want a thin async chat loop, a decorator that turns typed Python methods into tools, and per-engine control over which provider you call. Do not adopt it if you want a batteries-included agent runtime with built-in memory stores, planners or retries, because the README does not describe any of those.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 19 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What kani actually solves for tool-calling models
Most chat wrappers answer one question: how do I get text in and text out. kani answers a narrower one. How do I let a language model decide to call one of my Python functions, and how do I keep control of everything between the model's decision and my code running?
The README frames this as being "less opinionated" than other LM frameworks, with "more fine-grained customizability over the parts of the control flow that matter." That is the whole pitch. kani does not ship an agent abstraction on top of your functions. It gives you a Kani object that owns chat state, prompting and function calling, and lets you subclass it.
The intended audience is stated plainly in the README: NLP researchers, hobbyists and developers. The PyPI classifiers list "Development Status :: 4 - Beta" and "Intended Audience :: Science/Research". The beta classifier is worth noting if you are wiring this into something with an uptime requirement. The project has a paper (NLP-OSS @ EMNLP 2023), which tells you the design was written up for a research audience rather than derived from a product roadmap.
The Kani object, the Engine, and where your code sits
The architecture has two moving parts. An Engine talks to a provider. A Kani holds the conversation and decides what to do with the model's output.
The README is explicit that "kani uses an Engine to interact with the language model," and that "the kani manages the chat state, prompting, and function calling." Engines are model-agnostic by design. The README lists OpenAI, Anthropic, Google AI, Hugging Face transformers, llama.cpp and vLLM as first-party extras, and points to community extensions for the rest. A model zoo example file exists at examples/4_engines_zoo.py, which is where the README sends you for loading popular models.
Function calling is the part with a real mechanism behind it. You subclass Kani and decorate a method with @ai_function(). The docstring becomes the tool description, and Annotated types with AIParam add per-parameter documentation. The README states that "kani guarantees that function calls are valid by the time they reach your methods." That is a claim about validation happening before dispatch, and it is the reason the decorator is worth using over hand-rolling a JSON schema.
Streaming is supported token by token, and the README also mentions multimodal inputs via a separate extra. The repository has a sandbox/ directory and an examples/5_advanced_subkanis.py file, which suggests sub-agent composition is at least demonstrated, though the README excerpt here does not describe the semantics.
Installing kani and exposing your first function
kani requires Python 3.10 or above. It uses extras to pull in provider-specific dependencies, so the install line depends on which model you intend to call. The README gives these forms.
pip install "kani[openai]"For a local Hugging Face model you also install torch, and the README shows the two together.
pip install "kani[huggingface]" torchIf you would rather not decide, there is an all extra. The README notes you can also install the development version from the main branch of the Git repository if you need changes that have not been released.
The quickstart builds an engine, hands it to a Kani, and either chats in the terminal or awaits a single round. The README's own example looks like this, with the API key placeholder replaced by your real key.
import asyncio
from kani import Kani, chat_in_terminal
from kani.engines.openai import OpenAIEngine
api_key = "sk-..."
engine = OpenAIEngine(api_key, model="gpt-5-nano")
ai = Kani(engine)
async def main():
resp = await ai.chat_round("What is the airspeed velocity of an unladen swallow?")
print(resp.text)
asyncio.run(main())Running that should print a single assistant reply. To add a tool, subclass Kani and decorate a method. The README's weather example is the canonical one.
from typing import Annotated
from kani import AIParam, Kani, ai_function
class MyKani(Kani):
@ai_function()
def get_weather(
self,
location: Annotated[str, AIParam(desc="The city and state, e.g. San Francisco, CA")],
):
"""Get the current weather in a given location."""
return f"Weather in {location}: Sunny, 72 degrees fahrenheit."The docstring is what the model reads, so write it as an instruction, not as a note to yourself. For multi-step conversations the README uses full_round as an async iterator, which yields each message in the round as it is produced.
async def main():
async for msg in ai.full_round("What's the weather in Tokyo?"):
print(msg.role, msg.text)
asyncio.run(main())You should see the assistant request the function, then a tool message carrying the return value, then a final assistant answer. If you only want to poke at the behaviour interactively, chat_in_terminal(ai) gives you a REPL against the same object.
Where kani gets in the way, and where it does not help at all
The beta classifier in pyproject.toml is not decoration. It means the API can move between minor versions, and the release history shows that pattern: v1.9.0, then v1.9.1 six days later, then nothing until v1.10.0 in August 2026. Pin your version if you are building on top of it.
The dependency ranges are wide on purpose. pydantic is pinned to >=2.0.0,<3.0.0, openai to >=1.26.0,<3.0.0, transformers to >=4.28.0,<6.0.0. Wide ranges mean fewer resolution conflicts, and also mean a provider SDK bump can change behaviour under you without kani cutting a release. The pyproject comments show the maintainers tracking upstream breakage by hand, for example noting that transformers 4.56.1 changed ProcessorMixin.__call__.
kani is the wrong tool if you want an agent framework with built-in memory, retrieval or planning. The README does not describe a memory store, a planner or a retry policy. There is an examples/5_advanced_retrieval.py file, so retrieval is demonstrated, but it is an example rather than a framework feature. If your requirement is "give me an agent that remembers things across sessions," you will be writing that layer yourself.
It is also the wrong tool if you are not comfortable with async. The entry points are coroutines, the round methods are awaited or iterated, and the terminal helper is the only synchronous-looking affordance in the quickstart. That is a real constraint for scripts that were written against a blocking client.
kani compared with LangChain and with the raw provider SDK
The honest comparison is not kani versus another framework. It is kani versus the provider SDK you would otherwise call directly.
The OpenAI Python SDK already supports tool calling. You pass a tools list of JSON schemas, you get back tool_calls, you dispatch them yourself, you append the results and call again. That loop is maybe forty lines. kani's contribution is that the schema is generated from a Python signature and docstring, the loop is already written, and the same Kani subclass works against a different Engine without you rewriting the dispatch. The README's claim that kani is "model-agnostic" is the concrete difference here: swapping OpenAIEngine for a vLLM or llama.cpp engine should not change your subclass.
Against LangChain, the difference is scope, not quality. LangChain offers chains, retrievers, memory and a large integration surface. kani offers a chat object, an engine, and a decorator. If you want the integration surface, kani will feel like it is missing pieces. If you have been fighting a framework's abstractions to get at the prompt, kani's positioning as "less opinionated" is the argument for it. The cost of that position is that every piece you need beyond the chat loop is yours to build and yours to maintain.
Maintenance, licensing and the upgrade bill
The last push to the default branch was on 2026-08-27, and v1.10.0 was released the same day. The repository is not archived. The gap between v1.9.1 in March 2026 and v1.10.0 in August 2026 is roughly five months, so treat this as a project that releases when there is something to release rather than on a cadence.
kani is MIT licensed, and pyproject.toml declares the license by file reference to LICENSE. MIT is permissive: you can use it commercially, modify it and redistribute it, provided the copyright notice and permission notice travel with it. That is the standard reading and not legal advice; if you are redistributing kani inside a product, have someone check the notice requirements.
The upgrade cost is mostly the provider extras, not kani itself. Because the extras pull openai, anthropic, google-genai, transformers, llama-cpp-python and vLLM with upper bounds, a fresh install months later can resolve to a newer provider SDK than the one kani was tested against. The repository pins pytest below 9.1.0 in requirements.txt because of pytest-lazy-fixtures, which tells you the test suite is sensitive to its own tooling. For applications, that suggests locking your resolved dependency set rather than tracking kani's ranges.
What to check before you commit to it
Read the model table in the engines documentation before installing anything. The extras are not interchangeable, and installing kani[all] pulls every provider SDK plus transformers, which is a large dependency tree for a project whose only required runtime dependency is pydantic.
Then look at the examples directory. It is unusually well populated for a microframework: entry points, fewshot prompting, function calling with MCP tools, exception and prompt customization, logging, tracking function calls, FP4 quantization, multimodal message parts, retrieval, and sub-Kanis. Those files are the real documentation of what the framework can do, and the README leans on them rather than reproducing their content.
Finally, check the MCP path if remote tools matter to you. The README points to the function calling docs for local and remote MCP servers, and there is an examples/2_function_calling_mcp_tools.py file. The mcp extra requires mcp>=1.15.0,<2.0.0. The README excerpt does not describe the MCP API surface, so that page is the place to confirm it fits your setup.
Editorial conclusion
Adopt kani if you want a thin async chat loop, a decorator that turns typed Python methods into tools, and per-engine control over which provider you call. Do not adopt it if you want a batteries-included agent runtime with built-in memory stores, planners or retries, because the README does not describe any of those. Before writing code, read the model table in the engines docs to pick the right extra, and check the docs page on function calling for the current MCP tool path.
Frequently asked questions
What Python version does kani require?
kani requires Python 3.10 or above, as stated in the README and in the requires-python field of pyproject.toml.
How do I install kani for OpenAI models?
Run pip install "kani[openai]". The README uses extras to select provider dependencies, and there is also an all extra if you want every provider installed.
How does kani expose a Python function to the model?
You subclass Kani and decorate a method with the @ai_function decorator. The docstring becomes the tool description, and Annotated parameters with AIParam add per-parameter documentation.
Does kani support models other than OpenAI?
Yes. The README lists Anthropic, Google AI, Hugging Face transformers, llama.cpp and vLLM as first-party extras, and points to community extensions for additional models.
Can kani stream responses token by token?
The README states that kani supports streaming responses from the underlying language model token by token. The full_round method is shown as an async iterator that yields each message in the round.
Does kani support MCP tools?
The README notes that kani supports local and remote MCP servers and links to the function calling documentation for MCP tools. The mcp extra requires mcp>=1.15.0,<2.0.0.
Community notes