kalshi-ai-trading-bot: an edge-measuring harness that refuses to claim edge

Ten forward-settled trades is the line between a result and a claim

The headline command reconciles Kalshi's authoritative settlements against your own decision journal and scores what actually resolved:

python cli.py edge

The reconciliation happens in memory and the command writes nothing. Four metrics come out. The Brier score asks whether a stated 70% is calibrated, with lower being better. Log-loss covers the same ground but punishes confident wrongness harder. Edge vs. book is the interesting one: the price you paid is the book's implied probability for your side, so `edge_vs_book` is realized win-rate minus mean implied price, where a positive number means you beat the book and roughly zero means the book was right. The forward-only guard counts only trades whose event resolved strictly after you traded, quarantining leakage and thin samples rather than counting them quietly.

Then the gate. With fewer than 10 forward-settled trades, or on data that is not forward, the command refuses to claim edge at all. The stated target of that refusal is a specific failure mode: asserting skill from a handful of lucky settlements.

The policy engine only ever tightens

Measurement feeds back into what you are allowed to trade next:

python cli.py policy
python cli.py improve

`cli improve` pulls settlements, reconciles them into the decision journal, and re-derives a machine-readable Edge Policy from the whole settled record. Three rules come out of it. Block fires when a category or method has at least 5 settled trades and negative realized P&L, and `cli trade` then refuses it; `--override-policy` exists, and the override is recorded. A side warning fires for a net-negative side but stays advisory, on the reasoning that losses concentrate in a few categories rather than one side of a market. A haircut applies where an `est_prob` band is at least 10 percentage points overconfident, shrinking the journaled estimate down to your calibrated win rate so future calibration corrects itself.

The same honesty threshold applies here: a group with fewer than 5 settled trades earns no opinion at all. Nothing here ever loosens a constraint on its own, and re-enabling a blocked group stays a manual decision. To see the machinery before supplying credentials, `cli policy --demo` runs against a shipped example.

The measurement code is pure, and every side effect stays in the CLI

The split is deliberate and worth noting for anyone auditing the scoring logic. The measurement code in `src/agent/edge.py` is pure and deterministic, while all IO lives in the command-line layer. That is what lets the Brier score and the edge arithmetic be reasoned about without a network or a database in the way, and it is why running `cli edge` leaves your journal untouched.

What the repository does not ship is a captured price and outcome corpus, and the project is direct about the consequence: this is not a backtest, and `cli backtest` explains why. A backtest needs historical prices and outcomes to replay, which means a data problem this toolkit declines to solve. The self-improvement loop sidesteps it, because it learns from the settlements you already have rather than from simulated history. If your question is how a strategy would have done on past markets, this repository cannot answer it.

A skeptic sits between the catalyst and every live buy

The agent-facing surface is a set of small tools in `cli.py`, each meant to do one thing and report honestly, callable by hand, wired into a loop, or handed to an MCP client. Two of them are read-only, and the second one carries most of the weight.

`brief` is a one-shot situational awareness call returning the governor verdict, equity, positions and resting orders as JSON. `verify` is described as the adversarial-verify gate that runs before any live buy, and it has three stages: research the catalyst, let a SKEPTIC role try to refute the fade, then have a deterministic gate recompute the edge. The shape matters. A generated thesis does not get to approve itself; a second role is tasked with breaking it, and the final check is not a model at all. The remaining rows of that table sit beyond what the README shows, so the full tool inventory has to be read from `cli.py` itself.

Five Python entry points share the repository root with src/

The root carries 22 entries, and five of them are Python programs rather than directories: `cli.py`, `setup_env.py`, `paper_trader.py`, `beast_mode_bot.py` and `beast_mode_dashboard.py`. Alongside them sit `src/`, `tests/`, `experimental/`, `data/`, `deploy/`, `docs/` and `scripts/`, plus `env.template` and a `CHANGELOG.md`. That layout means the documented entry point is `cli.py` while several others remain reachable at the top level, and the `experimental/` directory is not described anywhere in the README.

The documentation set is small and pointed. Two files are linked from the top: `docs/TRACK_RECORD.md` for the live track record and `docs/JEV.md` for a single market measured end to end. The `.claude/` directory at the root corresponds to the Claude skill named in the feature list, and the MCP server is described as a first-mover. Setup is a script rather than a manual environment: `python setup_env.py` creates `.venv` and installs dependencies, after which `env.template` is copied to `.env` and filled with `KALSHI_API_KEY` and `OPENROUTER_API_KEY`.

Three HTTP clients and two pinned crypto libraries in 24 dependencies

The runtime dependency list mixes restraint and pinching. Thirteen of the 24 entries are pinned to exact versions, including `aiohttp` at 3.9.1, `requests` at 2.31.0, `cryptography` at 42.0.0, `pycryptodome` at 3.20.0 and `openai` at 1.51.2. Seven carry ranges, such as `httpx>=0.27.1,<0.28`, `pandas>=2.0.0` and `pydantic>=2.11,<3`. Four are open-ended: `anthropic`, `pydantic-settings`, `uvicorn` and `json-repair`.

Three separate HTTP stacks ship at once, which is more than one client usually needs. Both crypto libraries are pinned exactly, and the signed Kalshi API client is the obvious consumer. Three provider SDKs appear, `openai`, `anthropic` and `xai_sdk`, while the LLM client itself is described as pluggable across any model on OpenRouter. `json-repair` in the list points at defensive parsing of model output rather than a strict decoder. Around them sit `structlog` for structured logs, `sentry-sdk` for error reporting and `ratelimit` for API throttling. Development extras add `pytest` 7.4.3, `pytest-asyncio` 0.21.1, `black` 23.12.1, `isort` 5.13.2 and an unpinned `mypy`.

Version 2.0.1, classified as beta, with no published release

The package manifest declares version 2.0.1, a licence of MIT, and a floor of Python 3.12, with classifiers naming 3.12 and 3.13 and a development status of Beta. The build backend is setuptools with wheel, and the manifest carries a typed marker alongside the mypy extra.

The repository itself has no GitHub releases. So the 2.x version number exists in `pyproject.toml` and in a root changelog, not in a tagged artefact, and anyone installing by tag will find nothing to pin. The last push was on 2026-09-26 and the repository is not archived, so the code is moving even though the release history is empty.

The most important line in the repository is the warning above the fold, and it is worth reading as a specification rather than boilerplate: nothing is guaranteed to make money, the example strategies lose money on certain markets, and this is a toolkit to fork rather than a turnkey bot.

Editorial conclusion

Use this if you intend to build your own Kalshi strategy and want a measurement harness that tells you when you have no edge yet. Do not use it as a turnkey bot: the shipped example strategies lose money on certain markets and the authors disclaim responsibility for losses. Before trading live, run `cli run --paper` to see the plumbing, read `docs/TRACK_RECORD.md` for what the harness reports on its own history, and expect `cli edge` to refuse to give a verdict until you have ten forward-settled trades.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. ryanfrigo/kalshi-ai-trading-bot on GitHub