# cascadeflow: In-Process Model Cascading for AI Agents

> cascadeflow is an MIT-licensed Python and TypeScript library that makes per-step model and budget decisions inside an agent's execution loop. It is a good fit when you control the agent code and want enforcement rather than observation.

**lemony-ai/cascadeflow** — Cascading runtime for AI agents. Optimize cost, latency, quality, and policy decisions inside the agent loop.

- Repository: https://github.com/lemony-ai/cascadeflow
- Website: https://cascadeflow.ai
- Stars: 3,993 · Forks: 892
- Language: Python
- License: MIT
- Published: 2026-09-09 · Updated: 2026-09-09 · Language: en
- Canonical page: https://hysenlabs.com/projects/lemony-ai-cascadeflow

## The problem cascadeflow targets: model choice inside the agent loop

Most cost control for LLM applications happens at the HTTP boundary. A proxy sits in front of the provider API, sees a request, and decides what to do with it. That works when a request is the unit of work. Agents break that assumption. An agent run is a sequence: plan, call a tool, read the result, decide again, maybe escalate. The expensive step is often not the first one. A proxy cannot see that the third tool call returned something ambiguous, because by the time the ambiguity exists the request has already been sent. cascadeflow's answer is to move the decision into the process where the agent runs. The README frames this as the difference between an external proxy and an in-process harness: scope inside the execution loop, dimensions covering cost, quality, latency, budget, compliance and energy, and enforcement actions the README names as stop, deny_tool and switch_model. The stated audience is developers building agent workflows who already own the agent code. If your agents run inside a hosted platform you cannot edit, this library has nowhere to attach.

## How the cascading mechanism and decision traces work

The core idea is speculative execution with escalation. The README states that 40 to 70 percent of queries do not require slow, expensive flagship models, and that domain-specific smaller models often outperform large general-purpose ones on specialized tasks. So cascadeflow tries a cheaper model first and escalates to a flagship model only when needed. What separates this from a plain fallback chain is where the signal comes from. The README describes the library as accumulating insight from every model call, tool result and quality score, which means the escalation decision can be informed by what happened earlier in the same run rather than by the prompt alone. Decisions are recorded as per-step decision traces, which the README contrasts with the request logs a proxy produces. That distinction matters for debugging: a request log tells you what was sent, a decision trace tells you why the runtime chose that model at that step. The README also claims sub-5ms overhead for the in-process path, against 10 to 50ms of network round-trip for an external proxy. Treat both numbers as the project's claims rather than measured results.

## Installing cascadeflow and running a first cascade

The README gives two install commands, one per language. The Python package is on PyPI as cascadeflow; the TypeScript core is on npm as @cascadeflow/core.

```bash
pip install cascadeflow
```

```bash
npm install @cascadeflow/core
```

The base Python install pulls only pydantic, httpx and tiktoken, according to requirements.txt. Provider SDKs are optional extras, and the file lists them explicitly: cascadeflow[openai], cascadeflow[anthropic], cascadeflow[groq], cascadeflow[providers] for OpenAI plus Anthropic plus Groq, and cascadeflow[all] for everything. For a first run with OpenAI, install the extra rather than the bare package.

```bash
pip install cascadeflow[openai]
```

Credentials come from environment variables. The repository ships a .env.example you copy to .env and fill in. The keys it documents are OPENAI_API_KEY, ANTHROPIC_API_KEY, GROQ_API_KEY and TOGETHER_API_KEY. It also documents OPENAI_TOOL_MODEL, which it says is used by live end-to-end tests that verify tool calling and streaming, with gpt-4o-mini given as an example of a tool-capable chat model. The file notes that Ollama and vLLM need no API key: Ollama is reached over HTTP at localhost:11434, and vLLM can be used over HTTP or through the cascadeflow[vllm] extra.

For a worked first example, the repository has examples/basic_usage.py and examples/custom_cascade.py, plus a custom_validation.py example for scoring outputs. The README does not inline a complete runnable snippet, so the honest first step is to open those example files and adapt one rather than to guess at an API from the README alone.

## Where cascadeflow is the wrong tool

The clearest limitation is architectural. cascadeflow is a library, so it only helps where you can change the code that calls the model. If your agents are assembled in a vendor's hosted builder, or if the calls originate from a service you do not own, the in-process harness has no insertion point. The README's own comparison table makes the trade explicit: a proxy sees the HTTP boundary and enforces nothing, cascadeflow sees the loop and enforces stop, deny_tool and switch_model. Neither column is strictly better. A proxy is language-agnostic and catches every caller behind it, including ones you have not migrated. A library is per-caller. The second limitation is that cascading is a bet on a distribution. The README's premise is that most queries do not need a flagship model. If your workload is the minority case, where nearly every call genuinely needs the strongest model, the cheap-first path adds a speculative call and its latency without buying much. The third is verification. The README reports savings of 69% on MT-Bench, 93% on GSM8K, 52% on MMLU and 80% on TruthfulQA while retaining 96% GPT-5 quality. Those are the project's figures on those benchmarks. They are not a prediction for your traffic, and the README does not present a method for estimating the gap on a private workload.

## cascadeflow compared with a gateway proxy

The real alternative for most teams is an LLM gateway or proxy that sits in front of the provider API. The difference is not quality of implementation, it is where the decision lives. A gateway routes on what it can see in the request: model name, token count, headers, maybe a routing rule you configured. It cannot see the tool result that just came back, because that result never passes through it. cascadeflow routes on agent state, which is exactly the information a gateway lacks. The cost of that extra context is integration work. A gateway is usually a URL change plus a key. cascadeflow is a dependency in your agent process, with provider extras to install and framework integrations to wire up. The README lists those integrations: LangChain, OpenAI Agents SDK, CrewAI, PydanticAI, Google ADK, n8n, Vercel AI SDK and Hermes Agent, with per-integration docs at docs.cascadeflow.ai. If your stack is on that list, the integration cost is bounded. If it is not, you are writing the adapter yourself. A reasonable split is to run a gateway for coarse routing and observability across all callers, and cascadeflow inside the agents where the per-step decision actually changes the outcome.

## Maintenance, licence and the cost of upgrading

The repository is not archived, and the last push was on 2026-09-08, so the project is being worked on. Releases are less frequent than pushes: v1.0.0 landed on 2026-02-22, v1.1.0 on 2026-03-08, and v1.2.0 on 2026-04-02, with no release listed since. That gap between push activity and releases is worth knowing before you plan an upgrade cadence, because unreleased changes on main are not the same as a versioned artifact. The licence is MIT, declared in both pyproject.toml and package.json, which permits commercial use and modification, but this is a description of the licence text and not legal advice; check the LICENSE file and your own counsel for anything that matters. The upgrade surface is small in one respect and awkward in another. The core dependency list is short (pydantic, httpx, tiktoken), so the base package is unlikely to drag in conflicts. Provider SDKs are extras, which means a provider's breaking change reaches you only when you install or bump that extra. The awkward part is that cascadeflow sits inside your agent loop, so a behaviour change in model selection or budget gating can alter outputs, not just throw an error. Pin the version and diff the CHANGELOG.md before moving.

## Conclusion

Adopt cascadeflow if your agent code is yours to edit and you want model selection, budget gating and stop/deny_tool/switch_model enforcement at the step level rather than at an HTTP boundary. Do not adopt it if your agents run behind a platform you cannot modify, or if you only need request-level cost logging, where an external proxy is simpler. Before committing, verify two things against your own stack: that the provider extras you need are listed in pyproject.toml, and that your framework integration (LangChain, CrewAI, PydanticAI, Google ADK, n8n, Vercel AI, Hermes Agent) appears at docs.cascadeflow.ai. The README's headline savings figures (69% on MT-Bench, 93% on GSM8K, 52% on MMLU, 80% on TruthfulQA, retaining 96% GPT-5 quality) are the project's own claims, so reproduce them on your traffic before you put them in a budget forecast.

## FAQ

### What is cascadeflow and what does it do?

cascadeflow is an MIT-licensed library and agent harness, published for Python on PyPI and for TypeScript on npm as @cascadeflow/core. The README describes it as an in-process intelligence layer that optimizes cost, latency, quality, budget, compliance and energy inside the agent execution loop, using speculative execution to pick a model per query or tool call.

### What is the purpose of cascading in cascadeflow?

The README states that 40 to 70 percent of queries do not require slow, expensive flagship models, and that smaller domain-specific models often outperform large general-purpose ones on specialized tasks. Cascading tries the cheaper model first and escalates to a flagship model when the query needs advanced reasoning.

### What is a good example of a cascade in cascadeflow?

The repository ships examples/basic_usage.py for a straightforward cascade and examples/custom_cascade.py for a customized one, alongside examples/custom_validation.py for scoring outputs. The README does not inline a full runnable cascade snippet, so those files are the concrete starting points.

### What is an example of a cascade in cascadeflow?

The README describes the pattern as trying a cheaper model first and escalating to a flagship model when the query needs advanced reasoning. The repository's examples directory holds the runnable versions of that pattern rather than the README itself.

### What is cascade flow in the context of cascadeflow?

In this project the term refers to the cascading runtime the README describes: per-step model decisions made inside the agent loop, with enforcement actions named as stop, deny_tool and switch_model, plus per-step decision traces. It is distinct from the process-control meaning of cascade control used in industrial engineering.

## Sources

- [lemony-ai/cascadeflow on GitHub](https://github.com/lemony-ai/cascadeflow)
- [License: MIT](https://github.com/lemony-ai/cascadeflow/blob/main/LICENSE)
- [Project website](https://cascadeflow.ai)
- [README](https://github.com/lemony-ai/cascadeflow/blob/main/README.md)
- [Releases](https://github.com/lemony-ai/cascadeflow/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/lemony-ai-cascadeflow