ICE traces how a language model program decomposes a task, one recipe step at a time

A recipe decomposes, an agent executes, and the trace records it

ICE is a Python library and trace visualizer for language model programs, and the terminology section is the clearest explanation of what it does.

Recipes are decompositions of a task into subtasks. The meaning of a recipe is given precisely: if a human executed these steps and did a good job at each workspace in isolation, the overall answer would be good. The decomposition may be informed by what we think machine learning can do at this point, but the recipe itself, as an abstraction, does not know about specific agents.

Agents perform atomic subtasks of predefined shapes, like completion, scoring or classification. They do not know which recipe is calling them, they do not maintain state between subtasks, and they generally try to complete all subtasks they are asked to complete, however badly, though some will not have implementations for certain task types.

The third concept is the mode in which a recipe runs, which is a global setting that can affect every agent call, for instance whether to use humans or agents. Recipes can also run with certain RecipeSettings, which can map a task type to a specific agent_name, and that in turn modifies which agent is used for that specific type of task.

Put together, that is a debugging story: a program made of prompts is decomposed into steps, each step is executed by a swappable agent, and the whole thing is recorded so you can inspect where the reasoning went wrong.

Three modes and a browser view of the execution

The feature list is short, and the first item is the one that changes how you develop against a model.

You can run language model recipes in different modes: humans, human plus LM, and LM. That matters because it lets you substitute a person for the model at a particular step, which is a practical way to check whether a decomposition is sensible before spending tokens on it.

The second feature is inspecting the execution traces in your browser for debugging, and the project links a wiki page for it, the ICE UI guide. The README shows an execution trace visualized in ICE, and the project site is at ice.ought.org.

The rest of the list covers extensibility and throughput. You can define and use new language model agents, for example chain-of-thought agents. Recipes run quickly by parallelizing language model calls, which is how a decomposition with many independent subtasks avoids becoming a serial chain. And component recipes can be reused, such as question-answering, ranking and verification, so a question-answering step is written once and dropped into several recipes.

The combination of the three is the interesting part. Parallel execution is what makes a decomposition affordable, and the trace view is what makes it debuggable, and a browser UI is what makes the trace legible to someone who is not reading Python.

Installing means a virtualenv, one pip command, and a secrets file

The getting started section is four steps, and the first one is ordinary Python hygiene rather than anything specific to ICE.

The instruction is to create and activate a virtual environment to avoid installing ICE globally, and then to install the package from PyPI:

   python -m venv venv
   source venv/bin/activate
   pip install ought-ice

The third step is to run the Hello World recipe in the Primer to see the trace rendered, which tells you the thing you want to see is a rendered trace, and the Primer is where you see it.

The fourth step is optional and is about secrets. You can set secrets like your OpenAI API key in ~/.ought-ice/.env, and the .env.example file in the repository shows the format. If they are not set, you are prompted for them when you run recipes that need them, which is a sensible default for a tool where a recipe may or may not call a model.

The .env.example file lists three variables:

OPENAI_API_KEY=sk-...
OPENAI_ORG_ID=""
OUGHT_INFERENCE_API_KEY=""

OPENAI_API_KEY is required to use any OpenAI models, OPENAI_ORG_ID is optional because OpenAI lets you specify an organization id, and OUGHT_INFERENCE_API_KEY is not required and is only used for calling Ought internal models.

The requirements section is where the version constraints live. ICE requires Python 3.9, 3.10 or 3.11, and the project recommends pyenv if you do not have a supported version installed. If you use Windows you will need to run ICE inside WSL. That is a narrow range of Python versions, and it is the first thing to check.

Developing ICE means a Node frontend and a Python backend side by side

The repository layout shows a split project. There is an ice/ directory for the Python package, a ui/ directory for the frontend, main.py at the root, a tests/ directory, and a gold_standards/ directory alongside papers/ and scripts/.

The development instructions cover the frontend setup, and the commands are in one block:

   python -m venv venv
   source venv/bin/activate
   pip install --upgrade pip
   pip install -e '.[dev]' --config-settings editable_mode=compat
   pre-commit install
   npm --prefix ui ci
   npm --prefix ui run dev

Two things in there are worth pausing on. The editable install uses the compat editable mode setting, which is what makes the setuptools editable install work with a src layout and the setuptools_scm version plugin. And the frontend is a separate npm project, so you are running two toolchains to work on ICE.

The backend has its own cleanup step, because a cached model call is a cached model call whether you wanted it or not:

   rm -r ~/.ought-ice/cache

The pre-commit section is worth reading for anyone who hits a confusing failure. The hooks can reject your commit and make you fix the problems before you can commit again, and the readme advises reusing the same commit message. It also notes that you do not technically need to run pre-commit install, but that not doing so may cause your commits to fail CI, which can be noisy since it generates commits that fix formatting.

For the frontend, Storybook is used for UI tests and can be run locally with npm --prefix ui run storybook, with a note that build-storybook is only for CI and should not be run locally.

The API may change, and the project says so in a warning

The section titled ICE is pre-1.0 carries a warning that is unusually direct for a README: the ICE API may change at any point, the interface is being actively developed and the API may change at any point, including removing functionality, renaming methods, splitting ICE into multiple projects, and other similarly disruptive changes. Use at your own risk.

That is worth taking at face value rather than reading as boilerplate, because the specific list of changes named includes splitting the project. If ICE is split into multiple packages, your dependency list changes, your import paths change, and any code you wrote against the API may not survive.

The version story explains part of it. The repository has no GitHub releases, and the version is derived from git tags through setuptools_scm, with the write_to setting pointing at ice/version.py. So the version number in your environment reflects a tag, and there is no release history in the usual sense to pin against.

The practical consequence is that ICE belongs in the same category as a research tool rather than a dependency. If you are experimenting with how to decompose a task for a model, an API that moves is an acceptable cost. If you are building a service on top of it, you are taking on churn you cannot schedule around.

There is a citation to point to, which is a sign the work is intended to be used in research. The suggested citation is the Ought technical report Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes, arXiv 2301.01751, with a Bibtex entry. The repository also links a recording of an Ought lab meeting to explain the high-level goals and the connection to alignment research.

Where ICE stops and a hand-rolled pipeline starts

The first limitation is the one already named, the pre-1.0 API, and it is the whole story for anyone considering a production dependency.

The second is the scope of what a recipe can express. The recipe model is a decomposition into subtasks with agents of predefined shapes, and agents generally try to complete all subtasks asked of them. That is a reasonable design for a research tool that wants to make decomposition explicit. It is a constraint for a general workflow engine, where you might want conditional branching, loops, or a subtask whose shape is decided at runtime by the model itself. The readme does not claim those, so treat them as out of scope rather than as missing features to file issues about.

The third is the dependency on a hosted model for anything interesting. The .env file expects an OpenAI key, and while the Ought inference key is optional, the built-in agents are built around that ecosystem. Running everything locally is possible in principle, but the readme does not describe a local model backend, and the agents for the predefined task shapes are what the parallelism is designed around.

The alternative to weigh is writing the decomposition yourself in plain Python with plain function calls, which gives you a control flow you understand, and instrumenting it with a tracing library or simple logging. That is more code and less insight, because you lose the browser trace view and the agent abstraction, and it is the trace view that is the reason to use ICE rather than a notebook.

There is also the question of what the gold_standards/ directory is for, which the repository listing shows but the readme does not describe. If you are evaluating the project, that directory is worth looking at directly, because it tells you what the expected outputs look like and whether the tool can be used to check itself against known good answers.

The project is open source under the MIT licence by Ought, an applied machine learning lab building the research assistant Elicit, and it welcomes contributions from developers fixing bugs, improving code quality and performance or adding features, and from language model researchers adding agents or refining recipes. For larger contributions, the readme asks for an issue discussion before a pull request.

Editorial conclusion

Adopt ICE if you are building programs that chain language model calls and need to see where a bad answer came from, because the trace view and the recipe decomposition are the whole point of the project. Do not adopt it in a production pipeline without pinning, since the readme states the API may change at any point including removing functionality and splitting ICE into multiple projects. Verify first that Python 3.9, 3.10 or 3.11 is available, that the agents you need exist for the task types you use, and that you are comfortable with the OpenAI dependency implied by the environment file.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. oughtinc/ice on GitHub
4. Project website
5. README