Graph of Thoughts: Orchestrating LLM Operations on a Directed Graph

What Graph of Thoughts Actually Solves

Graph of Thoughts (GoT) addresses a specific weakness in prompt engineering: when a single LLM call is not enough, how do you structure multiple calls that depend on each other? Chain-of-thought (CoT) forces a linear sequence, and tree-of-thought (ToT) allows branching but not merging. GoT generalizes these by letting you model the entire reasoning process as a directed graph, where each node is an operation like Generate or Score, and edges define the flow of thought states. The framework then executes that graph with an LLM as the engine. This matters for problems where intermediate results need to be combined, compared, or refined, which a linear or tree structure cannot express cleanly. The README positions it for "elaborate problems" and the included examples, sorting and keyword counting, are tasks where partial solutions can be merged. The target user is someone who has hit the ceiling of a single prompt and wants a programmatic way to explore alternative reasoning topologies without hand-coding every LLM call.

The Mechanism: Graph of Operations and the Controller

The core abstraction is the GraphOfOperations class. You build it by appending operation instances, each of which is a step the framework will execute. The README shows a minimal CoT-style graph: Generate, then Score using a scoring function, then GroundTruth. For the GoT approach, a prebuilt graph function (got) is imported from the example module. The Controller ties everything together. It takes a language model, the graph, a prompter, a parser, and an initial thought state dictionary. The prompter turns a thought state into a prompt, the parser extracts the output back into a thought state, and the Controller walks the graph, invoking the LLM where needed. The output is a JSON graph file that records the thought states and their scores. This design is deliberately extensible: you can implement graphs that resemble CoT or ToT, as the README states. The mechanism is not a new inference algorithm; it is an orchestration layer that gives you explicit control over the data flow between LLM calls. That is its real value, and also its complexity, because you must define the scoring and parsing logic yourself.

Getting It Running: Commands and Configuration

Installation is straightforward. For a user, pip install graph_of_thoughts works. For a developer who wants to modify code, the README instructs cloning the repository and running pip install -e . from the source directory. Python 3.8 or newer is required. After that, you configure the LLM by following the Controller README, which is referenced but not reproduced. The quick start assumes a config.json file in the current directory with an OpenAI API key, and it instantiates language_models.ChatGPT("config.json", model_name="chatgpt"). This means the framework is not model-agnostic out of the box; you must adapt the configuration to your provider. The examples can be run directly as modules, for instance python -m examples.sorting.sorting_032. The two code snippets in the README differ only in the graph they use and the initial state dictionary, which for GoT includes a phase field. This shows that switching between reasoning strategies is a matter of swapping the graph and adjusting the initial state, not rewriting the controller logic.

A Concrete Limitation: Scoring and Ground Truth Depend on You

The framework does not magically evaluate LLM outputs. In the sorting example, the Score operation takes a scoring_function that counts errors, and the GroundTruth operation compares against a test function. You must supply these for every problem you want to solve. That is a real burden. For sorting, writing an error counter is easy. For open-ended tasks like summarization or creative writing, defining a numeric score is arbitrary and often requires another LLM call, which the framework does not provide. The README's examples are all problems with objective correctness criteria. If your problem lacks a clear ground truth, you will spend more time engineering the scoring function than the graph. This is a genuine failure mode: GoT is a scaffold for reasoning, not a solution to evaluation. The paper may discuss more, but the repository material does not show built-in support for qualitative scoring.

When Graph of Thoughts Is the Wrong Tool

If your task fits in a single prompt, GoT is overkill. The overhead of defining operations, a prompter, a parser, and a scoring function is not justified for a question that a direct LLM call answers. Similarly, if you need low-latency responses, executing a graph with multiple sequential LLM calls will be slower than a single call. The framework is also not a drop-in replacement for a prompt management system; it requires you to write Python code and understand the internal state flow. The README's examples are research-oriented, and the framework appears designed for experimentation rather than production serving. If your application needs real-time streaming or cost control per call, you would have to build that yourself. The absence of a declared license (NOASSERTION) is another red flag for commercial adoption, because you cannot legally redistribute or modify the code without clarity. That alone may disqualify it for many teams.

Alternatives: LangChain and Direct Prompting

A common alternative is LangChain, which also chains LLM calls but uses a different abstraction: chains and agents rather than a graph of operations. LangChain provides many prebuilt integrations for models, vector stores, and tools, and it has a permissive MIT license. In contrast, GoT gives you fine-grained control over the graph topology and operation semantics, but you must implement the prompter and parser for each new problem. Direct prompting is another alternative: you can manually write a script that makes sequential LLM calls and passes outputs as inputs to the next prompt. That approach is simpler but does not scale when you have branching or merging logic. The real difference is that GoT makes the reasoning structure explicit and reusable, while LangChain optimizes for breadth of integrations and direct prompting for simplicity. If your goal is to compare different reasoning strategies in a research setting, GoT's graph model is more expressive than LangChain's linear chains.

Maintenance and Upgrade Cost

The repository shows signs of low activity. The last release was v0.0.2 in September 2023, and the last push was March 2026, which is a long gap. The README points to a paper from 2024, but the codebase has not seen a new release in over two years. That means you should expect to maintain it yourself if you depend on it. The framework's reliance on a specific ChatGPT model configuration suggests that newer model APIs may not work without code changes. The documentation is described as thorough, but the only concrete README files referenced are for the Controller and Operations modules, which are not included in the material. You will need to read those to understand how to extend the framework. The license being NOASSERTION is a significant upgrade cost: you cannot assume you have permission to modify and redistribute the code, so legal review is a prerequisite for any serious use. All of this points to a project that is more of a research artifact than a maintained library.

Editorial conclusion

Adopt Graph of Thoughts if you are a researcher or developer who needs to prototype and compare multi-step LLM reasoning strategies beyond chain-of-thought, and you are comfortable writing Python and configuring an LLM API. Do not adopt it if you need a production-grade, actively maintained framework with a clear license, or if you expect plug-and-play support for many model providers. Before using it, verify the controller configuration for your chosen LLM, check the license status, and confirm the last push (March 2026) still aligns with your project's timeline.

Official sources

1. Issues
2. Project website
3. README
4. Releases
5. spcl/graph-of-thoughts on GitHub