ThunderAgent: program-aware scheduling between your agents and vLLM or SGLang

The scheduling gap ThunderAgent targets

Standard inference servers treat each HTTP request as an independent unit. An agentic workflow is not that. A single agent run issues a sequence of calls: a planning call, a tool call, a follow-up reasoning call, another tool call. Those calls share context and often share a prefix. When the server sees them as unrelated requests, it has no way to place them on the same GPU node or keep their KV cache resident, so prefixes get recomputed and memory across nodes drifts out of balance. ThunderAgent's answer is the program abstraction. A program is the unit of scheduling, and every request that carries the same program_id is treated as part of one logical job. The README describes the result as an agentic program-aware scheduler that increases KV-cache hit rate and reduces memory imbalance across nodes. The audience is narrow and specific: teams running multi-step agents against vLLM or SGLang on more than one GPU node, where the cost of a cache miss is measured in GPU seconds.

Where ThunderAgent sits in the request path

ThunderAgent is a proxy with a scheduler inside it. The README's architecture description places it between agent clients and the infrastructure layer, acting as an agentic workflow scheduler. On the upstream side it speaks the OpenAI chat completions API, so an existing client keeps working after one change: add program_id to extra_body. On the downstream side it forwards to vLLM or SGLang. The README states that both backends are supported. The second responsibility is tool management. ThunderAgent provides what the README calls a unified tool management interface for resources like Docker containers and remote APIs, with tool-call lifecycle management and automatic resource reclaim. That matters for long rollouts, where a container started for one tool call can leak if the agent crashes or the rollout is cancelled. The request payload reflects this: alongside program_id, the README shows an optional docker_ids field, described as relevant when you use Docker for your agentic workflow. So the same client that sends a prompt can also declare which containers belong to the current program, and ThunderAgent is expected to track their lifecycle.

Installing and launching the proxy

The README gives a source install, and there is no package index step documented. Clone the repository, change into it, and install in editable mode:

git clone https://github.com/ThunderAgent-org/ThunderAgent.git cd ThunderAgent pip install -e .

You then pick a backend. The README's example uses vLLM, installed with uv pip install vllm --torch-backend=auto, followed by vllm serve Qwen/Qwen3-32B --port 8000. ThunderAgent itself is started with the thunderagent command:

thunderagent --backend-type vllm --backends http://localhost:8000 --port 9000 --metrics --profile

The README is explicit that requests must go through port 9000, not the backend port. The flags visible in the example are --backend-type, --backends, --port, --metrics and --profile. The first two are the ones you cannot omit: --backend-type selects vLLM or SGLang, and --backends takes the backend URL. --metrics and --profile turn on the real-time visualization of agentic trajectory metrics, which the README describes as covering total tokens, tool-use time, and per-program profiling. The README does not document the full flag list, the config file format, or the default values for anything beyond what the example shows. Treat the example as the whole documented surface until you read the source.

The client-side change is one field, and that is the point

The integration example is short enough to quote directly. An existing call looks like openai.client.chat.completions.create(model=..., messages=messages). The ThunderAgent version adds an extra_body dictionary with a single key:

extra_body["program_id"] = "unique_id"

If the workflow uses Docker, the README adds a second optional key, extra_body["docker_ids"], holding a list of container identifiers. Nothing else in the call changes. That design choice is the strongest part of the project. It means you can adopt ThunderAgent without rewriting your agent loop, and you can compare it against a direct connection to vLLM by flipping the base URL and dropping the extra_body. The cost is that correctness now depends on the client generating a program_id that is stable across the whole agent run and unique across concurrent runs. The README does not specify what happens when two live programs share an identifier, or how the scheduler reacts to a program_id that never terminates. Those are the questions to answer before production use, and the README does not answer them.

What the throughput numbers do and do not cover

The README claims throughput improvements of 1.5x to 3.6x, and repeats the figure in the results section as applying to vLLM across SWE-Agent, OpenHands and ToolOrchestra workloads. It attributes the gain to higher KV-cache hit rate and reduced memory imbalance across nodes. These are the project's own numbers, reported in its paper and README, and they are not reproduced here. The range itself is informative: a 1.5x floor and a 3.6x ceiling means the benefit is workload-dependent by a factor of more than two. Workloads with long shared prefixes and many sequential calls per program should land near the top. Workloads with short, unrelated prompts should land near the bottom, or below 1x once you account for the extra network hop through the proxy. The README does not break the range down by workload, so you cannot tell from the documentation which of SWE-Agent, OpenHands or ToolOrchestra produced which end of the range. Separately, the news section states that SkyRL integration includes an example recipe reporting a 3x SWE Agent rollout speedup on 40 H100 GPUs. That is a training-rollout figure on a specific cluster, not a serving figure, and it should not be read as a general claim.

Research-grade packaging and the maintenance question

Several signals point the same way. No releases were retrieved for this repository, so installation is from source and there is no tagged version to pin. The README's own integration news is dated 2026, and the project was accepted to ICML 2026 as a Spotlight. It has been integrated into NVIDIA Dynamo, where the proposed program abstraction is described as operating as a first-class scheduling unit, and into SkyRL for agentic RL training. The citation block lists ten authors across Georgia Tech, Rice and elsewhere, and the contact address is a university email offering technical consulting and sponsorship discussions for enterprises. That is the profile of a research system with industrial interest, not a product with a support contract. The practical consequences: expect the CLI flags and the extra_body contract to move, expect to read source when the README runs out, and budget for a fork if you depend on it. The MIT license is permissive, so forking and vendoring are available options, but this is a description of the license text, not legal advice.

When a plain inference server is the better answer

The direct alternative is what you are probably running now: point the agent at vLLM or SGLang and let the server's own continuous batching and prefix caching do the work. vLLM already caches shared prefixes and already batches across concurrent requests. ThunderAgent's premise is that the server cannot see which requests belong together, so it cannot make placement or eviction decisions at the program level. If your agent traffic is already grouped by a load balancer with session affinity, or if your prefixes are short, that premise does not hold for you and the proxy is a net loss. The same reasoning applies to single-turn or few-turn agents: with one or two calls per run there is no program structure to exploit, and you are paying for a hop. The second alternative is a general orchestration framework that manages agents and tools but does not touch the inference scheduler. That gets you lifecycle management without the scheduling claim, at the cost of leaving the KV-cache problem unsolved. ThunderAgent's distinguishing move is doing both in one process, and that is also why it is harder to adopt than either alternative alone.

Editorial conclusion

Adopt ThunderAgent if you already run vLLM or SGLang and your agent traffic is fragmented enough that per-request scheduling is leaving KV-cache capacity idle. Do not adopt it if you need a stable, versioned dependency with published releases, or if your agents are single-turn calls where program grouping has nothing to group. Before rolling it out, clone the repository, run pip install -e ., start vllm serve on port 8000, launch thunderagent --backend-type vllm --backends http://localhost:8000 --port 9000 --metrics --profile, and confirm in your own metrics that program-aware grouping actually moves KV-cache hit rate on your workload. If it does not, the extra hop on port 9000 is pure latency.

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. ThunderAgent-org/ThunderAgent on GitHub