AIOpsLab: a benchmark framework for evaluating autonomous AIOps agents on live Kubernetes

The evaluation gap AIOpsLab addresses: live Kubernetes instead of replay data

AIOps agents are typically evaluated against static logs or replay data, without a live environment that responds to their actions. AIOpsLab replaces that static setup with one in which agents interact with a running Kubernetes microservice deployment, observe real telemetry, and take actions that the framework scores.

Microsoft Research built AIOpsLab with four explicit goals: reproducibility, standardization, interoperability, and scalability for AIOps benchmarks. Putting those goals in a single framework means a research team can run the same set of problems with different agent backends and compare results on equal footing. The repository carried no GitHub releases as of its last push on 2026-10-01, and version 0.1.0 in pyproject.toml reflects that early stage.

A built-in benchmark suite ships with a set of problems. The registry file at aiopslab/orchestrator/problems/registry.py holds the current list starting at line 15. Adding domain-specific problems means extending that registry rather than rewriting the evaluation harness.

Fault injection, workload generation, and the scoring loop

Four subsystems form the evaluation loop: environment deployment, fault injection, workload generation, and telemetry export. Separating those concerns means faults can be changed without touching the workload generator, and telemetry can be rerouted without modifying the injector.

An agent in this loop reads telemetry, decides on an action such as restarting a service or changing a configuration, and submits that action through the framework interface. Scoring follows from whether the action resolves the injected fault. Because the environment is live Kubernetes, the telemetry reflects actual state rather than a prerecorded trace.

Agents that do well on replay data may behave differently when their actions have consequences, and this framework is designed to surface that difference. For agents that call external LLMs, multiple client implementations are in the clients/ directory, covering GPT-4, Qwen, DeepSeek, Groq, and OpenRouter. Some clients use Azure identity-based authentication instead of a static API key, which removes the need to store a secret in .env for Azure deployments.

Python's version ceiling and why apt breaks the Poetry install

The README's requirements list says Python >= 3.11. On closer inspection, pyproject.toml constrains the supported range to `python = ">=3.11,<3.13"`, which means Python 3.13 is not supported even though it is a stable release. Anyone who upgraded their system Python to 3.13 will need to install an earlier version alongside it.

Installation steps begin with a standard apt command:

sudo apt update
sudo apt install python3.11 python3.11-venv python3.11-dev -y

Poetry is the recommended package manager. An explicit warning in the README says not to install Poetry with apt, because the apt version is outdated and may not work with the lock file. Using the official installer is the required route:

curl -sSL https://install.python-poetry.org | python3.11 -
export PATH="$HOME/.local/bin:$PATH"
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc

Cloning and installing then takes four commands:

git clone --recurse-submodules <CLONE_PATH_TO_THE_REPO>
cd AIOpsLab
poetry env use python3.11
poetry install

A "lock file not compatible" error means running `poetry lock` first, then `poetry install`. pip is available as an alternative with `pip install -e .`, but Poetry is what the troubleshooting guidance covers.

Three cluster backends and the fault injector gap in Mode B

AIOpsLab requires a Kubernetes cluster to run. The Quick Start offers three paths to one. A local simulated cluster using kind is the first:

kind create cluster --config kind/kind-config-x86.yaml

An ARM variant uses kind/kind-config-arm.yaml. Proxy users must avoid exporting the HTTP proxy as 172.17.0.1, because cluster nodes inherit the proxy from the host environment and the Docker container. Docker has no built-in SOCKS5 proxy support, and the README directs users to Privoxy to forward SOCKS5 to HTTP in that case. Running VLLM and an LLM agent locally with Privoxy requires setting `export no_proxy=localhost` to prevent routing errors.

A remote Kubernetes cluster managed with Ansible is the second path, using playbooks the repository provides for CloudLab and similar providers.

The third path provisions Azure VMs with Terraform and Ansible:

python3 scripts/terraform/deploy.py --apply --resource-group <your-rg> --workers 2 --mode B

Mode B runs AIOpsLab on the local laptop with a remote kubectl context. Mode A runs it on the controller VM and is described as recommended for full functionality. Certain fault injectors, VirtualizationFaultInjector is named specifically, require Docker on the local machine when using Mode B, which Mode A avoids.

Running problems from cli.py and setting API keys for baseline agents

Once a cluster is running and config.yml is in place, cli.py provides a human-in-the-loop entry point. Config starts from aiopslab/config.yml.example, which you copy to config.yml. Setting k8s_host to the hostname of the cluster control plane and k8s_user to the username on that node connects the framework to the cluster. For kind clusters, k8s_host is "kind"; for on-cluster deployments, it is "localhost".

python3 cli.py

Inside the shell, starting a problem and submitting a solution takes two commands:

(aiopslab) $ start misconfig_app_hotel_res-detection-1
(aiopslab) $ submit("Yes")

Running the GPT-4 baseline agent requires an OpenAI API key in a .env file at the project root:

echo "OPENAI_API_KEY=<YOUR_OPENAI_API_KEY>" > .env
python3 clients/gpt.py

.env.example also shows slots for DEEPSEEK_API_KEY, DASHSCOPE_API_KEY for Qwen, OPENROUTER_API_KEY, and GROQ_API_KEY. OPENROUTER_MODEL sets the model for OpenRouter clients, with openai/gpt-4o-mini as the example value. Weights and Biases logging is controlled by USE_WANDB, which defaults to false.

vllm in the default install and what it requires

pyproject.toml defines a clients dependency group that includes vllm = "^0.7.3". VLLM requires CUDA and a compatible GPU to run inference. This group installs by default with `poetry install`.

A note in pyproject.toml names CI as the primary case where excluding clients makes sense:

poetry install --without clients

The clients group also includes autogen-agentchat = "^0.2.40", transformers in the range >=4.49,<6.0, groq = "^0.28.0", and flwr = "^1.19.0". All install alongside the core framework unless you pass that flag.

Running the benchmarking loop without GPU hardware is possible using API-backed agents, where inference happens on a remote endpoint. Any client that runs inference locally through VLLM cannot operate on a CPU-only machine. Whether your deployment needs VLLM at all depends on which clients you plan to use, and the flag makes that separation clean.

Where AIOpsLab is the wrong tool, and what the Quick Start leaves out

AIOpsLab is not a production monitoring or incident response tool. Deploying it in a production environment would require replacing its microservice test targets with production services, which the README does not describe as a supported use case.

An explicit TODO comment appears inside the Quick Start section of the README: "Add instructions for both local cluster and remote cluster." That comment is still present in the repository today, which means the cluster setup guidance is explicitly acknowledged as incomplete. Teams following the Quick Start will encounter that gap.

Evaluating an agent against real telemetry from a past production incident is not a supported workflow. AIOpsLab deploys its own microservice targets and generates its own workloads, so offline replay of historical data is not a path the framework provides.

Teams without Kubernetes administration experience will find the setup cost high. All three cluster paths require external tooling: kind for local runs, Ansible for remote clusters, and Terraform plus Ansible for Azure. kubernetes = "^30.1.0" and Helm are listed as core requirements, not optional add-ons, so the cluster dependency is present from the first install command.

Editorial conclusion

AIOpsLab suits research teams and AIOps practitioners who already operate Kubernetes and want a structured way to compare agent strategies against reproducible fault scenarios. It is not ready for teams without cluster administration experience: setup requires Helm, Poetry, and kubectl access before a single problem runs. Before adopting it, verify your Python version against the upper bound in pyproject.toml, which caps support below 3.13, and decide whether you need Mode A on Azure Terraform for the fault injectors that require Docker on the controller node.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. microsoft/AIOpsLab on GitHub
4. README