# XBOW Validation Benchmarks and the problem with a saturated evaluation set > XBOW Validation Benchmarks is a set of 104 Jeopardy-style CTF challenges in Docker Compose, built to evaluate web-based offensive tooling and AI agents that use it. The repository itself now warns that the set is outdated and saturated, which makes it a study in how benchmark lifecycles end. **xbow-engineering/validation-benchmarks** — XBOW Validation Benchmarks - Repository: https://github.com/xbow-engineering/validation-benchmarks - Website: https://xbow.com - Stars: 735 · Forks: 227 - Language: PHP - License: Apache-2.0 - Published: 2026-10-21 · Updated: 2026-10-21 · Language: en - Canonical page: https://hysenlabs.com/projects/xbow-engineering-validation-benchmarks ## The repository now carries a warning that it no longer measures anything The first thing the README says is that these benchmarks are outdated. The set was released in 2024, with minor bug fixes and build changes in 2025, and the reasoning is spelled out: LLM performance on cybersecurity tasks has improved significantly since then, and as of mid-2026 the benchmarks are saturated while the underlying vulnerabilities have been trained into the models. The stated conclusion is blunt. Industry-standard performance against the set is now approximately 100 percent, and the set is no longer useful for discriminating between models or attack frameworks. The repository is kept for historical purposes only. That framing is unusual and worth honouring in any evaluation of this project. A benchmark set published for a security audience, kept open after it stops discriminating, is a more useful artefact for benchmark designers than for model testers. It documents the shape of the problem, the packaging conventions, and the fact that secrecy of a test set buys a limited amount of time. ## One benchmark is one Docker Compose stack with one objective Each folder inside `benchmarks` corresponds to a single benchmark, and the README is precise about the constraint: a benchmark may contain multiple vulnerabilities and multiple steps, but there should only be one objective. The challenge format is restricted to Jeopardy-style capture the flag. A flag is hidden somewhere in the benchmark and the objective is to find it. The flag is not hardcoded in the repository. Instead, the design expects the flag to be injected at build time, which is what makes the set reusable and what prevents a solved flag from being copied between runs. At the top level the repository holds `LICENSE`, `Makefile`, `NOTICE`, `README.md`, `SECURITY.md`, `benchmarks/` and `common.mk`. The primary language of the repository is listed as PHP, and the topic tags are `ai`, `benchmarks` and `security`. The set contains 104 benchmarks in total, curated to mirror the variety of vulnerability classes the security team meets during routine pentesting and bug bounty work, with external contractors involved in the development. ## make build is the whole interface, and FLAG is the only variable Building an individual benchmark is a single make invocation from the benchmark directory: ```bash make build ``` That builds with a fixed flag. To override the flag value: ```bash make build FLAG=someflaggoeshere ``` The design separates the flag from the image content so the same benchmark can be handed to a different agent with a different answer, which is the mechanism that allows repeated evaluation without memorisation of the answer key. Underneath, the build is Docker Compose. The README states that the benchmark is built by running `docker compose build --build-arg flag=` and started with `docker compose up -d --wait`. The `--wait` flag is the important part. The compose file must ensure the benchmark is ready and running as soon as that command completes, which means service startup has to be deterministic. An agent that starts poking at a half-booted stack will score differently from one that does not, and the healthcheck requirement exists to remove that difference. ## docker-compose.yml has to expose ports to the agent and to other services The compose file carries a set of house rules that are worth reading as a specification for containerised challenge design. Each service should either use a commonly recognised base image, such as nginx, or be built from a Dockerfile in an appropriate sub-directory. Two directives handle networking and they mean different things. To provide ports accessible to another service, use the `export` directive. To provide ports accessible to the AI agent, use the `ports` directive. And to ensure services are started without race conditions, use `healthcheck` and `depends_on`. That split is the kind of detail that separates a benchmark that measures reasoning from one that measures luck with port guessing. The agent's entry point is published explicitly, and inter-service communication stays internal. Each service that is built in the compose file needs a sub-directory holding the Dockerfile and any other files required to build it. The Dockerfile must be named `Dockerfile`, must use `ARG flag` as the directive for the flag build argument, and must have that `ARG flag` placed as late in the file as possible so that changing the flag does not invalidate earlier layers. The README's closing remark on this point is that writing a good Dockerfile is an art, and the bar is that it works. ## benchmark-config.json carries the difficulty, the win condition and the canary Each benchmark has a metadata file at `benchmark/benchmark-config.json`, described as a dictionary with a fixed set of keys. `name` is the benchmark name. `description` is a short description of the kind a competitor in a competition would be given. `level` is `1` for easy, `2` for medium and `3` for hard, with the note that the original difficulty is chosen by the author or by an estimation of the difficulty the challenge may represent. `files` is an optional list of additional files required to solve the benchmark, with a warning to include only the files required for solving and no extraneous files. `tags` is a list of tags, and `tags.csv` holds the available ones. `win_condition` takes the value `flag` or `question`, and it states the condition the AI needs to satisfy. `canaries` is a list of canary strings taken from the Misaligned Power seeking Evaluations (MAPS) canary string published by the Alignment Research Center at alignment.org/canary. That canary mechanism is the point of the whole file. A model that has memorised the benchmark can be detected, because the published canary string appears in its output. ## The root Makefile dispatches on a BENCHMARK environment variable At the repository root, `Makefile` and `common.mk` provide a thin dispatcher over the per-benchmark directories. The Makefile sets `BENCHMARK_PATH` to `benchmarks/$(BENCHMARK)` and `BENCHMARK_JSON` to the benchmark.json file in that path. Every target depends on two checks. `check-env` fails with an error if `BENCHMARK` is not defined, and the message tells you it should be of the form `XBEN-xxx-yy`. `check_valid_bechmark` tests that the benchmark JSON file exists, and exits with a missing or invalid message otherwise. The `build`, `clean`, `run` and `stop` targets then delegate with `make -C` into the benchmark directory. The naming convention implied by that error message, `XBEN-xxx-yy`, is a useful detail for anyone writing tooling around the set, since it identifies each benchmark by a dataset-style identifier rather than by its folder name. The checks also mean that a mistyped identifier fails immediately with a clear message rather than part way through a Docker build. ## Apache 2.0, a NOTICE file, and deliberately unpatched vulnerabilities The project is licensed under the Apache License 2.0, with the text in the LICENSE file. There is also a NOTICE file and a SECURITY.md at the root, and the README carries a Notice section explaining the unusual nature of the contents: the repository contains intentional unfixed vulnerabilities because of its educational and experimental nature, deliberately left unpatched so that users can learn about security issues and their impacts in a controlled environment. The warning in that section is explicit. Do not use this code in production environments. The unfixed vulnerabilities can pose serious security risks, and the repository should be used only in isolated or sandboxed settings to prevent unintended consequences. On maintenance, the repository is not archived, but the last push was on 2026-07-07, and there are no GitHub releases. That gap between the last push and the mid-2026 saturation date in the README is consistent with the stated intent to keep the set for historical purposes only. The README also asks that questions be directed to the maintainers, which is the sensible route for anyone considering whether a given challenge is still appropriate for their environment. ## Conclusion XBOW Validation Benchmarks is worth reading as a specification for how to package an agent evaluation, and the `docker-compose.yml`, `benchmark/benchmark-config.json` and `make build` conventions are reusable even now that the results are meaningless. It is the wrong thing to measure a model with, since the README states the set is saturated at roughly 100 percent and that the underlying vulnerabilities have been trained into the models. Use it only in an isolated or sandboxed setting as the NOTICE file instructs, and if you need a discriminating evaluation, build a fresh challenge set rather than reviving these. ## FAQ ### How do I build a single XBOW Validation Benchmark? From the benchmark directory, run `make build` to build it with a fixed flag, or `make build FLAG=someflaggoeshere` to override the flag value. The build calls `docker compose build --build-arg flag=`, and the benchmark is started with `docker compose up -d --wait`. ### What do the XBOW Validation Benchmarks actually contain? The set has 104 benchmarks, each a Jeopardy-style capture the flag challenge in its own folder under `benchmarks`, built with Docker Compose. A flag is hidden in the benchmark and injected at build time rather than hardcoded, and each challenge should have a single objective even if it involves multiple vulnerabilities and multiple steps. ### Are the XBOW Validation Benchmarks still useful for evaluating models? The README states that as of mid-2026 the benchmarks are saturated, that performance against them is approximately 100 percent, and that they are no longer useful for discriminating between models or attack frameworks. The repository is kept for historical purposes only. ### What metadata does benchmark-config.json require? The file carries `name`, `description`, `level` of `1`, `2` or `3`, an optional `files` list, `win_condition` of `flag` or `question`, a `tags` list drawn from `tags.csv`, and a `canaries` list using the MAPS canary string from the Alignment Research Center. ### Can I run the XBOW Validation Benchmarks in production? No. The README states that the repository contains intentional unfixed vulnerabilities, warns against production use because those vulnerabilities can pose serious security risks, and says the repository should be used only in isolated or sandboxed settings. ## Sources - [Issues](https://github.com/xbow-engineering/validation-benchmarks/issues) - [License: Apache-2.0](https://github.com/xbow-engineering/validation-benchmarks/blob/main/LICENSE) - [Project website](https://xbow.com) - [README](https://github.com/xbow-engineering/validation-benchmarks/blob/main/README.md) - [xbow-engineering/validation-benchmarks on GitHub](https://github.com/xbow-engineering/validation-benchmarks) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/xbow-engineering-validation-benchmarks