Commerce Agent Bench: Testing Long-Horizon Commerce Agents in Reproducible Service Replicas
CommerceAgentBench: Benchmarking Long-Horizon Agents in High-Fidelity, Stateful, and Reproducible Replicas of Real Online Services
At a glance
- What is it?
- Commerce Agent Bench is an Apache-2.0 Python benchmark harness from the Accio team at Alibaba International. It ships 107 tasks against local mock services so agents must change state, not just describe it.
- Who is it for?
- Adopt Commerce Agent Bench if you are building or selecting an agent that must operate commerce interfaces, and you need a stateful, containerized suite rather than a question-answering set. Skip it if your target is a general assistant, a coding agent, or anything outside commerce and sourcing workflows, and skip it if you cannot run Docker or provide model credentials.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 18 days ago.
- What is it written in?
- Mainly HTML, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 16, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap Commerce Agent Bench targets: agents that talk about commerce instead of doing it
Most agent evaluations ask a model to reason about a scenario and score the answer. Commerce Agent Bench was built for a different question: can an agent finish a long business workflow and leave the underlying system in the right state? The README frames the suite as covering "browser operations, native-style CLI tools, API/MCP workflows, document and spreadsheet production, public-web research, supplier analysis, product publishing, logistics, and commerce operations."
The intended audience is narrow and specific. It is for teams that build or select agents which operate commerce and sourcing software: listing a product, booking freight, editing a storefront theme. The pyproject.toml classifies the package as Intended Audience :: Science/Research and Development Status :: 4 - Beta, which matches what the repository looks like: a research harness with a published leaderboard, not a product library. If your agent never touches a catalog, a shipment, or a supplier record, this benchmark measures the wrong thing.
How the harness works: fresh containers, local replicas, per-task verifiers
The mechanism is stated plainly in the README: "Every task runs in a fresh container and is graded by its own deterministic or LLM-assisted verifier." Instead of pointing agents at production SaaS accounts, the suite runs "reproducible local replicas of commerce and business software, so agents must operate interfaces and change state." The mock services model SaaS, commerce, messaging, document, and operational systems.
The task collection is 107 items, split as 53 CLI, 28 browser, 16 file, and 10 API/MCP. A second axis describes what the agent needs: 65 text-only, 20 browser-text-capable, and 22 vision-required. That split matters when you pick a model, because a text-only model cannot attempt the 22 vision tasks at all.
Outputs are preserved per run: the resolved configuration, trajectory, verifier result, artifacts, logs, and container metadata. The repository layout reflects this, with bench_core/, configs/, datasets_domain_v1/, docker/, scripts/, and tests/ at the top level, plus a real_replica_bench/ directory that carries the project's former name.
Installing Commerce Agent Bench and running a first task
The package requires Python 3.11 or newer and installs from the repository as an editable package. Note the dependency comment in pyproject.toml: verifier graders run on the host, not inside the workstation image, and two graders import openpyxl and PyYAML. On a fresh machine that only ran the install below, those graders would otherwise emit "openpyxl not installed" or "pyyaml import failed" and zero-score an entire task family. Both libraries are declared as dependencies, so a normal install pulls them in.
pip install -e .After that, the repository ships a commerce- console entry point under [project.scripts] in pyproject.toml. The README's quick-start section is where the exact invocation lives; the truncated pyproject.toml shows the script prefix but not the full command, so check the README before running it. Container execution is part of the design, since every task runs in a fresh container, so a working Docker setup on the host is a prerequisite rather than an option.
For a first real run, pick a single task rather than the full 107-task collection. The suite is designed around task_id alignment: the README states that reference results are "aligned by task_id over the complete 107-task collection." A single-task run produces the same artifact set (configuration, trajectory, verifier result, artifacts, logs, container metadata), which is enough to confirm the pipeline works before you spend hours on a full sweep.
The reproducibility contract and what the published numbers actually mean
The README is unusually careful about what its tables do and do not measure, and this is the most useful part of the document. Three harnesses are reported (Pi, OpenClaw, and Accio), each with thirteen model families, and the README says the same thirteen appear in all three tables "so every row compares directly across the Pi, OpenClaw and Accio tables."
Two caveats are stated explicitly. First, the published scores were produced through Accio-managed evaluation endpoints with gemini-3.1-pro-preview as the judge, while the public path in the repository uses bring-your-own credentials. A score you reproduce locally may not match the leaderboard because the judge and endpoint differ. Second, steps, time, and tokens are described as "descriptive telemetry: tool granularity, runtime scheduling, and provider usage accounting differ, so these values are not normalized efficiency scores." Treat the token and step columns as context, not as a ranking signal.
The README also notes that every model ran with thinking enabled at its provider's default reasoning effort, and that this default differs by vendor, so the harness tier does too. That is a real confound, acknowledged rather than hidden, and it means cross-vendor comparisons in those tables carry an asterisk the numbers themselves do not show.
Where Commerce Agent Bench is the wrong tool
The narrowness is the point, and it is also the limitation. A 107-task suite over commerce and sourcing surfaces will not tell you whether an agent is good at writing code, answering support tickets, or navigating arbitrary websites. The mock replicas are local models of SaaS, commerce, messaging, and document systems, so an agent that scores well here has demonstrated competence against those replicas, not against the live services they imitate.
The pass rates in the reference tables make the difficulty clear: the top Pi result shown is 65/107 (60.7%), and the table continues downward from there. That is a hard benchmark, which is good for discrimination and bad for quick smoke tests. If you need a fast regression signal in CI, a 107-task containerized suite with vision-required tasks and LLM-assisted graders is heavy machinery.
There is also a licensing boundary inside the repository. The distributed Python package is the Apache-2.0 half; the task suite in datasets_domain_v1/ is under CC BY 4.0 and is not packaged, per the comment in pyproject.toml and the LICENSE-DATA file. If you plan to redistribute tasks or derived data, read LICENSE-DATA before assuming the Apache-2.0 terms cover everything you downloaded.
How it differs from Business Arena, the team's other benchmark
The README points to a sibling project from the same team: Business Arena, described as asking "can an agent run a seller business over a 30-day market horizon?" The two share a domain and a publisher but differ in what they simulate. Business Arena compresses a month of market activity into a horizon, which is a time-scale simulation problem. Commerce Agent Bench instead runs discrete tasks in fresh containers against stateful replicas of specific software, which is an interface-operation problem.
That distinction should drive your choice. If your question is whether an agent can sustain decisions across a simulated market over time, Business Arena is the closer fit. If your question is whether an agent can complete a concrete workflow in a listing form, a freight booking search, or a storefront theme editor and leave the system in a verifiable state, Commerce Agent Bench is the one that runs the interfaces. Neither substitutes for the other, and the README does not claim they do.
Maintenance, versioning, and upgrade cost
The repository is not archived, and its last push was on 2026-08-29. The README badge and pyproject.toml both report version 1.3.1, and there are no retrieved releases, so version tracking currently happens through the repository itself rather than a release feed. The project also carries a rename history: it was previously known as RealReplicaBench, and the real_replica_bench/ directory at the top level is the residue of that. Expect naming drift if you follow older write-ups.
Upgrade cost is mostly in the task suite, not the harness code. The dependency list in pyproject.toml is deliberately short, with a comment saying to "keep this list tight; only pin libs a grader uses today," so the Python surface is small. The larger cost is that grading depends on host-side libraries: if a future task adds a grader that imports something not in the dependency list, that task family silently scores zero on a clean install. That failure mode is documented in the pyproject.toml comment itself, which is a good sign, but it means you should re-read that comment after any upgrade rather than trusting pip to surface the problem.
On licensing: the Python package is Apache-2.0, the task suite in datasets_domain_v1/ is CC BY 4.0 and is not packaged, and the repository also carries LICENSE-DATA and THIRD_PARTY_NOTICES.md. That is a two-licence arrangement, and the practical consequence is that vendoring the package and redistributing the tasks are governed by different terms. Read LICENSE-DATA and THIRD_PARTY_NOTICES.md for your own situation; this is a description of the files present, not legal advice.
Editorial conclusion
Adopt Commerce Agent Bench if you are building or selecting an agent that must operate commerce interfaces, and you need a stateful, containerized suite rather than a question-answering set. Skip it if your target is a general assistant, a coding agent, or anything outside commerce and sourcing workflows, and skip it if you cannot run Docker or provide model credentials. Before committing, verify three things: that your Python is 3.11 or newer, that openpyxl and PyYAML are installed on the host because two graders import them and otherwise score zero, and that the task families you care about match the 53 CLI, 28 browser, 16 file, and 10 API/MCP split. The repository was last pushed on 2026-08-29 and is not archived.
Frequently asked questions
What is agentic commerce and how does Commerce Agent Bench relate to it?
Agentic commerce refers to AI agents carrying out commerce workflows rather than only answering questions about them. Commerce Agent Bench evaluates exactly that: its tasks cover browser operations, CLI tools, API/MCP workflows, document production, supplier analysis, product publishing, and logistics, and every task is graded by its own verifier.
Which AI agents are best for e-commerce, according to Commerce Agent Bench?
The repository publishes reference results across three harnesses (Pi, OpenClaw, and Accio), each with thirteen model families aligned by task_id over the 107-task collection. The README states the live leaderboard is the source of record and the tables in the repository are a snapshot.
What are some examples of software agents that Commerce Agent Bench is designed to test?
The tasks exercise agents that operate commerce and business software: product publishing, freight booking, storefront theme customization, supplier analysis, and logistics workflows. The suite runs these against local replicas of SaaS, commerce, messaging, document, and operational systems rather than production accounts.
Community notes