MCPMark: a stress test for MCP tool use across Notion, GitHub, Filesystem, Postgres and Playwright

The gap MCPMark targets: tool use measured against live servers, not canned calls

Most function-calling evaluations hand a model a schema and a mocked response. MCPMark instead runs tasks against actual MCP servers for Notion, GitHub, Filesystem, Postgres and Playwright, and checks the result with what the README calls strict automated verification. The intended audience is stated plainly: researchers and engineers who need a reproducible, extensible benchmark. That framing matters, because the project is not a library you embed in a product. It is a harness you run to get a number.

The practical problem it addresses is comparability. If two teams each wire up their own Notion workspace and their own GitHub org, their pass rates are not measuring the same thing. MCPMark's answer is a fixed task inventory with isolated sandboxes and per-task verifiers, plus a documented convention for where results land on disk. The README claims 127 tasks in the standard suite as of the current revision, with 10 easy tasks per MCP server added for smaller open-source models.

One caveat sits at the top of the README and should be read before anything else. MCPMark Verified is now the default task set, with every environment version-pinned and every verifier stabilized. Earlier task versions are deprecated and, in the maintainers' own words, not directly comparable. Any pass rate you see quoted without a Verified label is suspect.

How the harness is put together: pipeline entry point, task suites, and the results tree

The orchestration entry point is a Python module invoked as python -m pipeline. It takes an MCP service name, a repeat count, a model list and one or more task identifiers. The README's minimal example runs a filesystem task with --mcp filesystem, --k 1, --models gpt-5 and --tasks file_property/size_classification. Filesystem is the deliberate starting point because it needs no external accounts.

Tasks are laid out on disk as tasks/<mcp>/<task_suite>/<category>/<task>/. Two suites exist. standard is the default and holds the full benchmark; easy holds 10 lightweight tasks per MCP and is described as suitable for smoke tests and CI. The suite is selected with --task-suite easy, and the README notes GitHub's easy tasks are already present under tasks/github/easy.

Output paths encode what was run. Standard-suite results go to ./results/{exp_name}/{model}__{mcp}/run-*/ and easy-suite results to ./results/{exp_name}/{model}__{mcp}-easy/run-*/. The README gives ./results/test-run/gpt-5__filesystem/run-1/ and ./results/test-run/gpt-5__github-easy/run-1/ as illustrations. That naming convention is the mechanism that keeps a model-plus-service pair separable when you aggregate, and it is worth preserving if you script around the output.

Two agent scaffolds are available. MCPMarkAgent is the default; passing --agent react to pipeline.py switches to a ReAct-style agent. The README states that PRs for new agent scaffolds are welcome, which tells you the scaffold layer is treated as an extension point rather than a fixed part of the design.

Getting it running: .mcp_env, pip install -e ., and the credentials each service wants

Setup starts with a clone and a single env file at the repository root named .mcp_env. The README's guidance is to set only what you need. The OpenAI block is OPENAI_BASE_URL and OPENAI_API_KEY. Notion tasks need SOURCE_NOTION_API_KEY, EVAL_NOTION_API_KEY and EVAL_PARENT_PAGE_TITLE, which the example sets to "MCPMark Eval Hub". The source and eval keys are separate, which is consistent with the isolation claim: the harness reads from one workspace and writes into another.

GitHub tasks use GITHUB_TOKENS, described as a comma-separated pool for rate limits, plus GITHUB_EVAL_ORG. Postgres tasks take POSTGRES_HOST, POSTGRES_PORT, POSTGRES_USERNAME and POSTGRES_PASSWORD. Playwright tasks take PLAYWRIGHT_BROWSER, either chromium or firefox, and PLAYWRIGHT_HEADLESS.

Installation is pip install -e . from the repository root. If you intend to run browser-based tasks, playwright install must be run first. Docker is the alternative path, built with ./build-docker.sh. The README states the project is fully validated on macOS and Linux; Windows is not mentioned, so treat it as unverified rather than supported.

A few behaviours are documented as operational rather than optional. Failed tasks auto-retry and resume. GitHub @mentions are obfuscated during evaluation to avoid notification spam, which is the kind of detail that only appears after someone has been paged by a benchmark. And --compaction-token summarizes long conversations to avoid context overflow, added in December per the news list.

Version pinning is the design decision that costs the most to maintain

The README's January note pins GitHub MCP Server to v0.15.0 and says the project switched to Docker for version control. Notion MCP Server is pinned to @1.9.1, with an explicit remark that Notion released 2.0 but it has many bugs and is not recommended. Those are not incidental notes. They are the reason the Verified set can claim reproducibility at all, and they are also the reason the benchmark has an ongoing maintenance bill.

Every upstream server release creates a decision: bump the pin and invalidate prior numbers, or hold the pin and accept that you are measuring against a version the vendor no longer ships. The Notion 2.0 comment shows the maintainers choosing to hold. That is defensible for a benchmark, but it means a high score on MCPMark's Notion tasks is a statement about @1.9.1, not about Notion's current server.

The same tension appears in the deprecation notice. Moving to Verified deprecated earlier task versions and made their results non-comparable. If your team published a number last year, it no longer sits on the same axis as a number published today. For a one-off evaluation that is fine. For anything you intend to track over time, budget for re-runs.

There is also a subtler cost: the task inventory itself moves. The README records 50 easy tasks added in November and ReAct agent support in October. A benchmark whose task set grows will produce different aggregate scores even at a fixed model and fixed server versions, because the denominator changed.

Where MCPMark is the wrong tool

MCPMark is a benchmark, not a regression suite for your own MCP server. The tasks are written against Notion, GitHub, Filesystem, Postgres and Playwright, and the verifiers check outcomes in those environments. If you ship an internal MCP server for, say, a ticketing system, MCPMark has no tasks for it. Extending the benchmark means writing tasks and verifiers that match the tasks/<mcp>/<task_suite>/<category>/<task>/ layout and the results convention, which is real work and not a configuration change.

The second boundary is credentials and isolation. The README describes isolated environments that do not pollute your accounts or data, and the Notion setup separates SOURCE and EVAL keys for that reason. But the GitHub path writes into GITHUB_EVAL_ORG, and Postgres tasks point at a live host and port. If you cannot provide throwaway infrastructure for those services, you are limited to the Filesystem tasks, which need no external accounts. That is a genuine narrowing of what you can measure.

The third is scale. Running the full standard suite across several models and multiple repeats multiplies both API spend and wall-clock time, and the harness is doing browser automation for Playwright tasks on top of that. The --k flag exists precisely because single runs are noisy. A one-shot --k 1 run, as in the quickstart, is a smoke test, not a measurement you should quote.

Finally, the README's own numbers illustrate how much headroom remains. On the Verified set, gpt-5.5 at xhigh leads at 92.9% and kimi-k2.7 reaches 81.1%. Earlier, non-Verified figures in the news list sit far lower: gemini-3-pro-preview at 50.6% plus or minus 2.3%, gpt-5-high at 51.6%, deepseek-v3.2-thinking at 36.8%. Those two clusters are not a trend line. They are different task versions, and the README says so.

Alternatives and the actual difference in approach

The natural comparison is a general agentic benchmark such as SWE-bench, which scores patches against repository test suites. The difference is what counts as success. SWE-bench verifies a code change by running tests; MCPMark verifies the state of an external service after an agent has driven it through MCP tool calls. A model that writes excellent code but cannot sequence tool calls across a Notion page hierarchy will score well on one and poorly on the other. If your product is an agent that operates third-party services, MCPMark's axis is closer to your problem.

The second comparison is a homegrown harness. Many teams already have a script that points a model at their MCP server and checks a handful of outputs. That approach gives you exact fit to your server and no comparability with anyone else. MCPMark gives you the reverse: a fixed, version-pinned inventory you can compare against published numbers, at the cost of measuring services you may not care about.

A third option is the trajectory logs the README links on Hugging Face. If your goal is to study how agents fail rather than to rank models, reading recorded trajectories may be more informative than running the harness yourself, and it costs no API budget. MCPMark's contribution there is the artifact, not only the score.

Licence, upkeep, and what to verify before you publish a number

The repository is Apache-2.0. That permits commercial and internal use, modification and redistribution, with the usual conditions around preserving notices and stating changes. It does not grant rights to the Notion, GitHub or Playwright services the tasks drive, and each of those has its own terms plus its own rate limits. GITHUB_TOKENS is a pool for exactly that reason. Nothing here is legal advice; if you plan to redistribute a modified task set, read the licence text and the service terms yourself.

Upgrade cost is the part most teams underestimate. Pinned server versions mean a bump can change scores without any change to your model or your prompts. Verifier stabilization in the Verified set means earlier results are explicitly deprecated. If you maintain a dashboard, record the MCPMark revision alongside the model name, and record the task suite. The results tree already separates standard from easy via the -easy suffix on the directory name, so the information is there if you keep it.

Before quoting any figure, confirm four things: that the run used the Verified set, which --task-suite produced it, which pinned server versions were in play, and what --k was. The README's own quickstart uses --k 1 for speed, and the gemini-3-pro-preview result is reported with a plus or minus 2.3% interval, which tells you single runs are not the intended unit of reporting. Start from the filesystem task in the quickstart to confirm your environment works, then decide whether the services you actually care about are among the five the benchmark covers.

Editorial conclusion

Adopt MCPMark if you need a reproducible way to compare models or agent scaffolds against real MCP servers and you are willing to re-run after each Verified-set change. Do not adopt it if you need a frozen leaderboard across quarters: the maintainers deprecated earlier task versions, so numbers from before the Verified switch are not comparable with current ones. Before publishing any figure, confirm which task suite and which pinned server versions produced it, and check whether the task you are reporting lives under tasks/<mcp>/standard or tasks/<mcp>/easy.

Official sources

1. eval-sys/mcpmark on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases