EdgeBench: a 12-hour agent benchmark built on SForge
EdgeBench: Unveiling scaling laws of learning from real-world environments
At a glance
- What is it?
- ByteDance Seed's EdgeBench measures how AI agents improve over long interaction windows rather than one-shot accuracy. The repository ships SForge, the Python harness that runs those tasks in Docker.
- Who is it for?
- Adopt EdgeBench if you need a long-horizon signal for an agent and can afford multi-hour container runs; the 51 open tasks and the SForge harness are enough to reproduce a trajectory. Skip it if you want a fast pass/fail check on a small model, since the published curves only separate models after several hours.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 9 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 17, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What EdgeBench measures that a one-shot eval cannot
Most agent benchmarks hand a model a task, collect one answer, and score it. EdgeBench is built around the opposite assumption: that the interesting signal is the shape of the improvement curve. The README describes 134 real-world tasks in executable environments with multi-level feedback, where agents iterate for 12+ hours per task and the benchmark tracks the full trajectory rather than the final number. The public release covers 51 of those tasks plus the evaluation framework. The stated finding is that performance follows a log-sigmoid scaling law as a function of interaction time, with R^2 = 0.998 across the 134 tasks. That claim is the point of the project. If you only care whether a model can solve a task at all, the leaderboard's @2h column is a worse use of your GPU hours than a conventional benchmark. The 12-hour column is where the models separate. On the full benchmark, the top entry moves from 39.0 at 2 hours to 51.3 at 12 hours, and the gap between the top two entries widens from 2.2 points at 2 hours to 2.9 points at 12 hours, which tells you the ranking is stable but the absolute headroom is small. The audience is research teams comparing agent scaffolds and model backends on the same task set, not application developers looking for a smoke test.
The SForge harness: Docker containers, a CLI, and a local web UI
EdgeBench is not a library you import into your own agent loop. It ships a separate harness called SForge, distributed as the Python package sforge (version 1.1.0) and installed as the console script sforge. The dependency list in pyproject.toml is the clearest description of the architecture: docker>=7.0 for container control, fastapi, uvicorn[standard], python-multipart and jinja2 for a local web interface, pyyaml>=6.0 for task configuration, huggingface-hub>=0.20 for pulling task assets, and tqdm and rich for terminal output. There is an optional e2b extra pinned to e2b>=2.46.0,<3 for running tasks on E2B sandboxes instead of local Docker. The repository layout confirms two execution modes: examples/all-tasks-k8s/ and examples/single-task-docker/. So a task is a container image plus a YAML description, the harness schedules it, the agent interacts with the running environment, and scores come back at fixed time budgets. Python support is declared as >=3.10,<3.14, which is a real constraint: a 3.14 interpreter will not install the package. The README does not document how partial failures are handled when a container dies mid-run, and the per-task table uses a dash for missing valid results, which suggests those cases are recorded rather than retried.
Installing SForge and running one Docker task
The package is distributed as sforge, and pyproject.toml gives the console script entry point as sforge.cli:main, so installation is a normal pip step. Create an environment on a supported interpreter first, because the declared range is >=3.10,<3.14.
pip install sforgeAfter this, the sforge command should be on your PATH. The Docker dependency is not optional for local runs, so the daemon has to be reachable from the shell before you start a task. The repository provides a single-task Docker example under examples/single-task-docker/, which is the smallest thing to try first; the all-tasks-k8s example is the path for running the whole set on a cluster. Task assets come from the Hugging Face dataset ByteDance-Seed/EdgeBench, and huggingface-hub is already a dependency, so the harness can fetch them itself.
The README does not print a full command line for launching a task, so read the example directory's own files for the exact flags and the task YAML before assuming a subcommand name. What you should expect from a successful run is a container per task, progress output from tqdm or rich, and a score recorded against a time budget such as @2h or @12h. Budget accordingly: a single task at the full window is a twelve-hour container.
Where EdgeBench stops being the right tool
The cost model is the main limitation. Twelve hours per task per model is the unit of measurement, and the public leaderboard reports six time budgets per model, so a full comparison is not a laptop exercise. If your question is whether a small fine-tuned model can follow a tool schema, EdgeBench will answer it eventually and expensively. The open release is also a subset: 51 of 134 tasks. The README does not say why the other 83 are withheld, and it does not document a contribution path for adding your own task, so if your domain is not represented in the 51 you may be writing the environment yourself. There is a second, subtler failure mode in the scoring design. Because the metric is a trajectory over time, a task where the agent plateaus early and a task where it never starts look similar at the 2-hour mark but diverge completely later; a team that stops the run early will read the wrong conclusion. The per-task table makes this concrete. One Systems & SE task shows a model at 26.2 at 2 hours and 59.7 at 12 hours, while another model on the same task sits at 27.5 at 2 hours and only reaches 50.2. Short runs invert the ranking. Finally, the benchmark needs a container runtime with enough memory and disk for the task images, and the README does not publish per-task resource requirements.
How this differs from SWE-bench and from interactive agent arenas
SWE-bench and its variants score a patch against hidden tests, once. The unit of work is a pull request, and the metric is resolved or not resolved. EdgeBench keeps the executable environment but changes the metric to a score at a time budget, which is why the leaderboard has six columns per model instead of one. An interactive arena such as an agent-versus-agent game benchmark is closer in spirit, since it also runs for a long time, but it usually scores a single match outcome rather than a fitted curve across many tasks. The practical difference for an adopter is what you can conclude. A SWE-bench number tells you whether a model can produce a passing patch. An EdgeBench number tells you how much a model gains per additional hour, which is the quantity you need when deciding how long to let an agent run before giving up. The tradeoff is that EdgeBench numbers are not comparable to single-shot benchmark numbers, and the README does not offer a way to convert between them.
Licence, maintenance and the cost of upgrading
The repository is licensed Apache-2.0, and pyproject.toml declares the same for the sforge package. Apache-2.0 permits commercial use and modification and includes a patent grant; it also requires that you keep the licence and notice files with redistributed code. That is a permission statement, not legal advice, and if you plan to redistribute a modified harness you should read the LICENSE file in the repository root rather than this summary. On maintenance, the last push to the default branch was on 2026-09-09. The project is not archived. The package version is 1.1.0 and no releases were retrieved, so version pinning has to be done against the package index rather than a GitHub release tag. The upgrade surface is small but not trivial: the Docker SDK is pinned at >=7.0 with no upper bound, and the e2b extra is capped below 3, so a future e2b 3.x would need a harness change. Task assets live on Hugging Face and are versioned there, not in the repository, which means a task definition can change without any change to the Python package.
Editorial conclusion
Adopt EdgeBench if you need a long-horizon signal for an agent and can afford multi-hour container runs; the 51 open tasks and the SForge harness are enough to reproduce a trajectory. Skip it if you want a fast pass/fail check on a small model, since the published curves only separate models after several hours. Before committing, verify three things: that the tasks you care about are inside the 51-task open subset, that your Docker host can hold a container for 12 hours, and that the scoring scripts for those tasks are in the repository rather than only in the tech report.
Frequently asked questions
What is EdgeBench used for?
It evaluates how autonomous AI agents learn from real-world environments across 134 tasks, tracking improvement over 12+ hours per task rather than one-shot performance. The public release covers 51 tasks and the SForge evaluation framework.
How does EdgeBench work?
Agents are placed in executable task environments with multi-level feedback and iterate for 12+ hours, and the benchmark records the full trajectory of improvement. The README states that performance follows a log-sigmoid scaling law as a function of interaction time with R^2 = 0.998 across the 134 tasks.
Is running EdgeBench safe?
The README does not address safety, sandboxing guarantees or isolation properties of the task containers. The repository does include a SECURITY.md file at the root, which is the place to check before running untrusted task code.
How much does it cost to run EdgeBench?
The README gives no pricing. The cost driver is time: each task is scored at budgets up to 12 hours per model, and the harness runs tasks in Docker containers or, with the optional e2b extra, on E2B sandboxes.
Community notes