Harvey LAB: Benchmarking Agents for Legal Work

Why Harvey LAB Targets Legal Agent Evaluation

Harvey LAB addresses a specific measurement problem: determining whether an LLM agent can perform legal work when the task includes instructions, a realistic document collection, and explicit success criteria. It is intended for engineers, researchers, and legal AI teams evaluating agents rather than for lawyers looking for a document-management application. The benchmark contains 2,010 tasks covering 24 or more contracting practice areas, according to the repository README.

The project separates the benchmark data from the software that runs an agent. A task carries its instructions, documents, and rubrics. The execution harness starts the agent against that task, then evaluates the resulting work. This design matters because a model response alone does not show whether it located the right records, followed the assignment, and satisfied the stated requirements.

Harvey LAB is therefore closer to an evaluation system than an autonomous legal agent library. Its users supply the agent and model integration, while the project supplies the legal assignments and a consistent path to grading. That boundary reduces the risk of treating the benchmark as a complete production agent stack, but it also means engineers must connect their own agent implementation before evaluation.

How Tasks, Harnesses, and Rubrics Form the Evaluation Path

Based on the repository layout, `tasks/` contains the benchmark assignments, while `lab_core/` contains the packaged harness and grading core. Each task includes an agent instruction, supporting documents, and a rubric. The README also points to `docs/architecture.md` for the task model, harness, tools, adapters, reports, and sweeps, and to `docs/eval-strategies.md` for all-pass rubric scoring and LLM judge behavior.

The visible data flow is therefore: select a task, inspect its legal materials and instructions, run an agent against the task, grade its output, review a report, and compare results. The tutorial in `docs/tutorial.md` follows that sequence with an M&A data-room assignment. Comparison dashboards are part of the package through `evaluation.compare`, as indicated by the `pyproject.toml` dependency on `matplotlib` and `seaborn`.

The core package also exposes provider SDK dependencies for Anthropic, OpenAI, and Google, plus an optional Mistral adapter under the `mistral` dependency. The README describes model adapters as part of the wider architecture. However, the available documentation excerpt does not enumerate every adapter command or describe a complete custom-agent API, so an integration requiring a different agent framework would need details from the architecture guide and tutorial rather than assumptions based only on the package description.

Installing lab-core and Preparing the Task Checkout

The README directs first-time users to `docs/tutorial.md` for an end-to-end M&A data-room assignment covering setup, task inspection, an agent run, scoring, report review, and comparison dashboards. For library use, the harness and grader are distributed as the `lab-core` Python package with the import name `lab_core`. A wheel is published for each release. The documented installation command for v1.2.0 is:

uv add "lab-core @ https://github.com/harveyai/harvey-labs/releases/download/v1.2.0/lab_core-1.2.0-py3-none-any.whl"

This command adds the release wheel to a `uv` project. The package requires Python `>=3.12,<3.14`, so an engineer should check the interpreter before installing it.

The wheel does not replace the benchmark tasks. The README says to point `LAB_ROOT` at a checkout of `tasks/`. After setting `LAB_ROOT=/path/to/harvey-labs`, the documented form for invoking the CLI outside the repository is:

python -m lab_core.…

The ellipsis is shown in the README as an abbreviated command family, not as a complete executable tutorial. The exact first command should therefore be taken from `docs/tutorial.md` rather than inferred. This is a real documentation limitation for readers who want a single copy-and-paste evaluation command from the root README alone.

Document Conversion for DOCX, XLSX, and PPTX Grading

Document grading is not limited to plain text. `pyproject.toml` states that the grader uses `pandas` with the `openpyxl` engine for `.xlsx` files and `markitdown` with the `python-pptx` converter for `.pptx` files. Those packages are explicit dependencies because they do not arrive transitively. The project warns that scoring converts an `ImportError` into document text, which can affect how a missing converter is interpreted during evaluation.

Grading `.docx` output has an external requirement: the `pandoc` CLI version 3.5 or later must be available on `PATH`. On Linux, the README gives this installer command:

sudo sh "$(uv run python -c 'import lab_core, pathlib; print(pathlib.Path(lab_core.__file__).parent / "sandbox" / "install_pandoc.sh")')"

The command locates the installer bundled inside the installed `lab_core` package and runs it with elevated privileges. The repository states that release CI and the sandbox image use the same bundled installer. On non-Linux systems, `pyproject.toml` mentions `brew install pandoc`, but the available excerpt does not establish support for Windows.

This dependency chain affects reproducibility. A developer who omits `pandoc` can attempt to grade `.docx` output without the documented conversion path. Teams evaluating several agent versions should record the package release, Python version, converter versions, model configuration, and task version because those inputs form part of the evaluation environment.

Rubric Scoring, Reports, and Model Comparison

Harvey LAB uses rubrics as part of each task rather than treating free-form output as the only result. The repository links to a dedicated evaluation methodology document for “all-pass rubric scoring” and LLM judge behavior. That distinction matters: an all-pass policy can require every applicable rubric criterion to succeed before an assignment receives a passing result, while LLM judges introduce model-dependent assessment where the methodology permits them.

The Python dependencies include `anthropic`, `openai`, and `google-genai`, which supports evaluation across several model providers. Mistral support is optional through the `mistral` extra. The reason documented in `pyproject.toml` is dependency compatibility: `mistralai` 2.x pins a newer OpenTelemetry train than some downstream consumers can install. The default Claude and GPT judge pair and the other providers are intended to remain available without that optional dependency.

Reports and dashboards are useful for inspecting failures, but the excerpt does not state that automated scores replace legal review. It also does not quantify judge agreement, variance, or reproducibility across repeated runs. Those omissions are significant for benchmark adoption. A buyer should inspect the task rubric and methodology rather than treating a single aggregate score as proof of production readiness.

Harvey LAB Compared with Task-Specific Evaluation Scripts

A real alternative is a task-specific evaluation script written around one legal workflow. Such a script can remain in an existing Python repository, invoke an agent directly, and apply assertions tailored to one dataset. It avoids installing a general benchmark package, loading unrelated task collections, and paying the setup cost of a multi-provider harness.

Harvey LAB takes a different approach. It packages task inspection, agent execution, rubric grading, reports, provider integrations, and comparison tooling as a coherent benchmark environment. Its 2,010-task collection gives an engineer a way to ask whether an agent generalizes across many assignments and practice areas instead of succeeding on one handcrafted case. The package also separates the wheel from the task checkout, which may help organizations version execution code independently from benchmark content.

The trade-off is breadth and dependency weight. A single-workflow script may be easier to align with an existing grader and may avoid model-provider SDKs that the current test never uses. Harvey LAB is the better fit when the objective is comparative evaluation across standardized legal tasks. It is the wrong choice when the immediate need is only to validate one narrow prompt against one private dataset and no reusable reports or benchmark comparison are needed.

Release Cadence, Python Constraints, and MIT Licensing

The repository is not archived, and its last push was on 2026-10-15. Recent releases include `v1.2.0` for `lab-core` on 2026-10-01 and `v1.1.0` on 2026-09-18. This supports describing recent project activity through the recorded dates, but the README also describes Harvey LAB as ongoing and expects the task set and harness to continue changing.

That expectation creates an upgrade consideration. Users who need stable experimental conditions should record the release and task checkout used for each result. The root README citation still names `v1.0`, while installation documentation points to the `v1.2.0` wheel, so users should follow the release-specific installation link rather than assuming that citation metadata represents the latest package.

The project uses the MIT License, and `pyproject.toml` declares the same license text and license file for the package. That means the distribution includes explicit MIT licensing terms, but organizations still need to review the actual `LICENSE` file for their compliance process. The license does not remove costs associated with Python 3.12 compatibility, external `pandoc` installation, provider usage, model calls, task storage, or benchmark maintenance.

Editorial conclusion

Harvey LAB fits legal AI teams that need a repeatable benchmark for comparing agents and inspecting rubric-based results. It is less suitable for projects seeking a general-purpose legal chatbot framework or teams that cannot supply Python 3.12, model-provider access, and task documents. Verify first that the released `lab-core` package, `LAB_ROOT` task checkout, supported Python version, model credentials, and required document converters align with the deployment environment. For adoption governed by a release, the concrete boundary is that the wheel contains the harness and grader while the task checkout remains a separate input.

Frequently asked questions

Official sources

1. harveyai/harvey-labs on GitHub
2. License: MIT
3. Project website
4. README
5. Releases