# human-eval: the code generation harness, and the part you must uncomment yourself

> The human-eval repository is the evaluation harness for the hand-written problem solving dataset from the paper Evaluating Large Language Models Trained on Code. It ships a sample format, a pass@k scorer and a deliberately disabled execution call, so the first thing an adopter does is read a security warning and edit a line of Python.

**openai/human-eval** — Code for the paper "Evaluating Large Language Models Trained on Code"

- Repository: https://github.com/openai/human-eval
- Stars: 3,413 · Forks: 474
- Language: Python
- License: MIT
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/openai-human-eval

## A harness for one specific dataset, not a general code benchmark

The scope here is narrow and deliberate. This is an evaluation harness for the HumanEval problem solving dataset described in the paper Evaluating Large Language Models Trained on Code, and it is the code that came out of that paper, not a framework for building benchmarks. The distinction matters if you arrive looking for a general code quality gate, because what you get is a scorer for a specific set of hand-written problems, a fixed sample format, and a pass@k metric. The repository layout is small and matches that: a data directory holding the problems, an example_problem.jsonl and example_solutions.jsonl to illustrate the format, a human_eval package, a setup.py, a requirements.txt and a LICENSE. The setup.py registers one console script, evaluate_functional_correctness, mapped to the module human_eval.evaluate_functional_correctness, and it reads its dependency list out of requirements.txt at install time rather than hard coding it. The requirements are short and readable: tqdm, fire, numpy. That is the whole dependency surface of a tool whose job is to run a Python program many thousands of times and count how many passed.

## The execution call ships commented out on purpose

This is the single most important thing to know before you run anything, and the README states it plainly. The program exists to run untrusted model-generated code, and the documentation strongly encourages users not to do so outside a strong security sandbox. The mechanism is deliberately blunt: the execution call in execution.py is commented out so that users read the disclaimer before running code in a potentially unsafe manner, and the comment in execution.py carries the information and instructions. The consequence for an adopter is that the tool does not work out of the box. You install it, you point it at a samples file, and it will not run the completions until you go into execution.py and re-enable that call yourself. That design is a reasonable safety default for a tool whose input is arbitrary code produced by a model, and it is also a hard stop for anyone who expected a one command install to produce a score. Anyone automating this in a pipeline is signing up to own the isolation, and the README does not describe a container configuration, a seccomp profile or an unshare invocation, so that part of the design is yours.

## The sample format is a task_id and a completion, nothing else

The interface between a model and the scorer is a JSON Lines file, and the format is deliberately minimal. Each sample is one line in the file, formatted as a JSON object with a task_id corresponding to a HumanEval task ID and a completion holding the completion only without the prompt. The distinction between completion and prompt is worth internalising, because it determines how you generate samples. The prompt for a problem is not included in what you submit, so a harness that concatenates the prompt with the generated text and stores the whole thing will fail to match, and a harness that stores only the raw generation is correct. The example_problem.jsonl and example_solutions.jsonl files under data exist to illustrate the shape and to help with debugging, and the README provides nearly functional example code that shows the whole loop: read_problems() loads the problems, a comprehension builds a list of dicts with a task_id and a completion, and write_jsonl writes samples.jsonl. The example sets num_samples_per_task to 200 and calls a generate_one_completion function that you have to provide yourself, which is the honest shape of the integration point. This is a library plus a scorer with a narrow contract, not a turnkey runner for your model.

## pass@k is estimated from a sample count, and k has a floor

The metric is pass@k, and the way it is computed has a constraint that bites people regularly. The script does not evaluate pass@k when there are fewer samples than k, because there is no unbiased way of estimating pass@k in that case, and the tool will not pretend otherwise. To evaluate with other k values you pass a comma-separated list through a --k flag, and the documentation points at evaluate_functional_correctness --help for the remaining options, with a recommendation to keep the defaults for everything else. The default run in the documentation produces a result object with pass@1, pass@10 and pass@100 keys, which tells you the default k set. The example data gives a calibration point you can use to check your installation: the shipped example samples should yield 0.5 pass@1, and the documented invocation with the --problem_file flag produces pass@1 of 0.4999999999999999 for six samples. If your run of the example data does not land near 0.5, something in your environment is wrong before any real measurement begins. A second output is produced alongside the score, a file whose name ends in the input path plus _results.jsonl, where each row records whether the completion passed and the execution result, which is one of passed, timed out, or failed.

## Installing it and running the shipped example

The installation path is short and pinned to a Python version. The project documentation asks for Python 3.7 or later and starts by creating a conda environment:

```bash
$ conda create -n codex python=3.7
$ conda activate codex
```

Then the repository is cloned and installed in editable mode:

```bash
$ git clone https://github.com/openai/human-eval
$ pip install -e human-eval
```

The editable install is what puts the evaluate_functional_correctness script on your path, since setup.py registers it as a console script entry point. The sample generation side is ordinary Python, and the example code in the documentation is the whole contract:

```python
from human_eval.data import write_jsonl, read_problems

problems = read_problems()

num_samples_per_task = 200
samples = [
    dict(task_id=task_id, completion=generate_one_completion(problems[task_id]["prompt"]))
    for task_id in problems
    for _ in range(num_samples_per_task)
]
write_jsonl("samples.jsonl", samples)
```

Once you have a samples file, the scorer runs as a command, and the documented output shows the three phases it goes through, reading the samples, running the test suites, and writing the results file:

```bash
$ evaluate_functional_correctness samples.jsonl
Reading samples...
32800it [00:01, 23787.50it/s]
Running test suites...
100%|...| 32800/32800 [16:11<00:00, 33.76it/s]
Writing results to samples.jsonl_results.jsonl...
100%|...| 32800/32800 [00:00<00:00, 42876.84it/s]
{'pass@1': ..., 'pass@10': ..., 'pass@100': ...}
```

The per-task count of 200 in the example is what makes pass@100 meaningful, since 200 samples is above the largest default k. To check the install against the shipped data before generating anything of your own, run the quick sanity check with the problem file:

```bash
$ evaluate_functional_correctness data/example_samples.jsonl --problem_file=data/example_problem.jsonl
```

That run reads six samples, runs the example suites, writes data/example_samples.jsonl_results.jsonl, and reports a pass@1 of 0.4999999999999999, which is the number to expect.

## A memory error can silently turn a correct program into a failure

The known issues section is short and the one issue it names is not cosmetic. Evaluation uses very little memory in principle, but when the system runs out of RAM you may see a malloc error, and the documentation is explicit that this may cause some correct programs to fail, with a recommendation to free some memory and try again. Read that carefully, because it describes a scoring error rather than a crash. A completion that should have passed is reported as failed, and the pass@k number comes out lower than the truth. This is the failure mode that most often gets misread as a model regression, and the mitigation is not in the tool. The harness spawns a large number of executions, and the documented run above shows 32800 samples taking over sixteen minutes of test suite time, so the memory pressure scales with how many samples you ask for and how parallel the suite runs. The other documented error string is the bare allocator message itself:

```bash
malloc: can't allocate region
```

If you see it, the honest response is to free memory and rerun, not to report a lower score as a finding.

## The repository itself says not to rely on it

The project documentation opens with a deprecation notice that anyone evaluating this project should read before anything else. It states that the repository is no longer maintained and is provided for historical reference, without ongoing support or updates, and that it should be used at your own risk. The repository metadata does not show the project as archived, and the last push to the default branch master was on 2026-10-16, so code changes are still landing in the tree even though the documentation disclaims support. The project publishes no GitHub releases, so there is no versioned artifact to pin: you install from a branch tip, and the version in setup.py is a flat 1.0 that has not tracked anything. That combination, an actively pushed branch with no releases and an explicit no-support statement, is the situation to plan around. The licence is MIT, which is permissive enough that the licensing question is not the obstacle; copying this harness into a closed internal evaluation tool is straightforward, and the licence's real obligation is keeping the copyright notice, and it does not extend to the model outputs you score. The real obstacle is that a deprecated, unsupported scorer will not track changes in what a model emits, and the 2021 dataset will not grow to cover the failure modes that matter to you now.

## Conclusion

Adopt human-eval if you need to reproduce or extend the pass@k numbers from the 2021 paper, because the sample format, the scorer and the reference data are all here and the arithmetic is the original one. Do not adopt it as a general code quality gate for a modern model, since the dataset is small and fixed and the repository's own README declares it deprecated and unmaintained. Verify first that you have a real sandbox for the untrusted completions before you re-enable the execution call in execution.py, that your samples carry a task_id matching a problem in the data directory, and that the pass@k values you intend to publish use more samples than k, because the scorer refuses to estimate otherwise.

## FAQ

### Why does human-eval not run my completions when I install it?

The execution call in execution.py is deliberately commented out so that users read the security disclaimer first, because the program exists to run untrusted model-generated code. You have to re-enable that call yourself, and the README strongly encourages you to do it only inside a strong security sandbox.

### What format does human-eval expect samples in?

JSON Lines, one sample per line, with a task_id field holding the corresponding HumanEval task ID and a completion field holding the completion only without the prompt. The repository ships example_problem.jsonl and example_solutions.jsonl under the data directory to illustrate the format.

### How do I check that my human-eval installation works?

Run the shipped example with evaluate_functional_correctness data/example_samples.jsonl --problem_file=data/example_problem.jsonl. The documentation states that the example samples should yield 0.5 pass@1, and the shown run reports a pass@1 of 0.4999999999999999 over six samples.

### What does the pass@k value in the human-eval results mean?

The script reports pass@1, pass@10 and pass@100 by default, computed from a pool of samples per task. It does not evaluate pass@k when there are fewer samples than k, since there is no unbiased way to estimate it in that case, and you can pass other k values with the --k flag.

## Sources

- [Issues](https://github.com/openai/human-eval/issues)
- [License: MIT](https://github.com/openai/human-eval/blob/master/LICENSE)
- [openai/human-eval on GitHub](https://github.com/openai/human-eval)
- [README](https://github.com/openai/human-eval/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/openai-human-eval