RewardBench: a harness for scoring reward models, not a leaderboard you can trust blindly

A benchmark that ships the inference code, not just a score

RewardBench is a benchmark for evaluating reward models, and the project description calls it the first evaluation tool for reward models. The homepage is a Hugging Face Space hosting the leaderboard, and the repository publishes a first version paper and a second version paper with a new evaluation dataset, results, and a collection of trained models.

What the repository actually contains is a harness rather than a dataset alone. The README lists three things: common inference code for a variety of reward models, naming Starling, PairRM, OpenAssistant and DPO among them; common dataset formatting and tests for fair reward model inference; and analysis and visualization tools. The top level directory listing matches that, with a rewardbench package directory, a scripts directory, an analysis directory, a tests directory, a Dockerfile and a Dockerfile.vllm.

The audience is therefore a model developer rather than a casual reader of a leaderboard. The person installing this is training a reward model, or a DPO variant, and wants to know whether it is better than the one they had. The same person also wants to know whether the leaderboard position that motivates them is a property of their model or an artefact of how the evaluation was run.

The licence is Apache-2.0, the package is published on PyPI as rewardbench, and the last push was on 2026-10-14.

Three entry points, because reward models come in three shapes

The README names the three primary scripts, and the split between them is the clearest statement of what a reward model can be.

The first is scripts/run_rm.py, which runs evaluations for reward models. The second is scripts/run_dpo.py, for direct preference optimization models and for other models using implicit rewards, such as KTO. The third is scripts/run_v2.py, which runs evaluations for RewardBench 2 and applies special data handling for best-of-4 and Ties data.

That third distinction is the subtle one. A sequence classifier reward model gives you a scalar per response, so comparing two responses is a matter of which scalar is larger. A DPO model does not output a reward directly; it was trained on a preference objective, so an implicit reward has to be recovered, and for a DPO model that recovery needs the reference model, which is why the CLI documentation tells you to pass a ref_model and the script then routes to the DPO path automatically. A generative judge, where a language model is asked to compare or to rate, is a third shape again and needs its own prompt handling.

So a single score across all three families is not a single operation. The code that makes the comparison fair is the shared formatting and the shared inference, and the README is explicit that this is what the tests are for.

Installing with uv, and the FastChat dependency you have to add yourself

The install instructions are short, and uv is the recommended tool.

uv pip install rewardbench

# For generative models (LLM-as-judge, vLLM, API providers)
uv pip install rewardbench[generative]

The pip form is the same with pip in place of uv pip. For development, the repository is cloned and synced.

git clone https://github.com/allenai/reward-bench.git
cd reward-bench
uv sync                      # base install
uv sync --extra generative   # with generative support

The part of the documentation that deserves attention is the caveat about the legacy version 1 scripts. The rewardbench-gen command, scripts/run_rm.py, run_dpo.py and run_bon.py, along with the --chat_template flag, need FastChat, and FastChat is described as unmaintained and therefore not a declared dependency. The instruction is to install it by hand with uv pip install fschat, and the note adds that a later plain uv sync removes it, so uv sync --inexact is the way to keep it.

That is an honest disclosure of a rough edge rather than a hidden one, and it is also a signal about the state of the project. The v2 path does not need FastChat; the v1 path does. A team starting fresh should use the v2 scripts and stay away from the legacy surface entirely.

The pyproject file also shows a set of optional dependency groups. The api group installs openai, anthropic, google-genai, google-generativeai and together, and it is labelled as API-based LLM judges with a fast install and no GPU required.

Running the v2 evaluation and the older single command

To run RewardBench 2, the README gives one command with the model name substituted in, and points at the eval config file for model specific parameters.

python scripts/run_v2.py --model={yourmodel}

Generative models need a different script, and there is a choice of prompt type. A rankings-based prompt compares four responses in one go and is the default, while a ratings-based prompt scores each response separately and then recombines them, and is selected with the --score_w_ratings flag. The reason that choice exists is stated in the README: the Ties subset, new in RewardBench 2, has up to more than twenty completions to score per prompt, so the code enforces that it runs in the ratings setting.

python scripts/run_generative_v2.py --model={yourmodel}

For the older evaluation set, the single command form is used, and the DPO case is handled by passing the reference model, after which the script routes itself to the right path.

rewardbench --model={yourmodel} --dataset={yourdataset} --batch_size=8

The README notes that chat templates are used automatically from the tokenizers, and that fastchat conversation templates can be used instead, with a note in the logging section that chat templates only need to be specified for older models.

Two worked examples follow in the documentation, one for a normal operation on a sequence classifier reward model with a dataset and a split and a raw chat template, and one for a DPO model reading a local JSONL file with the --load_json flag and a reference model.

rewardbench --model=OpenAssistant/reward-model-deberta-v3-large-v2 --dataset=allenai/ultrafeedback_binarized_cleaned --split=test_gen --chat_template=raw

The second example passes a path to a JSONL file on a network filesystem for the DPO model, which is a reminder that the dataset argument takes either a hub identifier or a local file.

What the accuracy number does and does not tell you

The Quick Usage section contains the caveat that should shape how anyone reads a RewardBench result.

The tool will detect if an instruction dataset is passed, by checking that the dataset does not have chosen and rejected fields and does have messages. For such a dataset, only model outputs are logged, and not accuracy. In other words, a preference dataset with a chosen and a rejected response can be scored, because there is a right answer, while a plain instruction dataset has no ground truth to be right about, so the tool records what the model produced and leaves it at that.

That is the difference between a benchmark and a logger, and it applies to any custom dataset you bring. If you want an accuracy figure, the dataset has to carry a reference response, and the tool has to be able to match the model's choice against it.

There is a second place where the wording of the README matters. The result of a preference evaluation is a comparison between two model outputs, and the accuracy reflects how often the reward model picked the one the dataset marked as preferred. A reward model can therefore score well by learning the preferences of the dataset it was evaluated on, which is the reason a benchmark needs held-out data and a reason the same number is not comparable across datasets.

The analysis and visualization tools in the repository exist to break those results down rather than to produce a single number, and the evaluation config file in scripts/configs/eval_configs.yaml is where the per-model settings live.

Pushing results to the Hub, and the metadata it can attach

The logging section describes two Hub integrations, and they exist for a specific workflow rather than as a convenience.

The --push_results_to_hub flag uploads a dataset of scores and correctness. The --upload_model_metadata_to_hf flag adds results directly to the model. The README states the purpose plainly: these can be tied in metadata to reward models you own or have uploaded as separate datasets to Hugging Face, such as for rejection sampling.

That is a meaningful workflow. A reward model is often used to sample candidate responses and keep the best ones, which is rejection sampling, and the reward scores produced by this tool are exactly the signal a sampling loop needs. Uploading them alongside the model means the scores travel with the checkpoint rather than living in a spreadsheet.

A worked command shows both flags at once, along with a large batch size and a tokenizer specified separately from the model.

rewardbench --model vwxyzjn/reward_modeling__EleutherAI_pythia-14m --batch_size 128 --tokenizer=EleutherAI/pythia-14m --push_results_to_hub --upload_model_metadata_to_hf --chat_template raw

The README links an example model with accuracy metadata and two example datasets showing outputs from a preference dataset and from an instruction dataset.

One limitation is stated at the end of that section, and it narrows the feature considerably: pushing results for preference datasets currently only works with DPO models. For a sequence classifier reward model, the metadata upload path is not described as working in the same way, so the workflow as written is aimed at DPO models.

A pinned transformers, a generous Python range, and a lot of optional pieces

The pyproject file describes the packaging, and two things in it will affect an install.

The first is the transformers pin. The dependency list ends with transformers==5.17.0, an exact version rather than a range, alongside torch, torchvision and torchaudio without pins, plus accelerate, bitsandbytes, datasets, huggingface_hub, hf_transfer, pandas, sentencepiece, tabulate, tokenizers, tiktoken and wandb. The exact pin on one library in the middle of a stack of unpinned ones is a maintenance decision with consequences: if your environment needs a different transformers version, the resolver will fight you, and the pin tells you the evaluation was validated against that release.

The second is the Python range, which is requires-python of 3.10 up to but not including 3.14, with classifiers for 3.10, 3.11 and 3.12. The build backend is hatchling and the version in the project metadata is 0.1.5, one patch above the newest tag in the release list, which is v0.1.4 for the RewardBench 2 release from 2025-06-03, with v0.1.3 from 2024-10-04 before it.

The project also ships a Makefile with two targets. The style target runs black with a line length of 119 and a target version of py310 over the rewardbench, scripts, analysis and tests directories, and then runs isort with the black profile. The quality target runs flake8 with the same maximum line length over the same directories. The Makefile sets PYTHONPATH to src so that the local checkout is tested rather than an installed copy, and it comments that the quotes should not be used, which is a small sign of a working development setup rather than a published one.

The container images are the other deployment path. The Dockerfile is built on a public NVIDIA CUDA image for reproducibility rather than an internal one, pinned to linux/amd64, with CUDA 12.1.0 and cuDNN 8 on Ubuntu 20.04, and it installs Miniconda with a verified SHA-256 checksum, then uv. A separate Dockerfile.vllm exists for the vLLM path, and the main Dockerfile sets HF_HUB_ENABLE_HF_TRANSFER=1 for faster model downloads.

Editorial conclusion

RewardBench fits a team that trains reward models or DPO variants and needs one evaluation path applied consistently across candidates, and it does not fit a team looking for a single score to settle an argument, since the ranking it produces depends on the chat template, the judge prompt and the model class you point it at. Before adopting it, reproduce one of the published example runs on a model you already know, because a wrong template or a mismatched reference model changes the result, and read the Ties handling before you compare a V1 number with a V2 number, since the V2 script uses special data handling for best-of-4 and Ties data.

Frequently asked questions

Official sources

1. allenai/reward-bench on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases