sacreBLEU: one BLEU implementation so scores can be compared across papers

The problem is comparability, not computation

The motivation section is the sharpest part of the README, and it explains why a second BLEU implementation was worth writing.

Comparing BLEU scores is harder than it should be. Every decoder has its own implementation, often borrowed from Moses, but maybe with subtle changes. Moses itself has a number of implementations as standalone scripts, with little indication of how they differ, and the README notes they mostly don't differ, but multi-bleu.pl expects tokenized input. Different flags passed to each of these scripts can produce wide swings in the final score. All of these may handle tokenization in different ways. On top of this, downloading and managing test sets is a moderate annoyance.

So the failure mode is not that BLEU is hard to compute. It is that a number without provenance is meaningless. A paper reporting BLEU 27.3 has told you almost nothing unless you know the tokenizer, the casing policy, the smoothing method and the reference set.

The fix is a signature. The defaults are set the way BLEU should be computed, and the script outputs a short version string that allows others to know exactly what you did. The JSON output makes that concrete.

{
 "name": "BLEU",
 "score": 20.8,
 "signature": "nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0",
 "verbose_score": "54.4/26.6/14.9/8.7 (BP = 1.000 ratio = 1.026 hyp_len = 62880 ref_len = 61287)",
 "nrefs": "1",
 "case": "mixed",
 "eff": "no",
 "tok": "13a",
 "smooth": "exp",
 "version": "2.0.0"
}

Six fields, each one a decision that changes the number, and a version field so the implementation itself is pinned. That is the whole contribution.

What the signature fields actually control

Reading the signature is worth doing, because the fields map onto real choices a scorer makes.

nrefs is the number of references. A multi-reference setup changes the brevity penalty and the clipping behaviour, so the count belongs in the signature.

case is mixed, which is the WMT convention of not lowercasing everything and not preserving case either. eff refers to the effective reference sentence length, the length of the shortest reference when references differ in length, which again changes the penalty.

tok is 13a, the WMT standard tokenization, and the README says it properly computes scores on detokenized outputs using WMT standard tokenization, and produces the same values as the official script, mteval-v13a.pl, used by WMT. That last claim is the load-bearing one, because matching the official script is what makes a sacreBLEU number directly comparable to a WMT leaderboard number.

smooth is exp, the exponential smoothing method, and the alternative is the floor method. Smoothing matters most for short outputs, where a single n-gram match can otherwise zero out a whole score.

version is the sacreBLEU version, so a number computed by 2.0.0 is distinguishable from one computed by an earlier release.

The README also notes the practical advantage over the scripts it replaces: it outputs the BLEU score without the comma, so you do not have to remove it with sed, with a nod to multi-bleu.perl. That is a small thing, and it is the kind of small thing that decides whether people actually use a tool.

Test sets download themselves when you name them

The dataset handling is the feature that saves the most time, and the mechanism is lazy. Downloading is triggered when you request a test set. If the dataset is not available, it is downloaded and unpacked.

So the reference side of a scoring run does not need to be prepared. You ask for wmt14 and it materialises.

$ sacrebleu -t wmt17 -l en-de --echo src > wmt17.en-de.en
$ cat wmt17.en-de.en | translate.sh | sacrebleu -t wmt17 -l en-de

That first line is a neat trick. The --echo src flag writes the source side of the test set to standard output, so you can pipe it through your own translation system and pipe the result straight back into the scorer.

Some test sets also have the outputs of systems that were submitted to the task, for example the wmt/systems test set.

$ sacrebleu -t wmt21/systems -l zh-en --echo NiuTrans

Which provides a convenient way to score your own output against a known system.

$ sacrebleu -t wmt21/system -l zh-en --echo NiuTrans | sacrebleu -t wmt21/systems -l zh-en

You can see a list of the available outputs by passing an invalid value to --echo, which is a cheap discovery mechanism that also tells you the exact system names to use.

For finding what is available, sacrebleu --list gives the full list, and sacrebleu --list -l en-fr narrows it to one language pair. DATASETS.md in the repository holds the up-to-date list of supported datasets.

More metrics than BLEU, and significance testing

The features list goes past BLEU, and the additions are the ones that matter for a paper.

It supports chrF, chrF++ and Translation error rate (TER) metrics. chrF was designed for character-level scoring, which is the standard in some WMT tasks and the sensible metric for languages where word segmentation is ambiguous.

It supports different tokenizers for BLEU including support for Japanese and Chinese, which is a practical necessity rather than a nicety, since word segmentation for those languages is not whitespace-delimited.

It performs paired bootstrap resampling and paired approximate randomization tests for statistical significance reporting. This is the most underrated feature in the list. A single BLEU number says nothing about whether a new system is better than the baseline, and the paired tests are how you establish that the difference is not noise. The pairing matters because you are comparing the same test sentences, which reduces variance considerably.

The README describes the project as providing hassle-free computation of shareable, comparable, and reproducible BLEU scores, inspired by Rico Sennrich's multi-bleu-detok.perl, and producing the official WMT scores while working with plain text.

The package description in the pyproject.toml is hassle-free computation of shareable, comparable, and reproducible BLEU, chrF, and TER scores, and the classifiers mark it as Development Status 5, Production/Stable, for Developers and Science/Research.

The v2.0.0 output change that broke pipelines

There is one breaking change documented, and it is worth reading before you adopt the tool, because it silently changes what a scoring script returns.

As of v2.0.0, the default output format is changed to json for less painful parsing experience. Software that parses the output of sacreBLEU should be modified to either parse the JSON using for example the jq utility, or pass -f text to sacreBLEU to preserve the old textual output. The latter change can also be made persistently by exporting SACREBLEU_FORMAT=text in relevant shell configuration files.

The example in the README shows the first route.

$ sacrebleu -i output.detok.txt -t wmt17 -l en-de | jq -r .score
20.8

And the old format is available on demand.

$ sacrebleu -i output.detok.txt -t wmt17 -l en-de -f text
BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0 = 20.8 54.4/26.6/14.9/8.7 (BP = 1.000 ratio = 1.026 hyp_len = 62880 ref_len = 61287)

The change is defensible, since parsing a text format with sed is exactly the kind of thing that breaks, and the JSON output is a superset of the fields in the text line. The risk is in the upgrade path rather than the design, and the mitigation is one environment variable.

For anyone with a research pipeline that has been collecting BLEU numbers for years, the safe move is to export SACREBLEU_FORMAT=text in the shell configuration and treat the change as a separate, deliberate migration later.

Install extras for Japanese and Korean, and the toolchain behind it

The README documents separate extras for Japanese and Korean tokenizers.

Install the official Python module from PyPI, Python 3.9 or newer only.

pip install sacrebleu

For Japanese tokenizer support through mecab-python3 you need the ja extra, which performs a full installation with dependencies.

pip install "sacrebleu[ja]"

For Korean tokenizer support through pymecab-ko, the ko extra does the same thing.

pip install "sacrebleu[ko]"

The pyproject.toml pins mecab-python3 at 1.0.9 or newer and below 2.0.0, with ipadic at 1.0 or newer and below 2.0, and mecab-ko at 1.0.2 or newer.

The runtime dependencies are otherwise small: portalocker, regex, tabulate at 0.8.9 or newer, numpy at 1.17 or newer, colorama and lxml.

The build uses setuptools at 77 or newer with setuptools_scm at 8 or newer, and the version is dynamic, taken from the git tag. The Makefile shows the check sequence: mypy over sacrebleu, scripts and test, then pytest, then a bash test.sh. The repository also carries a mypy.ini, a pytest.ini and a tox.ini, so the project is typed, marked as Typing:: Typed, and tested in three ways.

Editorial conclusion

sacreBLEU is the right tool for anyone reporting a BLEU number, because a score without its signature is not comparable to anything, and this tool makes the missing information impossible to omit while also handling test set download and WMT tokenization for you. It is the wrong tool if your task is not machine translation, since the tokenizers and the reference corpora are built for that, and it is the wrong choice if you need to compare against a pre-2020 number produced by multi-bleu.perl without a signature, because the values may differ. Before switching your pipeline, check whether anything downstream parses the output, since version 2.0.0 changed the default format to JSON and the fix is to export SACREBLEU_FORMAT=text or pass -f text.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. mjpost/sacrebleu on GitHub
4. README
5. Releases