sumeval: a pure Python ROUGE and BLEU scorer for English, Japanese and Chinese summaries

The gap sumeval fills: ROUGE without a Perl subprocess

ROUGE has a reference implementation, ROUGE-1.5.5.pl, and a lot of summarization papers report numbers produced by it. Calling that script from a Python training loop means managing a Perl dependency, writing files to disk for each pair of summary and reference, and parsing text output back into floats. sumeval moves the scoring into Python. The README states the ROUGE-X scores are tested against the original Perl script, and that BLEU is calculated by SacreBLEU, which the README says produces the same values as the official mteval-v13a.pl script used by WMT. That combination is the pitch: you get the numbers the reference scripts would give, in process, in Python. The audience is anyone building or evaluating a summarization system in Python who wants ROUGE and BLEU in the same language as the model code, and who works with English, Japanese or Chinese text.

What the API actually does with a summary and its references

The entry points are two calculator classes. RougeCalculator is constructed with stopwords and lang, for example RougeCalculator(stopwords=True, lang="en"), and exposes rouge_n, rouge_l and rouge_be. rouge_n takes a summary, a references argument and n. The README's example passes a single string as references for ROUGE-1 and a list of two strings for ROUGE-2 and ROUGE-L, so the references argument accepts either shape. BLEUCalculator is simpler: construct it, optionally with lang, and call bleu with a candidate and a reference string. The CLI is a thin layer over the same code. Running sumeval r-nlb with a summary and a reference prints a JSON object with three keys: options, averages and scores. The options block echoes back stopwords, stemming, word_limit, length_limit and alpha, which tells you the scoring is parameterised and that those parameters are visible in the output rather than hidden defaults. The averages block holds one number per metric across all references, and scores is a list with one entry per reference. In the README's example, one reference produces one entry in scores and identical values in averages.

Multi-language support is a tokenizer problem, and sumeval treats it as one

ROUGE and BLEU both depend on how you split text into units, and for Japanese and Chinese that decision is not whitespace. sumeval handles this through language-specific tokenization and dependency parsing, located according to the README in sumeval/metrics/lang. Adding a language means writing a class that inherits from BaseLang. That is a clean extension point, and the README points contributors at it directly. The cost is that each language pulls its own tokenizer. Japanese requires janome or MeCab, and ROUGE-BE for Japanese additionally requires GiNZA. Chinese requires jieba, and ROUGE-BE for Chinese additionally requires pyhanlp. English ROUGE-BE requires spaCy. So the dependency set is not fixed at install time; it is a function of which languages and which metrics you call. If you only need ROUGE-N and ROUGE-L on English, the README's dependency list suggests you need none of the tokenizer packages, because those are named only in connection with BLEU, ROUGE-BE, Japanese and Chinese.

Installing it and reading the options block

Installation is a single command: pip install sumeval. The CLI example in the README is sumeval r-nlb "I'm living New York its my home town so awesome" "My home town is awesome". The subcommand name r-nlb suggests a ROUGE-N, ROUGE-L, ROUGE-BE bundle, and the printed JSON confirms it: averages contains ROUGE-1, ROUGE-2, ROUGE-L and ROUGE-BE. Note that ROUGE-BE is 0 in that example while the other three are non-zero. The README does not explain why, and the comment in the Python example says spaCy is needed to calculate ROUGE-BE, so the most likely reading is that the CLI run did not have the ROUGE-BE dependencies available and reported 0 rather than failing. That is worth knowing before you treat a 0 as a real score. The README also says file input is supported and that sumeval -h gives more detail, so the CLI is not limited to inline strings. The options block is the part to read on every run: stopwords, stemming, word_limit, length_limit and alpha are all echoed, and those are the settings that determine whether your numbers match someone else's.

Where sumeval is the wrong tool

The metric set is narrow. sumeval covers ROUGE-N, ROUGE-L, ROUGE-BE and BLEU. It does not claim to implement BERTScore, MoverScore, BARTScore or any embedding-based metric, and nothing in the README suggests an extension path for new metrics the way BaseLang provides one for new languages. If your evaluation plan is built on semantic similarity rather than n-gram overlap, sumeval is not the library. The second limitation is the dependency surface. ROUGE-BE is gated behind spaCy for English and behind GiNZA or pyhanlp for Japanese and Chinese, and those are large packages with their own model downloads. In a constrained build environment, that can turn a one-line install into a project. The third is the ROUGE-BE zero in the README's own CLI output. Whatever the cause, the behaviour is that a missing dependency can surface as a score of 0 rather than an error, and a silent zero in an evaluation harness is worse than a crash. The fourth is that the repository shows no retrieved releases, so there is no changelog to consult for behaviour changes between versions; pinning a version is the only way to know what you are running against.

How it differs from pythonrouge and rougescore

The README names two other packages, and it names them in the testing section, which is the honest place to look for the comparison. pythonrouge calls the original Perl script. rougescore is described as a simple Python implementation of the ROUGE score. So the field has three positions. pythonrouge gives you the reference implementation's behaviour by running the reference implementation, which means a Perl runtime and file-based plumbing. rougescore gives you a pure Python implementation that is simple, with the implication that it is not validated against the Perl script. sumeval sits between them: pure Python, but with the ROUGE-X scores tested against ROUGE-1.5.5.pl and BLEU delegated to SacreBLEU. The README uses both pythonrouge and rougescore as test dependencies, which means the project's own test suite is comparing itself against a Perl-backed implementation and a plain Python one. That is a stronger claim than either alternative makes on its own, and it is the specific reason to pick sumeval over rougescore if reference parity matters to you.

Maintenance, versions and the Apache-2.0 terms

The repository is not archived and the last push date is 2026-04-13. No releases were retrieved, so there is no published version history to read. In practice that means you install from PyPI, which the README instructs, and you pin whatever version you get, because there is no changelog in the supplied material to tell you what changed between installs. The licence is Apache-2.0. That is a permissive licence that includes an explicit patent grant, which matters if you are embedding the library in a product rather than using it for internal evaluation. Apache-2.0 also carries notice and attribution obligations, so if you redistribute sumeval or a derivative, you need to keep the licence and notice files. The language classes you write by inheriting BaseLang are your own code, but if you contribute them upstream, the README's contribution section points at the repository, and contributions would fall under the project's licence. This is a description of the licence text, not legal advice; check the terms against your own distribution model.

Editorial conclusion

Adopt sumeval if you are scoring English, Japanese or Chinese summaries in Python and want ROUGE values that the repository says were tested against ROUGE-1.5.5.pl, without shelling out to Perl. Do not adopt it if you need a metric it does not implement, or if you cannot install spaCy, janome or jieba in your environment, since ROUGE-BE and the non-English paths depend on them. Before committing, run the CLI on one summary and one reference from your own corpus and compare the JSON against your existing scorer, because the tokenizer and stopword settings in the options block are the part most likely to differ from whatever you use now.

Official sources

1. chakki-works/sumeval on GitHub
2. Issues
3. License: Apache-2.0
4. README