COMET: A Neural Metric for Machine Translation Quality, and What It Costs to Run
A Neural Framework for MT Evaluation
At a glance
- What is it?
- COMET is a Python framework that scores machine translation output with neural models instead of n-gram overlap. It is straightforward to install and score with, but the useful models are large, and the reference-free ones carry separate licence terms on Hugging Face.
- Who is it for?
- Adopt COMET if you already produce translations and need a quality signal that correlates better with human judgement than n-gram overlap, and if you can afford the GPU memory that models such as Unbabel/XCOMET-XL require. Do not adopt it as a first-pass filter on a CPU-only CI runner, and do not assume every model is available under the same terms as the Apache-2.0 code.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 147 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem COMET addresses: scoring translations without n-gram counting
BLEU counts overlapping n-grams between a hypothesis and a reference. A translation can be perfectly fluent and preserve meaning while sharing few n-grams with the reference, and BLEU will punish it. COMET takes a different route: it loads a trained neural model and produces a single quality score per segment, with the model consuming the source, the hypothesis, and optionally a reference. The README frames this as evaluation for machine translation, and the repository topics list machine-translation and evaluation-metrics alongside natural-language-processing. The intended user is someone who has translation output in hand and needs a number that tracks human judgement more closely than string overlap does. That includes MT researchers comparing systems, and engineering teams running translation pipelines who want a regression signal when a model or prompt changes. The default model named in the README is Unbabel/wmt22-comet-da, a reference-based model. The same framework also ships reference-free models such as Unbabel/wmt22-cometkiwi-da, which matters when no reference translation exists, for instance when evaluating production output that no human has translated.
How the scoring pipeline works: models, GPUs, and the reference-free split
The architecture is a thin CLI over downloaded neural checkpoints. comet-score reads source, hypothesis and reference files, selects a model, and runs inference, with the --gpus flag controlling device placement; the README notes that 0 tests on CPU. The models are hosted on Hugging Face and pulled by name, which is why the reference-free path is a model swap rather than a separate tool: comet-score -s src.txt -t hyp1.txt --model Unbabel/wmt22-cometkiwi-da drops the reference entirely. Document-level evaluation is handled by concatenating context into the input, separated by a token that belongs to the underlying tokenizer. The README is explicit that </s> is the separator for the xlm-roberta-large tokenizer, and that you enable this with --enable-context against a model such as Unbabel/wmt20-comet-qe-da. That detail matters: the separator is not a COMET constant, it is a property of the checkpoint's tokenizer, so switching models can silently change how context is segmented. XCOMET models extend the output beyond a scalar. According to the README, XCOMET-XL and XCOMET-XXL detect which errors are minor, major or critical following MQM typology, and --to_json writes those identified errors to a file. The README also mentions a method for extracting free-text explanations from XCOMET outputs, pointing to a section of the repository rather than describing the mechanism inline.
Getting it running: install, score, compare, decode
Installation is a single PyPI package. The README gives pip install --upgrade pip followed by pip install unbabel-comet, and states that Python 3.8 or above is required. For local development the documented path is git clone, cd COMET, pip install poetry, poetry install, after which CLI tools can be run directly with PYTHONPATH=. ./comet/cli/score.py. The basic scoring invocation is comet-score -s src.txt -t hyp1.txt -r ref.txt. Multiple systems go in one call by passing several hypothesis files: comet-score -s src.txt -t hyp1.txt hyp2.txt -r ref.txt. For system-level comparison the README recommends comet-compare, which reports statistical significance using a paired T-test with bootstrap resampling, citing Koehn et al 2004. WMT test sets can be pulled through SacreBLEU with comet-score -d wmt22:en-de -t PATH/TO/TRANSLATIONS. Minimum Bayes Risk decoding is a separate entry point, comet-mbr -s [SOURCE].txt -t [MT_SAMPLES].txt --num_sample [X] -o [OUTPUT_FILE].txt, and for long candidate lists the README documents --rerank_top_k to prune candidates with a reference-free metric first, showing an example with --num_sample 1000 --rerank_top_k 100 --gpus 4 --qe_model Unbabel/wmt23-cometkiwi-da-xl. If you only need one aggregate number, --quiet --only_system suppresses per-segment output.
The licensing split between the code and the models
The repository is Apache-2.0. The models are not covered by that grant. The README states that to use some models, such as Unbabel/wmt22-cometkiwi-da, you must acknowledge the licence on Hugging Face Hub and log in to the hub, and repeats the warning for Unbabel/wmt23-cometkiwi-da-xl. That means an automated evaluation job needs a Hugging Face token present in its environment, and a human needs to have accepted terms for that specific model at least once. In an air-gapped or ephemeral CI container, this is a real friction point: the code installs cleanly from PyPI, but the first scoring run may fail on authentication rather than on anything in COMET itself. It also means the practical licence of your evaluation pipeline is the model's licence, not Apache-2.0. This is a description of what the README says, not legal advice; check the model card for the terms that apply to your use.
Where COMET is the wrong tool
The models are large. XCOMET-XL and XCOMET-XXL are named as such, and the MBR example allocates --gpus 4 for a candidate list of 1000. The README offers CPU execution through --gpus 0, but it presents that as a testing option, not a production path. If your constraint is a CPU-only build agent and a scoring step measured in seconds, COMET is a poor fit; a lexical metric will run there and COMET will not. Language coverage is another boundary the README treats as a topic in its own right, with a dedicated section on COMET for African Languages, which implies coverage is uneven across the model family rather than uniform. Reference-free models remove the need for a reference but change what the number means: Unbabel/wmt22-cometkiwi-da scores a hypothesis against the source without any human target, so it cannot detect a case where both the reference and the hypothesis are wrong in the same way, and it is not comparable to reference-based scores from a different model. Finally, the context feature depends on a tokenizer-specific separator, so a context-aware setup is coupled to the checkpoint you picked and does not transfer silently to another model.
SacreBLEU and BLEU: the difference is the reference and the cost
SacreBLEU computes corpus-level BLEU with a fixed, reproducible tokenization, and COMET itself uses it as a data source through the -d wmt22:en-de flag. The two differ in mechanism, not just in accuracy. BLEU is a deterministic count over n-grams that needs a reference and runs on any machine in milliseconds. COMET is a learned function over a neural encoder that needs a checkpoint download, a GPU for practical throughput, and, for the reference-based models, a reference as well. The practical consequence is that BLEU is the right choice when you need a cheap, stable number that never changes between runs, and COMET is the right choice when you need a number that reflects meaning. They are not mutually exclusive in a pipeline: you can compute BLEU for a fast regression gate and run COMET on a sample, or on the final candidate set, which is roughly the pattern the comet-mbr --rerank_top_k flow describes when it prunes a large candidate list before the expensive scoring step.
Maintenance, versioning and upgrade cost
The project is not archived, and the release cadence visible in the material is roughly a few patch releases per year: v2.2.5 in March 2025, v2.2.6 in April 2025, v2.2.7 in September 2025, with the last push to master in April 2026. That is a maintained project, not a frozen one, and the version numbering suggests API stability at the 2.x line. The upgrade cost is mostly in the models rather than the code. Because checkpoints are fetched by name from Hugging Face, pinning a model name is as important as pinning the package version: a model revision can change scores without any change to unbabel-comet. The README's own warnings about licence acknowledgement for specific model names reinforce this, since a new model may require a new acceptance step before it will download. The development path uses Poetry, so a local checkout is reproducible from the lockfile, but the README does not describe a deprecation policy or a compatibility matrix between package versions and model names. Treat the model identifier and the package version as two separate things to pin.
Editorial conclusion
Adopt COMET if you already produce translations and need a quality signal that correlates better with human judgement than n-gram overlap, and if you can afford the GPU memory that models such as Unbabel/XCOMET-XL require. Do not adopt it as a first-pass filter on a CPU-only CI runner, and do not assume every model is available under the same terms as the Apache-2.0 code. Before committing, verify three things: whether the specific model you intend to use requires licence acknowledgement and a Hugging Face login, how many GPUs the model needs at your batch size, and whether your evaluation is segment-level or system-level, since the CLI exposes --only_system for the latter.
Community notes