The COMET example commands append to the same four files

The worked examples append to the same four filenames

The quickstart builds its input by redirecting into files, and the redirect is append:

echo -e "10 到 15 分钟可以送到吗\nPode ser entregue dentro de 10 a 15 minutos?" >> src.txt
echo -e "Can I receive my food in 10 to 15 minutes?\nCan it be delivered in 10 to 15 minutes?" >> hyp1.txt
echo -e "Can it be delivered within 10 to 15 minutes?\nCan you send it for 10 to 15 minutes?" >> hyp2.txt
echo -e "Can it be delivered between 10 to 15 minutes?\nCan it be delivered between 10 to 15 minutes?" >> ref.txt

Nothing truncates those files first, so running the block a second time leaves two copies of every line and the scoring command that follows reads the doubled set. Worse is the later context example, which appends two more lines to `src.txt` and `hyp1.txt` and appends nothing at all to `hyp2.txt` or `ref.txt`. After it, the source and hyp1 files are four lines long while the other two remain at two, so the pairing the tool relies on is broken and the resulting numbers describe something other than the intended sentences. The four files also share a fixed alphabetical naming convention across the whole page, `src`, `hyp1`, `hyp2` and `ref`, with `hyp2` intentionally a worse translation than `hyp1` and `ref` the ideal one, which is what makes the multi-system comparison further down the page meaningful. The practical fix is to create the files yourself and pass them explicitly.

One dependency is pinned to an exact version while the rest float

Almost every requirement in pyproject.toml is a range: `transformers ^4.17`, `pytorch-lightning ^2.0.0`, `torch >=1.6.0`, `numpy ^1.20.0`, `sacrebleu ^2.0.0`, `scipy ^1.5.4`, `protobuf ^4.24.4`. Two are not. The configuration parser `jsonargparse` is fixed at `3.13.1`, an exact pin with no range at all, and the documentation helper `sphinx-markdown-tables` is fixed at `0.0.15`. An exact pin inside a floating dependency graph is where resolution goes wrong, because a transitive dependency that later moves past the fixed version has no way to be satisfied without a lockfile edit. The hub client carries a ceiling of its own, declared as anything from `0.19.3` up to but not including `1.0`, so a future major release of that library will stop resolving by design rather than by accident.

Version 2.2.7 still declares itself a Beta

The package metadata lists the version as `2.2.7`, which matches the newest GitHub release exactly, v2.2.7 published on 2025-09-01. The classifier list in the same file declares `Development Status :: 4 - Beta`, so a two-point-something line still advertises itself as pre-release software, which is a useful signal about how the maintainers view it. Time has moved on since that tag, though. The last commit on the default branch is dated 2026-04-21, roughly seven months after the release, with no version bump to follow it, so the code on the branch and the artifact on the index are not the same thing. That branch is also called `master` rather than `main`, which trips up tooling that assumes the modern default name.

The code and the weights carry different licences

The package itself is Apache-2.0, declared in the metadata and recorded in the licence field. The models do not inherit that automatically. A separate file, `LICENSE.models.md`, sits at the repository root next to the main licence, and the page points at it indirectly through the note that some models require you to acknowledge their licence on the Hugging Face Hub and log in before use. That applies to names such as `Unbabel/wmt22-cometkiwi-da` and to `Unbabel/wmt23-cometkiwi-da-xl`, which the reference-free section singles out with its own acknowledgement note. Practically, a gated model is a different dependency from an open one: a CI job that works on a laptop with cached weights fails on a fresh runner, and the failure looks like a network error rather than a licence one.

Four console commands, one of which trains

The scripts table registers four entry points: `comet-train`, `comet-score`, `comet-compare` and `comet-mbr`, each bound to a function under `comet.cli`. Three of them evaluate or rank, and one trains, which is worth knowing before you add the package to a runtime image and inherit a training stack you did not ask for. Installation from the index is two commands, `pip install --upgrade pip` and then `pip install unbabel-comet`, against a stated floor of Python 3.8 or above. Local development is described separately: clone, install poetry, then `poetry install`, with a `poetry.lock` at the repository root. For running the tools straight from a checkout, the page shows a path-based invocation with `PYTHONPATH` set and the script itself marked executable, which assumes a file mode bit survived the clone.

Context scoring depends on the tokenizer's own separator token

The context example puts a literal `</s>` between the two lines of a record, and the page explains that this is the separator token of whichever tokenizer the underlying model uses, giving `xlm-roberta-large` as the case in point. That means the separator is not a neutral delimiter you can choose: a model built on a different tokenizer expects a different token, and the same input file will segment differently under each model. Scoring with context then requires an explicit flag, `--enable-context`, alongside a reference-free model name. Context is also the reason DocCOMET exists as a document-level extension, which the announcements describe as improving accuracy on discourse phenomena and on referenceless evaluation of chat translation, where sentence pairs carry pronouns and ellipsis across turns.

GPU count is a flag, and CPU is the value zero

Device selection is one option on the scoring command: `--gpus`, with `0` documented as the way to test on CPU. The rest of the surface is ordinary. `--to_json` exports the identified errors, which matters because the XCOMET models return more than a score. `--quiet` with `--only_system` collapses the output to a single system-level number instead of segment-level rows. `-d` pulls a WMT test set through SacreBLEU instead of local files. The MBR ranker is the widest of the four, taking a source file, a sample file, an output path, `--num_sample`, and a `--qe_model` for its own quality estimation; on a large candidate list, `--rerank_top_k` prunes to the top candidates by that reference-free metric first, and the worked invocation combines 1000 samples, a top-k of 100, four GPUs and a specific quality-estimation model.

Only the default model states what its numbers mean

The concise model list gives two entries. The first, `Unbabel/wmt22-comet-da`, is described as a reference-based regression model built on XLM-R, trained on direct assessments from WMT17 to WMT20, producing scores from 0 to 1 where 1 means a perfect translation. That is the kind of sentence that saves you from misreading a dashboard. The second, `Unbabel/wmt22-cometkiwi-da`, is described as a reference-free model that employs a regression approach, and the sentence ends there. So the direction of the scale is stated for one model and not for the other, even though both are regression heads over the same family of encoders. Anyone comparing a reference score against a reference-free score across different models is reading two scales that the page only defines once. Two further gaps sit alongside it: the table of contents promises coverage of the languages a model handles and a dedicated section for African languages, and neither appears in the part of the page shown, while the full catalogue is delegated to a separate `MODELS.md` rather than to the package itself. So the model you end up scoring with is pinned by a hub repository name rather than by a dependency you can lock.

Editorial conclusion

COMET is the metric to reach for when you need a score you can defend in a system comparison rather than a single number, and the compare command with its paired test and bootstrap resampling is the reason. Two practical cautions. Copy the example data into fresh files instead of running the page's commands in place, because they append, and the context example appends to two files out of four, which silently misaligns your source and hypothesis pairs. And separate the code licence from the weights licence before you build on it: the package is Apache-2.0, but individual models carry their own terms and some cannot be downloaded until you accept them on the model hub.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. Unbabel/COMET on GitHub