# CL-bench asks whether a model can learn from the prompt it is given

> CL-bench is a Tencent Hunyuan benchmark for context learning, built on 1,899 expert annotated tasks with roughly 63 rubrics each, plus a second set, CL-bench Life, that uses messy real-life contexts. Scoring is binary and rubric driven, with an LLM judge deciding whether every rubric in a task passed.

**Tencent-Hunyuan/CL-bench** — CL-bench: A Benchmark for Context Learning

- Repository: https://github.com/Tencent-Hunyuan/CL-bench
- Stars: 589 · Forks: 27
- Language: Python
- License: NOASSERTION
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/tencent-hunyuan-cl-bench

## The mismatch the benchmark is built around

CL-bench evaluates whether language models can learn new knowledge from context at inference time, and the tasks require models to learn from domain-specific knowledge, rule systems, complex procedures and empirical laws.

The framing in the README is a mismatch between how language models are commonly optimized in practice and the capabilities required by real-world tasks. That is the argument for the whole benchmark: models are optimised to know things, while a lot of real work is about working out the rules you have just been handed.

The structure of an instance is the part to understand. Each instance comprises a system prompt, a task, the context containing the new knowledge necessary for solving the task, and rubrics to assess the solution. And the annotation is done by experienced domain experts, which is the difference between this and a synthetic benchmark.

The scale is stated precisely: 1,899 tasks across 4 categories and 18 sub-categories, with an average of 63.2 rubrics per context and an average of 20 hours of expert effort per context. That last number is the one that explains the design, and it is the reason the rubrics are as fine grained as they are. A benchmark with cheap annotations would not be able to say that a response fails because it missed one specific condition out of sixty.

The project is Tencent Hunyuan's, published under a custom licence that GitHub could not classify, with the licence file in the repository root. The primary language is Python, the default branch is main, the last push was on 2026-10-17, and there are no GitHub releases. The website is clbench.com, with a leaderboard, and the datasets are on Hugging Face under the tencent organisation.

## Four categories that stress different kinds of learning

The context categories are the substance of the benchmark, and each one is a different failure mode for a model.

Domain Knowledge Reasoning is specialised domain knowledge requiring understanding and application. Rule System Application is formal rule systems that must be learned and correctly applied. Procedural Task Execution is complex multi-step procedures to follow. And Empirical Discovery & Simulation is patterns and laws derived from empirical data, which is the hardest of the four, because the model has to infer a rule from observations rather than being handed it.

That last category is what separates this benchmark from a knowledge quiz. With a rule system, the context states the rules and the model applies them. With empirical discovery, the model has to work out what the pattern is before it can act on it, and a model that guesses the wrong law will produce a confident answer that satisfies none of the rubrics.

A model can also fail on the mechanics rather than the reasoning. A context with sixty rubrics is unforgiving about details like a unit, a threshold or a step order, so a response that has the right idea and one wrong number scores as a failure. That is intentional if you are measuring whether a model can be relied on to apply what it was told, and it is a reason to look at the per-rubric output rather than only the headline score.

The second member of the family, CL-bench Life, extends the same idea to everyday data, with 405 context-task pairs, 5,348 rubrics averaging 10.7 per task, across 3 categories and 9 sub-categories.

## CL-bench Life and the case for messy contexts

CL-bench Life extends context learning evaluation to real-life scenarios, and the README's description of the difference is precise: contexts are messy, fragmented, and grounded in everyday experience, the kind of data people actually deal with daily.

The three categories explain what that means in practice. Communication & Social Interactions covers group chats, meeting transcripts, private conversations and community threads. Fragmented Information & Revisions covers personal notes, news feeds, document edit histories and version logs. And Behavioral Records & Activity Trails covers game logs, browsing histories, transactions and fitness or health tracking.

Notice what is absent from all three: tidy, well structured documents written for the model to read. A group chat is fragmented across speakers and messages, a version log contains superseded information, and a fitness tracker is a stream of numbers with no labels explaining which trend matters. The challenge is not only learning the content, it is working out which part of the content is current and which part is noise.

The rubric counts also differ sharply between the two benchmarks. CL-bench averages 63.2 rubrics per context with 20 hours of expert effort each, while CL-bench Life averages 10.7 rubrics per task. That is consistent with the two designs. Professional and domain-specific tasks have many checkable conditions, while a real-life task usually has a few things that must be right.

The release history is recent. CL-bench was released in 2026.02 and CL-bench Life in 2026.04, each with a paper on arXiv, a dataset on Hugging Face and a blog post, and both feed the same leaderboard at clbench.com.

## Running inference and scoring with an LLM judge

The pipeline is deliberately small, and the README says so: both benchmarks share the same evaluation pipeline, you just point it at different input files.

The dependencies are two packages:

```bash
pip install openai tqdm
```

Inference takes a model name and an input file, and the workers flag controls concurrency:

```bash
export OPENAI_API_KEY="your_api_key"

# CL-bench
python infer.py --model <model_name> --input CL-bench.jsonl --workers 20

# CL-bench Life
python infer.py --model <model_name> --input CL-bench-Life.jsonl --workers 20
```

For anything that is not an OpenAI model, you give a base URL and an API key instead, which is how you point the harness at a local or third party endpoint.

Evaluation is where the design choices bite. CL-bench uses GPT-5.1 with low reasoning effort as the default judge, and CL-bench Life uses GPT-5.1 with high reasoning effort:

```bash
# CL-bench (low reasoning effort)
python eval.py --input outputs/<model_output>.jsonl --judge-model gpt-5.1 --reasoning-effort low

# CL-bench Life (high reasoning effort)
python eval.py --input outputs/<model_output>.jsonl --judge-model gpt-5.1 --reasoning-effort high
```

The two benchmarks get different judge settings because the tasks differ in difficulty, and a single judge configuration would be a compromise for one of them.

Scoring is binary: a task is solved only if the model's response passes all associated rubrics. For reasoning models, only the final solution is evaluated, and thinking traces are excluded. That last point is a meaningful methodological choice rather than a detail, because including reasoning traces would let a model be scored on the quality of its process rather than on whether it delivered the answer.

The obvious consequence is that the benchmark depends on a closed model as its judge, and that the judge configuration is part of the benchmark definition rather than a local preference.

## The data format is three fields and a message list

Both datasets use the same JSONL format, and the structure is compact enough to read in one go:

```json
{
  "messages": [
    {"role": "system", "content": "..."},
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."}
  ],
  "rubrics": ["Rubric 1", "Rubric 2", "..."],
  "metadata": {
    "task_id": "unique-task-identifier",
    "context_category": "..."
  }
}
```

The messages array is the conversation as the model sees it, with the system prompt holding the framing, the user turn holding the task, and the assistant turn holding what the model produced. The rubrics array is the check list the judge works through, and the metadata object identifies the task and its category.

The format is the same for both benchmarks, which is why the same infer.py and eval.py run against either file, and it is also the format you need if you want to write your own inference loop. Nothing in the pipeline requires the OpenAI client except the default judge, so a model served by any compatible endpoint can be evaluated as long as you supply the base URL and key.

The repository itself is thin because the work is in the data and the papers. The top level holds infer.py, eval.py, requirements.txt, the two paper PDFs as clbench-paper.pdf and clbench-life-paper.pdf, an assets directory with the figures used in the README, a CNAME file for the project site, and a .github directory.

The requirements file is equally short, pinning openai at 1.0.0 or later and tqdm at 4.60.0 or later.

## Joining the leaderboard and the licence question

The submission process is straightforward and it is the mechanism behind the public leaderboard. Run inference and evaluation with the scripts, then submit a pull request to the repository with your graded result file, which the README names with the pattern *_graded.jsonl. The project says the results will be verified and the leaderboard updated.

Verification is the part to be aware of. Your submission is a JSONL file of graded outputs, and the maintainers check it before it goes on the board. That is a reasonable arrangement, and it also means the leaderboard depends on a human step and on the maintainers having capacity to verify it.

The citation information is a BibTeX entry with a long author list, the year 2026 and the arXiv eprint 2602.03587, so the project expects to be cited as work rather than used as a one-off test set.

On licensing, the repository record lists the licence as one that GitHub could not classify, and there is a LICENSE file in the repository root. That is worth resolving before you redistribute the task data. The data lives on Hugging Face under the tencent organisation, and the terms for the datasets are what govern the tasks themselves, while the code in this repository is governed by the file in the root.

The last thing to weigh is what a benchmark with 1,899 tasks can and cannot tell you. It measures one specific capability, learned from context, with rubrics written by domain experts and judged by a model. It does not measure general reasoning, it does not measure factuality against a reference corpus, and a single judge model sits between your system and the score. Those limits do not make it less useful, they define what the leaderboard number means.

## Conclusion

Adopt CL-bench if you are evaluating whether a model can pick up new domain knowledge, rules or procedures from context rather than from training, and you can afford an LLM judge in your evaluation loop. Do not adopt it as a general capability benchmark, since the tasks are narrow by design and the scoring is all-or-nothing per task. Before you run it, read the evaluation section carefully, because only the final solution is judged and thinking traces are excluded, and check the licensing and data terms on the Hugging Face datasets before redistributing the task data.

## FAQ

### What does CL-bench measure?

It evaluates whether language models can learn new knowledge from context at inference time, across four context categories: domain knowledge reasoning, rule system application, procedural task execution, and empirical discovery and simulation. Each task supplies a system prompt, a task, the context with the new knowledge, and rubrics written by experienced domain experts.

### How is CL-bench scored?

Evaluation is binary: a task is solved only if the model's response passes all associated rubrics, and for reasoning models only the final solution is evaluated, with thinking traces excluded. The default judge is GPT-5.1 with low reasoning effort for CL-bench and with high reasoning effort for CL-bench Life.

### How do I run the CL-bench evaluation?

Install openai and tqdm, download the JSONL file from Hugging Face, set OPENAI_API_KEY, run `python infer.py --model <model_name> --input CL-bench.jsonl --workers 20`, then run `python eval.py --input outputs/<model_output>.jsonl --judge-model gpt-5.1 --reasoning-effort low`. Non-OpenAI models are supported through the --base-url and --api-key options.

### What is the difference between CL-bench and CL-bench Life?

CL-bench holds 1,899 professional and domain-specific tasks with an average of 63.2 rubrics per context and 20 hours of expert effort per context, across 4 categories. CL-bench Life holds 405 context-task pairs with 5,348 rubrics averaging 10.7 per task, across 3 categories, using messy real-life contexts such as group chats, version logs and activity trails.

## Sources

- [Issues](https://github.com/Tencent-Hunyuan/CL-bench/issues)
- [README](https://github.com/Tencent-Hunyuan/CL-bench/blob/main/README.md)
- [Tencent-Hunyuan/CL-bench on GitHub](https://github.com/Tencent-Hunyuan/CL-bench)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/tencent-hunyuan-cl-bench