KLUE gave Korean language models eight tasks to be compared on

A comparison gap, not a model gap

KLUE was introduced to push Korean natural language understanding forward, and the problem it names is about measurement rather than architecture. Korean pre-trained language models had already appeared and were solving Korean NLP problems, the project notes, because pre-trained models brought clear performance gains in other languages. Yet none of the proper evaluation datasets had been opened. The project argues that this lack of a shared benchmark limits fair comparison between models and slows progress on model architectures. The bet behind KLUE is that a common measurement surface, plus the models to reproduce on it, is what lets two research groups argue about methods instead of about preprocessing. Everything in the repository follows from that bet: datasets, metrics, recipes and baseline models, published together.

Eight tasks, each with its own metric

The benchmark is composed of eight tasks, and the choice of tasks is what gives the benchmark its shape. They are Topic Classification (TC), Sentence Textual Similarity (STS), Natural Language Inference (NLI), Named Entity Recognition (NER), Relation Extraction (RE), Part of Speech plus Dependency Parsing (DP), Machine Reading Comprehension (MRC) and Dialogue State Tracking (DST). The metrics are not interchangeable across them, which is the first thing to understand before quoting any score. Topic Classification reports F1, STS reports Pearson's r, NLI reports F1 and accuracy, NER reports entity F1 and character F1, RE reports F1 and AUPRC, DP reports UAS and LAS, MRC reports exact match and ROUGE, and DST reports joint goal accuracy and slot F1. A single headline number for a Korean model is therefore a compression of eight different measurements, and the choice of which task to quote says more about the claim than the number itself.

KLUE-BERT and KLUE-RoBERTa exist to make the numbers arguable

Two models are released alongside the benchmark, and their stated purpose is reproducibility. KLUE-BERT and KLUE-RoBERTa are provided so that baseline results on KLUE can be reproduced, which in turn is what lets future work be compared with something. The configurations span a useful range. KLUE-BERT-base has an embedding size of 768, a hidden size of 768, 12 layers and 12 heads, and KLUE-RoBERTa-small keeps the same 768 embedding and hidden size with 6 layers and 12 heads, while KLUE-RoBERTa-base returns to 12 layers. KLUE-RoBERTa-large is the outlier at 1024 embedding size, 1024 hidden size, 24 layers and 16 heads. The project also states that the pre-trained models are uploaded to the Huggingface Model Hub, and the note points at the org page for the collection.

The baseline table is where Korean models separate

The published table lists the evaluation results of these models and other baselines, with the best performance in bold and the best among the base models in italic. It is a wide table, one column per metric, and the shape of the results is the interesting part. On Topic Classification the F1 column runs from 81.55 for mBERT-base to 86.06 for XLM-R-large, and the Korean models sit above the multilingual baselines, with KR-BERT-base at 84.58, koELECTRA-base at 84.59 and KLUE-BERT-base at 85.73. On Machine Reading Comprehension the spread is much wider: exact match goes from 44.66 for mBERT-base and 27.48 for XLM-R-base up to 75.58 for KLUE-RoBERTa-large, with KLUE-RoBERTa-base at 68.67. On Dialogue State Tracking, joint goal accuracy runs from 35.46 for mBERT-base to 50.22 for KLUE-RoBERTa-large. Two takeaways follow. Korean pretraining pays off on Korean tasks, and the tasks themselves differ a lot in how far apart the models are, which is an argument for reporting per-task results rather than a single score.

Where to get the data, the models and the ranking

There is no installation procedure here, and the project does not describe one. The paths it gives are a website, a model hub and a submission archive. The leaderboard lives at the address below, and the same address is what a reader should open first to see current results.

https://klue-benchmark.com

For the pre-trained models, the project description says that all of them are uploaded in the Huggingface Model Hub, and points readers at https://huggingface.co/klue for the collection. Submitting a result is handled through a separate archive, which the project links as the submission guideline for the code and the required output, at https://aistages-prod-server-public.s3.amazonaws.com/app/Competitions/000065/data/klue_code.tar.gz. The paper that documents the datasets, the metrics and the design decisions is cited in a BibTeX entry that the README includes in full, so a write-up of your results can point at the same reference.

@misc{park2021klue,
      title={KLUE: Korean Language Understanding Evaluation},

So a first practical use of KLUE looks like this. Take a model, in practice one of the released KLUE checkpoints or a model of your own, run it on the eight datasets, and compute the metric that belongs to each task rather than a single averaged figure. The recipes for fine-tuning pretrained models per task are part of what the project publishes, which is what makes a reported score reproducible rather than a private measurement. A team without its own Korean evaluation data can point at KLUE leaderboard rows and at the per-task baselines as the reference point.

Four principles, and one of them is about access

The design principles are stated in the project description and they explain several otherwise puzzling choices. The benchmark aims to cover diverse tasks and corpora, to be accessible to everyone without any restriction, to include accurate and unambiguous annotations, and to mitigate AI ethical issues. The accessibility principle is the one with consequences. There is no gated access, no application form and no institutional licence, which is why the datasets can be used by a student with a laptop and by a company with a training budget on the same footing. The ethics principle is the one that is least specific in the repository. The description mentions mitigation of AI ethical issues as a goal of the design process, and the paper is where the detail sits, since the README points to the paper for how the four principles guided task selection, corpus selection, annotation protocols, metric choices and baseline construction. The general principle that the datasets are open also means the annotations are a shared artefact, and accuracy of those annotations is treated as a first class requirement rather than a follow-up task.

A thin repository, an old tag and a share-alike licence

The repository itself is small. The top level holds the README, a License.md, a klue_benchmark directory and a .github directory, and that is all. There is no package manifest, no loader class and no install command, so a new user is expected to follow the wiki, the paper and the dataset documentation rather than a quickstart script. The release history is thinner still. The two published tags are v1.0.0 on 2021-06-15 and v1.1.0 on 2021-07-26, while the last push to the repository was on 2026-06-30, so work continued on the code long after the last version tag. If you need a pinned, dated artefact, v1.1.0 is the newest one available. The licence is CC-BY-SA-4.0, and for a dataset that choice is the point: attribution is required and derivatives carry the same licence, which is a fair condition for the annotation effort but a real consideration for a company planning to publish a modified split.

Editorial conclusion

Use KLUE when you need a number for a Korean model that another team can reproduce, and reach for it instead of a private evaluation set. Do not expect a pip install or a loader class, because the repository ships the klue_benchmark directory and the documentation rather than a packaged tool, and the last tagged release is v1.1.0 from 2021-07-26 even though the last push was on 2026-06-30. Before you build on the datasets, read the CC-BY-SA-4.0 terms for redistributed derivatives, and check the leaderboard at klue-benchmark.com to see which baselines the maintainers still consider current.

Frequently asked questions

Official sources

1. KLUE-benchmark/KLUE on GitHub
2. License: CC-BY-SA-4.0
3. Project website
4. README
5. Releases