LLM-eval-survey: a paper index, not an evaluation harness

The problem is orientation, not measurement

Anyone asked to evaluate a language model for the first time faces the same problem: the relevant work is spread across arXiv preprints, conference proceedings and blog posts, and it is organised by whichever venue accepted it rather than by what is being measured. The README of LLM-eval-survey addresses exactly that. It describes itself as "a collection of papers and resources related to evaluations on large language models", and the list is grouped by the taxonomy of the companion survey rather than by publication venue. The intended reader is someone who needs to know which axes of evaluation exist before choosing one, or who is writing a related survey and wants a starting bibliography. It is not aimed at the engineer who wants a score for a model by Friday.

What the repository actually contains

The repository holds a README, a logo image, a framework diagram and a set of paper entries. Each entry follows a fixed shape: a title, an author list, a venue and year, and a link. The NLP section, for example, is subdivided into natural language understanding (with sentiment analysis, text classification, natural language inference and an others group) and reasoning, and the same pattern repeats under robustness, ethics, biases and trustworthiness, social science, natural science and engineering, medical applications, agent applications and other applications. There is no source directory, no requirements file and no evaluation script described in the material. The README also links two related projects, PromptBench and LLM-eval, and the framing makes clear that those are separate efforts rather than components of this one. If you came here expecting to run something, the repository will not meet that expectation.

The taxonomy is the contribution, and it is coarse

The table of contents is the part worth reading closely. It splits evaluation into what to evaluate and where to evaluate, and the what branch is a two-level tree: application area first, capability second. That structure is useful because it exposes gaps. Sentiment analysis and text classification sit in the same branch as natural language inference, which tells you the survey treats them as comparable units of measurement even though the tasks differ in how much they depend on annotation quality. The where to evaluate section is present in the table of contents but its anchor is written as href="where-to-evaluate" without the leading hash that the other entries use, so the link is likely broken as rendered. That is a small thing, but it is the kind of detail that shows the repository is maintained as a document rather than as software with link checks.

Getting it running means cloning a README

There is nothing to install. The practical commands are the ones for reading it locally or citing it. To keep a local copy for searching, clone the default branch, which the repository metadata gives as main:

git clone https://github.com/MLGroupJLU/LLM-eval-survey.git

If you want the rendered version with working anchors, read it on GitHub or on the project's own page at https://llm-eval.github.io/. To follow the underlying paper, the README points at https://arxiv.org/abs/2307.03109, and it states plainly that the arXiv version cannot be updated in real time, so the repository is the place to look for the latest entries. The README also gives a citation section, so if you are writing a related paper you should take the BibTeX from there rather than reconstructing the author list, which spans Jilin University, Microsoft Research, the Institute of Automation at CAS, Carnegie Mellon, Westlake University, Peking University, the University of Illinois and HKUST.

No licence is stated, and that is a real constraint

The repository metadata supplied here does not include a licence, and the README does not name one. That matters more than it does for a code project, because the repository is a compilation of third-party links and titles rather than original content. Linking to a paper is not the same as redistributing it, and the entries here are links, so the practical risk is low. But if you intend to reuse the taxonomy or the list wholesale in your own survey, you cannot point to a licence that grants you permission. The honest position is that the licence status is unverified from the available material, and anyone planning to mirror the list or republish it in another form should check the repository directly before doing so. This is a factual gap, not a legal conclusion.

It is a snapshot with a maintenance tail

The README carries a news section with two dated entries: the first arXiv version on 07/07/2023 and a second version on 12/07/2023, the latter accompanied by a Chinese blog post. The repository metadata shows a last push of 2026-09-05, which is later than those news entries, so the paper list has presumably grown since the news section was last touched. That asymmetry is the maintenance cost in miniature: the list can be appended by pull request, but the prose around it, the news entries and the arXiv paper itself, lag behind. The README invites exactly this, asking readers to open pull requests or issues and promising acknowledgement in the acknowledgements section. For a reader, the cost is that you cannot tell from the README alone how current any given subsection is. For a contributor, the cost is that a new entry has to be placed into an existing two-level tree, and there is no stated inclusion criterion, so the boundary between a relevant paper and an adjacent one is a judgement call.

When a runnable benchmark is the better tool

The README itself points to PromptBench, described as a prompt benchmark for robustness evaluation of LLMs. The difference in approach is the whole point. LLM-eval-survey tells you which robustness questions have been asked and by whom. PromptBench, by contrast, is the kind of project where you install a package and run evaluations against prompts and models. If your question is "how does this model behave under adversarial prompt perturbations", the survey index can tell you that the question is well studied, but it will not produce a number. If your question is "which axes should my internal evaluation cover", the index is faster than reading twenty papers, and it is the cheaper artifact to maintain because it carries no code and no dependency graph. Choose based on whether your output is a decision about scope or a decision about a specific model.

What to verify before you rely on it

Three things are worth checking before you cite or build on this repository. First, the licence, which the available material does not state. Second, the freshness of the subsection you care about, since the news entries stop in July 2023 while the repository has been pushed to since. Third, whether the paper you need is actually linked: the entries include well-known work such as Holistic Evaluation of Language Models and Chain-of-Thought Hub, but the README excerpt is truncated and the later application sections are not visible here, so coverage in medical or agent evaluation cannot be confirmed from this material. The repository's own note, that the arXiv paper is not updated in real time and that the repository is the current source, is the reason to treat the index as the primary artifact and the preprint as the stable citation.

Editorial conclusion

Adopt this repository as a reading list if you are scoping an evaluation effort or writing a related survey, because its taxonomy is the survey's own and the index is updated more often than the arXiv preprint. Do not adopt it if you need to execute anything: there is no harness, no metric code and no dataset loader here, so a runnable benchmark such as PromptBench is the correct starting point. Before relying on it, check the licence, which the repository materials do not state, and check the paper list's last update date rather than assuming the index is current.

Official sources

1. Issues
2. MLGroupJLU/LLM-eval-survey on GitHub
3. Project website
4. README