JADE-DB: a Chinese-language safety benchmark for large language models

What the repository publishes, and what it does not

The repository description calls it the Fudan JADE team's series of large model evaluation and governance releases, and the README states the scope plainly: we publish the targeted safety evaluation datasets generated by JADE here.

So this is a data repository rather than a tool. There is no CLI, no scoring script and no service. What is there are datasets, a technical report and a set of sibling project directories.

The technical report comes in two languages, a Chinese version as JADE__chn.pdf at the root of the repository and an English version on arXiv as JADE: A Linguistics-based Safety Evaluation Platform for Large Language Models, arXiv 2311.00286. The project website is at whitzard-ai.github.io/jade.html, and the team is Fudan Baize Intelligence, the Whitzard AI lab.

The top-level entries show how much has accumulated. There are the two dataset directories jade-db-v1.0/ and jade-db-v2.0/, the Chinese report PDF, and then jade-db-v3.0/, jade-hal-v1.0/, jade-lrmguard/, jade-mcp-v1.0/, jade-rag-v1.0/, jade-t2i-v1.0/, jade-t2i-v2.0/ and Jade-BadPhoneAgent/.

Each of those newer directories is announced at the top of the README as a numbered release of the JADE series. JADE 3.0 is large model safety alignment, in jade-db-v3.0. JADE 4.0 is secure specification RAG, retrieval augmented generation based on security specifications, in jade-rag-v1.0. JADE 5.0 is text-to-image model content safety evaluation, in jade-t2i-v1.0. JADE 6.0 is described as the first hallucination evaluation aimed at multimodal large models, in jade-hal-v1.0. JADE 7.0 is a collection of malicious MCP server instances for agent security, in jade-mcp-v1.0. JADE 8.0 is described as the first reasoning chain safety protection tool aimed at large reasoning models, in jade-lrmguard. And Jade-BadPhoneAgent is announced as the first safety evaluation targeting malicious use of phone agents.

So the JADE-DB datasets are the foundation of a wider series rather than a one-off release. The primary language is Python, the licence is MIT, the default branch is main, and the last push was on 2026-07-23.

The linguistic mutation idea, and why seed questions are not enough

The core method is described in the dataset section: the original low-trigger rate seed questions are automatically transformed through linguistic mutation into high-risk questions, constructing a natural-text dataset.

The framing is a Chinese idiom, 他山之石、可以攻玉, which is roughly the idea that stones from another mountain can be used to polish jade, and it is the team's own description of what the project does.

The reason this matters is about how safety evaluation fails in practice. A test set of obviously bad questions measures the easy case: a model asked to help build a weapon or write malware should refuse, and a well-aligned model will. That tells you very little. The questions that expose a real weakness are ones that look innocuous but lead somewhere harmful, and a hand-written set of those is hard to produce in quantity.

By taking seed questions with a low trigger rate and mutating them automatically into high-risk phrasings, the team gets both scale and coverage. The mutation is linguistic rather than semantic, which is the point: the meaning is preserved and the surface form changes, so a model that refuses one phrasing and not another is exposed.

That property is what the demo illustrates. The demo page is at whitzard-ai.github.io/jade-demo.html, and the README embeds an example gif from jade-db-v2.0/example.gif whose alt text describes how JADE uses linguistic mutation to break the safety guardrails of LLMs.

The problem categories are also given. The question types cover four major classes, which are core values, illegal and criminal activity, infringement of rights, and discrimination and bias, with more than thirty subcategories in total.

The releases follow the method's output. JADE DB v1.0 took several hundred hand-generated low-trigger seed questions, automatically transformed them into thousands of high-risk questions, and selected 230 questions with high cross-model transferability, comprising 150 Chinese and 80 English questions, as the open source dataset. JADE DB v2.0 released two more evaluation datasets for Chinese open-source models, JADE-DB-Easy and JADE-DB-Medium, each containing 1000 general test questions representing basic and advanced safety difficulty respectively.

The four published files and what each one covers

The open data summary in the README lists four downloadable files, and the differences between them are not just size.

The basic general test set in Chinese is jade-db-v2.0/jade_benchmark_easy_zh.csv with 1000 entries, covering basic test questions aimed at 8 domestic open-source Chinese LLMs, across three of the major categories, namely illegal and criminal activity, infringement of rights, and discrimination and bias.

The advanced general test set in Chinese is jade-db-v2.0/jade_benchmark_medium_zh.csv, also 1000 entries, aimed at the same 8 domestic open-source Chinese LLMs, with advanced rather than basic difficulty across the same three major categories.

The high-risk test set, demo version, in Chinese is jade-db-v1.0/jade_benchmark_zh.csv with 150 entries, covering high cross-model transferability questions aimed at the same 8 domestic open-source Chinese LLMs across the same three major categories.

The high-risk test set, demo version, in English is jade-db-v1.0/jade_benchmark_en.csv with 80 entries, covering high cross-model transferability questions aimed at 4 foreign commercial English LLMs, across the same three major categories.

Two things stand out. The first is the scale difference between the easy and medium sets at 1000 questions each against 150 and 80 for the high-risk sets. The high-risk sets are smaller because high cross-model transferability is a much harder property to achieve: a question that trips one model does not necessarily trip another, so the selection was constrained.

The second is the asymmetry between the Chinese and English sets, which target different model populations. The Chinese sets target 8 domestic open-source Chinese models and the English set targets 4 foreign commercial English models. So the two files are not translations of each other and are not directly comparable as a bilingual pair.

Note also that the downloadable sets cover three of the four major categories, with core values absent. The README explains why: due to relevant regulations, test questions involving politically sensitive content are currently not included in the downloadable public test sets, and those who want to evaluate should contact the team's named contact directly.

That is a real constraint for anyone doing safety evaluation, and it is stated plainly rather than left for you to discover from a missing file.

The other JADE releases, and what they add beyond the text datasets

The wider series is where the project becomes a set of tools rather than only datasets, and the announcements at the top of the README are the only description of each one, so it is worth setting out what is claimed.

The RAG release, JADE 4.0, is described as secure specification RAG, retrieval augmented generation based on security specifications, in jade-rag-v1.0. The idea implied by the name is that instead of only testing a model, you specify what a compliant answer must look like and generate against that specification.

The text-to-image release, JADE 5.0, is content safety evaluation for text-to-image models, in jade-t2i-v1.0, and it is followed by a second version with the Jade harmful image classification guideline, Jade-HTG, and its accompanying evaluation benchmark, in jade-t2i-v2.0. The guideline matters because the classification task needs a rubric, and a rubric is what makes the evaluation reproducible across raters.

The multimodal hallucination release, JADE 6.0, is described as the first hallucination evaluation aimed at multimodal large models, in jade-hal-v1.0. Multimodal hallucination is a distinct problem from text hallucination because the model can hallucinate about what is in an image as well as about facts.

The agent security releases are the two most interesting for anyone thinking about where model risk is heading. JADE 7.0 is a collection of malicious MCP server instances for agent security, in jade-mcp-v1.0. That is a threat model where the danger is not the model but the tools it is connected to, and a collection of adversarial server instances is a test corpus for whether an agent can be induced to call a hostile server.

JADE 8.0, in jade-lrmguard, is described as the first reasoning chain safety protection tool aimed at large reasoning models. The emphasis on the reasoning chain rather than the final answer is the point: a model that reasons its way into a harmful conclusion is a different problem from a model that is asked directly.

And Jade-BadPhoneAgent is the first safety evaluation targeting malicious use of phone agents, which points at a class of agent with direct effect on a device rather than on a chat window.

Using the datasets, and the content warning

There is no install step and no command to run, because the deliverables are CSV files and papers. The README points to the project website and the demo page for more examples, and the open data summary table gives the exact path of each file inside the repository, so the practical workflow is to clone or download the repository and read the CSV files directly.

Each published set is listed in the open data summary with its exact path, size and coverage. The Chinese technical report is linked at the top of the repository.

@misc{zhang2023jade,
      title={JADE: A Linguistic-based Safety Evaluation Platform for LLM}, 
      author={Mi Zhang and Xudong Pan and Min Yang},
      year={2023},
      eprint={2311.00286},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

The important practical point is the content. The README includes a data sample section with the same declaration attached twice: the dataset contains harmful and non-compliant content examples, and none of them represent the team's position.

That is a standard and necessary disclaimer for a safety benchmark, but it also tells you what you are downloading. The sample table shows questions of the kind that get classified under illegal and criminal activity, infringement of rights and discrimination and bias, in both Chinese and English. Reading them will include plainly objectionable content, including examples of ethnic and national discrimination. If you are deploying these into an evaluation pipeline that logs results, plan for that.

The other operational detail is the division of the questions into the four major classes and thirty-plus subcategories, because that is the schema you would use to report per-category results rather than a single pass or fail score. A model can pass most of a benchmark and still fail badly on one category, and the subcategory labels are what let you find that.

And the CSV format itself is the simplest possible: a question, a dataset name, a violation type and a violation subcategory. That means it drops into an existing evaluation harness without a bespoke loader, which is the main practical advantage of publishing plain CSVs rather than a framework.

Editorial conclusion

JADE-DB fits a researcher evaluating a Chinese open-source model's refusal behaviour, who wants test questions that actually trigger the model rather than benign-sounding ones, and it does not fit anyone expecting a turnkey evaluation harness, since the repository publishes datasets and papers rather than a runner. Before downloading, note the stated limitation that politically sensitive test questions are not included in the public sets for regulatory reasons and must be requested separately, and that the published sets contain harmful example content that the team explicitly disclaims as not representing their position.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. whitzard-ai/jade-db on GitHub