Dingo (MigoXLab/dingo): an AI data quality evaluation tool for LLM datasets

What Dingo evaluates, and who the README is written for

Dingo describes itself as a comprehensive AI data, model and application quality evaluation tool, aimed at ML practitioners, data engineers and AI researchers. The scope it claims is wider than dataset cleaning: the README lists pre-training datasets, fine-tuning datasets, RAG systems and production AI systems as targets. The topics attached to the repository name the same areas, including data-quality-assessment, hallucination-detection, llm-as-a-judge and agent-as-a-judge.

The practical unit of work is a record with fields. The README's own example builds a Data object with a data_id, a prompt and a content string, then applies evaluators to it. Multi-field evaluation is called out explicitly: different quality rules can run against different fields in parallel, with ISBN validation given as the example for an isbn field and text quality for a title field. That is the design decision worth noticing. Dingo is not a single pass/fail scorer over a whole file; it expects you to say which field each check belongs to.

The audience implied by the feature list is teams that already have data and want a repeatable report, not teams looking for a labelling tool. The README does not describe annotation workflows or human review queues.

Rule, LLM and agent evaluators in one run

The mechanism is a set of evaluator classes that you import and invoke directly from Python. The README's chat-data example imports EvaluatorLLMArgs from dingo.config.input_args, Data from dingo.io.input, LLMTextQualityV4 from dingo.model.llm.text_quality.llm_text_quality_v4, and RuleSpecialCharacter from dingo.model.rule.rule_common. Those four import paths show the split: configuration arguments, the input record type, model-based evaluators, and rule-based evaluators.

The README states there are 30 or more built-in heuristic rules, and describes the overall approach as a hybrid of rules, LLMs and agents, with rules for fast checks and LLM assessment for deeper judgement. The rule side is deterministic and cheap. The LLM side depends on an external model, and the example imports EvaluatorLLMArgs, which is where that model is configured. The repository also carries a litellm extra pinned as litellm>=1.80.0,<1.87.0, which is the dependency that lets model evaluators talk to different providers.

Execution is not limited to a single machine. The README offers running locally for rapid iteration or scaling with Spark for billion-scale datasets. That is a real fork in deployment: the local path is ordinary Python, while the Spark path changes how the evaluation is distributed. The README does not spell out the Spark configuration in the portion available here, so treat that as something to read up on before promising it to a team.

Data sources are handled separately from evaluators. The README lists local files, SQL databases (PostgreSQL, MySQL, SQLite), HuggingFace datasets and S3 storage. Each of those is a connector concern, and the repository keeps datasource support inside the core package rather than an extra.

Installing dingo-python and running a first check

The package name on PyPI is dingo-python, not dingo. The README gives one core install and three extras. Start with the core package, which the README says includes rule evaluation, LLM evaluation, the MCP server and datasource support.

pip install dingo-python

If you need hallucination detection, the README points to the hhem extra, which pulls in transformers and torch. The setup.py comments note that the hallucination model moved from Vectara HHEM to MiniCheck, a standard T5 model, and that this removed the old transformers<4.49 ceiling. The extra name is still hhem.

pip install "dingo-python[hhem]"

Retrieval benchmark evaluation is a separate extra, described as MTEB plus pytrec-eval-terrier. There is also an all extra combining HHEM, agent and retrieval, and a separately isolated lmdeploy extra that setup.py keeps out of all because it requires transformers>=4.56 and pulls in heavy dependencies.

pip install "dingo-python[retrieval]"

For a first real check, the README's chat-data example constructs a record and imports both a rule evaluator and an LLM evaluator. The content string in that example deliberately contains a replacement character and a stray caret, which is the kind of thing RuleSpecialCharacter is meant to catch.

from dingo.config.input_args import EvaluatorLLMArgs
from dingo.io.input import Data
from dingo.model.llm.text_quality.llm_text_quality_v4 import LLMTextQualityV4
from dingo.model.rule.rule_common import RuleSpecialCharacter

data = Data(
    data_id='123',
    prompt="hello, introduce the world",
    content="\ufffdI am 8 years old. ^I love apple because:"
)

What you should see after wiring up the evaluator call is a result object per record. The README's example is truncated at the point where the evaluators are invoked, so the exact return shape is something to confirm from the examples/ directory, particularly examples/llm_and_rule/, before you build reporting on top of it.

Where Dingo is the wrong tool

The LLM-based evaluators require a model endpoint. EvaluatorLLMArgs exists precisely because the model has to be supplied, and the litellm extra carries a narrow version range, litellm>=1.80.0,<1.87.0. If your environment cannot reach an external model, or if you pin litellm outside that window, the model-based evaluators are not available to you and you are left with the rule set. That is a meaningful downgrade, because the README positions rules as the fast layer and LLMs as the deep one.

The packaging itself is a constraint. Hallucination detection needs torch and transformers through the hhem extra. Retrieval evaluation needs MTEB and pytrec-eval-terrier through the retrieval extra. lmdeploy is deliberately kept out of the all extra because of its dependency weight, so installing all does not give you everything. A team that assumes all means everything will be surprised.

Nothing in the available material describes a built-in rollback or dataset versioning mechanism. Dingo evaluates and reports; it does not appear to store prior states for comparison. If you need to prove that a dataset change improved quality over last week's run, the comparison has to live in your own storage.

Finally, the README's own framing pushes teams that want a visual product toward the SaaS edition, which lists a web UI, JWT and Google OAuth 2.0 access control, interactive reports and a RESTful API. The open source package is a library. If your stakeholders will not read Python output, the library alone will not satisfy them.

Dingo compared with a general LLM evaluation harness

The natural comparison is a general LLM evaluation framework, such as OpenCompass, which appears in the repository's topic list. The difference in approach is what sits at the centre. A general evaluation harness treats a model plus a benchmark as the unit: you point it at a model, pick a benchmark, and get a score that lets you rank models against each other.

Dingo treats a record with fields as the unit. The README's multi-field example is the clearest statement of this: ISBN validation on one field and text quality on another, in the same run. That is a data-cleaning posture, not a leaderboard posture. It fits a workflow where you are deciding whether a training corpus is fit to use, or whether a RAG pipeline is returning good text, rather than deciding which model to buy.

The overlap is the LLM-as-a-judge layer, which both approaches use. If your actual question is which of two models performs better on a fixed benchmark, a benchmark harness is the better fit, and Dingo's per-field rule engine is overhead you do not need. If your question is whether this month's crawled dataset is cleaner than last month's, the field-level model is the one that maps onto the problem.

Maintenance, releases and licence

The repository is not archived and the last push was on 2026-09-11, three days before this writing. Recent releases are v2.5.0 on 2026-08-04, v2.4.0 on 2026-05-29 and v2.3.0 on 2026-05-29. The cadence visible here is a minor release roughly every one to three months, with two releases landing on the same day in May. setup.py reports version 2.5.0, matching the latest tag.

Upgrade cost is mostly dependency cost. The extras are declared separately in requirements/ files, and the litellm range is capped at the top end, so a future litellm release can break the model evaluators until that pin moves. The setup.py comments show this has already happened once in the other direction: the hallucination detector was swapped from Vectara HHEM to MiniCheck to escape a transformers ceiling. Expect the optional dependency surface, not the core, to be where upgrades bite.

The open source package is Apache-2.0, which is permissive and includes an explicit patent grant. The SaaS edition described in the README is a separate product reached through an application form with a stated review time of one to five business days, and the README does not state its licence. Do not assume the Apache-2.0 terms cover the hosted edition; check that separately. Nothing here is legal advice.

Editorial conclusion

Adopt Dingo if you already have a dataset or RAG pipeline and want field-level quality rules plus LLM judgement in one Python call, and if you accept that the packaging is split across extras that must be chosen up front. Do not adopt it if you need a hosted dashboard with access control, since the README points those users at the separate SaaS edition, or if you cannot run an LLM endpoint for the model-based evaluators. Before committing, verify which extras your checks require, confirm the licence terms of the SaaS edition separately from the Apache-2.0 open source package, and read examples/llm_and_rule/ to see how rule and LLM evaluators are combined on one field.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. MigoXLab/dingo on GitHub
3. Project website
4. README
5. Releases