presidio-research: synthetic PII corpora and a scoring harness for Presidio recognizers

The gap presidio-research fills: labelled PII sentences and a score you can compare

Presidio itself detects PII in text. It does not tell you how well it does so. To answer that you need two things that are awkward to assemble by hand: a corpus of sentences with exact character spans marked as PERSON, PHONE_NUMBER or whatever entity set you care about, and a scoring routine that tolerates the fact that two detectors rarely emit the same labels for the same span. The README frames the package as evaluation and data-science capabilities for Presidio and PII detection models in general, plus a fake data generator that builds synthetic sentences from templates and fake PII. The intended audience is stated directly: people developing or evaluating PII detection models, an existing Presidio instance, or a single Presidio recognizer, and people generating new data from previous datasets or sentence templates to widen entity coverage for NER models. That second group matters. If you already have a labelled dataset, this package is a conversion layer and a scorer. If you have no labelled data at all, the template generator is the entry point, and the quality ceiling of everything downstream is set by how good your templates are.

Templates in, tagged spans out: the generation pipeline

Generation starts from a file of templates such as `My name is {{name}}`. The generator samples templates and PII values, then tokenizes the resulting sentence and produces tags in IO, BIO or BILUO scheme along with spans for each new sample. The repository also ships a fake PII dataset, and notebook 2 is an exploratory analysis of it, which is the honest way to find out what entity coverage you are actually getting before you train on it. The fake identities come from Fake Name Generator under a Creative Commons Attribution-Share Alike 3.0 United States licence, and the README carries the trademark notice for Corban Works, LLC. That is worth reading before you redistribute a generated corpus: the MIT licence on the code does not automatically settle the terms of the names embedded in the data. The generator's weakness is structural. Because sentences are built from templates, a model trained on the output learns the template distribution as much as the entity distribution, which is precisely why the repository includes a dedicated splitting notebook rather than leaving you to call train_test_split.

InputSample is the contract between generation, conversion and scoring

Everything routes through one object. The README describes a standardized structure, `List[InputSample]`, defined in presidio_evaluator/data_objects.py, which holds the information needed for generating, analyzing, modeling and evaluating. The conversions are shown as short code blocks. To CoNLL, you read a JSON dataset with `InputSample.read_dataset_json("data/synth_dataset_v2.json")`, call `InputSample.create_conll_dataset(dataset)`, and write the result with `conll.to_csv("dataset.csv", sep="\t")`. Back the other way, a `CONLL2003Formatter` built with `files_path=Path("data/conll2003").resolve()` turns a folder of CoNLL2003 files into input samples via `to_input_samples(fold="train")`. For spaCy v3 there is `InputSample.create_spacy_dataset(dataset, output_path="dataset.spacy")`, and plain JSON goes out through `InputSample.to_json(dataset, output_file="dataset_json")`. The design choice here is a single in-memory shape with adapters at the edges. It keeps the evaluator agnostic about whether your data came from a template generator, a public NER corpus, or a production export. It also means an entity label mismatch anywhere in the chain surfaces at conversion time rather than silently skewing your metrics.

Evaluating a recognizer, and the entity mapping problem nobody escapes

The evaluation path covers three scopes: Presidio as a whole system, a NER model, or one specific PII recognizer, producing precision, recall and error analysis. Notebook 4 runs this against vanilla Presidio, and the README states plainly that the results are not very accurate. Notebook 5 then configures Presidio to detect PII much more accurately and reports an f-score boost of roughly 30 percent. Treat that number as what it is: a figure from the project's own notebook comparing a default configuration against a tuned one on synthetic data. It is not a claim about your data. The harder problem is labels. Different models return different entity sets for the same span, so notebook 6 covers interactive entity mapping, which the README calls crucial when evaluating multiple models. This is the part teams underestimate. A recognizer that emits `US_SSN` and a gold corpus that says `SSN` will score as a total miss unless the mapping is declared. The package gives you the tooling; it does not guess the mapping for you, and there is no automatic label alignment described in the README.

Pattern leakage: why the split notebook exists

Notebook 3 splits a generated dataset into train, test and validation while keeping each template in exactly one fold. The README notes this applies only to synthetically generated data, and the reason is obvious once stated: if `My name is {{name}}` appears in both training and test, the test set is measuring memorization of a sentence frame, not generalization to unseen phrasing. A random split of template-generated data will therefore report numbers that look better than the model deserves. This is the single most defensible design decision in the repository, and it is also a boundary. If your data is real text rather than generated, this splitting logic does not apply to you, and you are back to whatever deduplication and near-duplicate handling your own pipeline provides. Nothing in the README suggests the package solves that case.

Installation, the spaCy models, and the dependencies that are gone

From PyPI the sequence is `pip install presidio-evaluator`, then `python -m spacy download en_core_web_sm` for tokenization and `python -m spacy download en_core_web_lg` for NER. Note the package name difference: the distribution is presidio-evaluator while the import namespace is presidio_evaluator. From source you clone the repo, install uv with `pip install uv`, run `uv sync --extra dev`, download `en_core_web_sm` for tokenization and `en_core_web_lg` for the default Presidio configuration, then verify with `uv run pytest`. The README adds a warning that some dependencies, such as Flair and Stanza, are no longer supported, and directs you to Presidio Analyzer directly for adding custom NER models. That is a real constraint, not a footnote. The repository topics still list flair and transformers, so anyone arriving from a topic search may expect those integrations to work. The README says otherwise. Plan for the supported path, which is spaCy plus Presidio Analyzer, and check the notebooks before assuming a deep-learning training route is still maintained here.

Where it stops being the right tool

The generator cannot produce language it has not been templated for. If your production text contains PII in constructions your template file does not cover, no amount of generation will measure your detector on them, and the evaluation will be optimistic in a way that is hard to detect from the numbers alone. The second failure mode is scope confusion. This is a research and evaluation package; it does not sit in a request path, and nothing in the README describes a serving component, a latency budget, or a deployment story. If you need to redact text at runtime, this is not the library you install. Third, the entity mapping step is manual and interactive, which does not scale to dozens of heterogeneous models without someone maintaining the mapping definitions. Fourth, if you already have a large, well-labelled, real-world PII corpus and a scoring script you trust, the marginal value here is the conversion layer and the error analysis helpers, not the generator. A reasonable alternative for that situation is to stay with a general NER evaluation stack such as seqeval-style span scoring over your own CoNLL files. The difference in approach is that seqeval scores a tag sequence, while presidio-research scores span-level detections from an analyzer that may return overlapping or differently typed entities, and it adds the entity mapping layer on top. Pick based on whether your outputs are tag sequences or detector results.

Maintenance cost and what the licence does and does not cover

The code is MIT, which is permissive and places few obligations on how you use the library. The generated data is a separate question. The README attributes the fake identities to Fake Name Generator under CC BY-SA 3.0 US, a share-alike licence, and includes the Corban Works trademark notice. Share-alike terms can attach to derivative datasets in ways MIT does not, so if you plan to publish or redistribute a corpus built from the bundled fake PII, read those terms yourself rather than assuming the repository licence settles it. I am not giving legal advice here; the point is that the two licences cover different artifacts. On maintenance, the release cadence visible in the metadata is three releases in the weeks around July and August 2026, at versions 0.3, 0.3.1 and 0.3.2, with the last push in September 2026. The pre-1.0 version number is the honest signal: interfaces such as the data objects and formatters can still move. The dropped Flair and Stanza support shows that this package does shed dependencies rather than carry them indefinitely, so pin your versions and read the release notes before upgrading across a minor bump.

Editorial conclusion

Adopt presidio-research if you are building or tuning a Presidio recognizer and need labelled PII sentences plus a precision/recall harness before you touch production traffic. Do not adopt it as a detector or as a source of real-world test data: everything it generates is synthetic and template-bound. Verify three things first: that the spaCy models your pipeline needs (en_core_web_sm for tokenization, en_core_web_lg for the default Presidio configuration) are downloadable in your environment, that your train/test split goes through the pattern-number logic in notebook 3 rather than a naive random split, and that the entity mapping in notebook 6 covers the label sets of every model you intend to compare.

Official sources

1. data-privacy-stack/presidio-research on GitHub
2. License: MIT
3. Project website
4. README
5. Releases