# langtest: a Harness for 60 test types, shipped by two build backends that disagree on the version

> Pacific AI's testing library for NLP and LLM models, and the packaging details that decide what you actually install: poetry says 2.8.0, setup.py says 1.0.0, and the Python floor appears in two places that contradict each other.

**PacificAI/langtest** — Deliver safe & effective language models

- Repository: https://github.com/PacificAI/langtest
- Website: http://langtest.org/
- Stars: 561 · Forks: 54
- Language: Python
- License: Apache-2.0
- Published: 2026-09-17 · Updated: 2026-09-17 · Language: en
- Canonical page: https://hysenlabs.com/projects/pacificai-langtest

## setup.py declares 1.0.0 while the poetry project declares 2.8.0

The repository carries two independent packaging descriptions for the same distribution name, langtest, and they disagree about the version. The poetry table in pyproject.toml reads version = "2.8.0", matching the 2.8.0 release published on 2026-08-17. The setuptools module at the root is a separate file that calls setup() with version="1.0.0", and it is a template-style module with the packaging guide comments still in place. Both describe the same project, so which version string a given artifact carries depends on which build path produced it. The poetry half is the coherent one: it carries the homepage, repository URL, documentation link, keyword list, an Apache-2.0 licence field and a set of package classifiers. The setuptools half is a checklist with sample comments. For a consumer this is not visible at runtime, but for anyone packaging, mirroring or auditing the library it is the first thing to reconcile.

## Classifiers advertise Python 3.7 to 3.11 while the floor is 3.12

Inside the same poetry table, the interpreter support statements contradict each other. The classifier list names Python 3, then 3.7, 3.8, 3.9, 3.10 and 3.11, and ends there. The dependency line below it says python = ">=3.12,<3.15". A 3.11 install is therefore advertised in the metadata and refused by the resolver, and the upper bound of 3.15 is stated nowhere in the classifier list either, so the newest interpreter the project claims to work on is the one that has no classifier at all. The same file also marks the project as Development Status 5, Production/Stable, which is a claim about maturity that the interpreter spread does not support. For anyone gating a build matrix on the classifiers, this pair of lines is the place to look first.

## Two Transformers floors, 4.38 in one file and 5.5 in the other

The dependency conflict that most affects an install is again split across the two build descriptions. The setuptools REQUIRED_PKGS list asks for transformers>=4.38.0, alongside numpy, pandas, matplotlib, torch, sentencepiece, pydantic, nest-asyncio, jsonlines, langchain, evaluate, rouge_score and typing-extensions with a floor of 4.10.0 that carries an inline comment asking to remove it once the spacy issue is resolved. The poetry table asks for transformers >= 5.5.0, pydantic >= 2.10.0, and optional extras for johnsnowlabs, rouge-score, evaluate and huggingface_hub, with the latter floor set above 0.16.0. That last floor is worth reading carefully because it is an inequality with nothing after it, so any huggingface_hub version above 0.16.0 satisfies it. The poetry table also adds an explicit source for the CPU-only PyTorch wheel index, which is a deliberate choice for CI rather than an accident.

## The maintainer name changed and the links did not all follow

Ownership signals in this repository point at two different eras. The 2.6.0 release is titled a John Snow Labs release, and the 2.7.0 and 2.8.0 releases are titled Pacific AI releases. The package author contact is still an address at the johnsnowlabs.com domain, and the repository URL recorded in pyproject.toml points at github.com/Pacific-AI-Corp/langtest while the project now lives at PacificAI/langtest. Every badge in the page header follows the old path: releases, issues, the star history graph and the build workflow. One of them also names the branch as master, while the default branch is main, so that badge resolves against a branch name this repository does not use. None of this changes what the library does, and all of it matters if you are scripting against the repository URL or following the badges to find the issue tracker and the release notes.

## Three releases in eighteen months with an eleven month gap

The release history is short and unevenly spaced. 2.6.0 shipped on 2025-03-11 with de-biasing data augmentation, structured output evaluation, confidence tests for medical answers, expanded QA and summarization support and security work. 2.7.0 followed on 2025-09-22 with a clinical guideline benchmark, a robustness test named MedFuzz, randomized QA options, expanded clinical summarization, a MentalChat16K integration and security work. 2.8.0 arrived on 2026-08-17 with MedExQA and HeadQA support, a unified text-generation task for Hugging Face models, embedding evaluation fixes and security work. That leaves a gap of about eleven months between 2.7.0 and 2.8.0, with the last commit on the default branch dated 2026-08-28. Security enhancement appears in all three release titles, which tells you something about where the maintenance effort goes between new benchmarks.

## Three chained verbs and a model described by a dictionary

The entire usage example fits in four lines, and the shape of the API is worth reading closely because the model argument is not an object.

```python
# Install langtest
!pip install langtest[transformers]

# Import and create a Harness object
from langtest import Harness
h = Harness(task='ner', model={"model":'dslim/bert-base-NER', "hub":'huggingface'})

# Generate test cases, run them and view a report
h.generate().run().report()
```

The install line carries a leading exclamation mark because it is written for a notebook cell rather than a shell, which tells you the example assumes a notebook or IPython context. The extras marker in langtest[transformers] is what pulls the optional dependency group. The harness takes a task string and a dictionary whose model and hub keys name the checkpoint and where to fetch it from, and the three methods chain without any intermediate variable, ending in a report call.

## Data augmentation based on test results works for select models only

One feature line carries a qualifier that decides how much of the library applies to you: training data can be augmented automatically based on the test results, and that applies to select models. The rest of the dimensions listed are unconditional, namely robustness, bias, representation, fairness and accuracy, and the headline claim is that more than 60 distinct test types run from one line of code. The LLM surface is described separately and names its own targets: question answering, toxicity, clinical tests, legal support, factuality, sycophancy and summarization, against OpenAI, Cohere, AI21, the Hugging Face Inference API and Azure-OpenAI models. For classical pipelines the frameworks named are Spark NLP, Hugging Face and Transformers, scoped to NER, translation and text classification. The gap to be aware of is that the 60 test types are not enumerated anywhere in the repository, only in the online benchmark documentation.

## The test catalogue lives on a website while the repository holds the harness

What is in the tree is the machinery, not the catalogue. The repository carries the langtest/ package itself, a tests/ directory, a separate manual_tests/ directory, docs/ and sphinx/ for the documentation sources, a conda/ directory for the conda flavour, a poetry.lock, a .pre-commit-config.yaml, a CITATION.cff and a PULL_REQUEST_TEMPLATE.md. The demo/ directory holds tutorials, data, blogposts and a utils package. What the project considers its benchmark datasets is listed on langtest.org under the benchmarks path, together with the install documentation and the extended usage examples, all hosted off-repository. The page itself also leans on promotional blog posts for most of its narrative weight, several of which are written as marketing rather than as method, which is another reason to read the benchmark page rather than the README when you are deciding what a given test measures.

## Conclusion

LangTest earns a place in a pipeline that already has a task, a model identifier and an evaluation harness in mind, because the three-call chain gets to a report without a test-writing phase and the dimension list is broad enough to start from. It is a weak fit if you need to know exactly which test types exist before installing, because that catalogue lives on langtest.org rather than in the repository, or if your environment is Python 3.11 or older, since the dependency floor is 3.12 even though the classifiers still advertise older interpreters. Before you depend on it, check which build produced the wheel you hold, because the repository carries both a poetry project at version 2.8.0 and a setuptools setup.py that still reads 1.0.0, and confirm that the Transformers floor your environment has is the one the poetry build expects.

## FAQ

### How do I install LangTest and run my first test?

The page installs with `pip install langtest[transformers]`, imports Harness from langtest, builds a harness with a task and a model dictionary naming a checkpoint and its hub, then chains h.generate().run().report(). The install line is shown with a leading exclamation mark, so the example is written for a notebook cell.

### What does the LangTest harness actually check?

The stated dimensions are robustness, bias, representation, fairness and accuracy, with more than 60 test types said to run from one line of code. LLM tests cover question answering, toxicity, clinical tests, legal support, factuality, sycophancy and summarization. Automatic augmentation of training data from test results is limited to select models.

### Which Python versions does LangTest support?

pyproject.toml requires python >=3.12,<3.15, while its classifier list still names Python 3.7 through 3.11. A separate setup.py in the same repository declares transformers>=4.38.0 and version 1.0.0, against transformers >= 5.5.0 and version 2.8.0 in the poetry table.

### Who publishes LangTest releases now?

The release titles changed hands: 2.6.0 is a John Snow Labs release, while 2.7.0 and 2.8.0 are Pacific AI releases. The package author contact is still at the johnsnowlabs.com domain, and the licence is Apache-2.0.

### Which models and frameworks can LangTest test?

NER, translation and text classification are supported across Spark NLP, Hugging Face and Transformers. LLM testing is offered for OpenAI, Cohere, AI21, the Hugging Face Inference API and Azure-OpenAI models, and the listed benchmark datasets are enumerated off-repository at langtest.org.

## Sources

- [License: Apache-2.0](https://github.com/PacificAI/langtest/blob/main/LICENSE)
- [PacificAI/langtest on GitHub](https://github.com/PacificAI/langtest)
- [Project website](http://langtest.org/)
- [README](https://github.com/PacificAI/langtest/blob/main/README.md)
- [Releases](https://github.com/PacificAI/langtest/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/pacificai-langtest