JSON Schema Test Suite, the shared conformance tests for schema validators

What the suite is and is not

The JSON Schema Test Suite contains a set of JSON objects that implementers of JSON Schema validation libraries can use to test their validators. The README is careful about the distinction, saying the repository exists to verify specified behaviour defined by the JSON Schema specification and should not be confused with a style guide.

That distinction is the single most important thing to understand before reading the test files. It is not intended to demonstrate how schemas ought to be written. Tests may appear unusual or unintuitive, but they exist solely to exercise behaviour prescribed by the specification. A test that looks like a bad schema is not a mistake; it is a case where the specification requires a particular validation outcome, and your validator has to produce it even though you would never write that schema yourself.

The suite is meant to be language agnostic and should require only a JSON parser. The conversion of the JSON objects into tests within a specific language and test framework of choice is left to the validator implementer. That is a deliberate design choice: the shared artefact is the data, and each implementation writes the small amount of code that reads it.

The recommended workflow is to clone the main branch as a git submodule or a git subtree, and the README notes that the main branch is always stable. That is worth relying on, because the alternative of copying the files into your repository means your copy goes stale and your conformance report becomes a claim about a year-old specification.

The project status badge says Active, the project has reached a stable, usable state and is being actively developed, and there is a Contributor Covenant badge for version 2.1. The licence is MIT, and there is a Zenodo DOI badge, which means the suite is archived with a DOI for citation. It is also funded through Open Collective under the json-schema organisation.

Coverage across specification versions

All JSON Schema specification releases should be well covered by the suite, and the README lists them: draft-2020-12, draft-2019-09, draft-07, draft-06, draft-04 and draft-03.

That range matters more than it might look. JSON Schema has had six published drafts, and they differ in ways that break validators. A library that claims to support draft-07 is not automatically draft-2020-12 compatible, because the later draft changed how references are resolved across schema documents, how vocabularies are declared, and how some keywords are applied. A validator that passes the tests for one draft and fails the tests for another is a common and confusing failure, and the suite exists precisely to make that visible.

The README also explains the asymmetry in maintenance effort. Draft-04 and draft-03 are considered frozen, in that less effort is put in to backport new tests to those versions. So if you support an old draft, the coverage you get is good but not identical to the newer drafts, and the frozen label is the reason.

Additional coverage is always welcome, particularly for bugs encountered in real-world implementations, and the README invites issues and pull requests. The framing is telling: the most valuable contributions are not new specification features but the cases where a real implementation was found to disagree with the specification, which is exactly the kind of gap a test suite can close and a specification text cannot.

There is also a related project worth knowing about. The README notes that gregsdennis has started a separate test suite modelled on this one to cover third-party vocabularies. That matters because the core suite covers the keywords in the specification itself, and a lot of real schemas use vocabularies such as the format vocabulary, which this suite does not cover.

The file layout and the vocabulary it uses

The tests live in the tests directory at the root of the repository, with a subdirectory for each released version of the specification. There is one additional directory, latest/, which is a symbolic link pointing to the directory of the most recent release, and the README says this may be useful for implementations providing specific entry points for validating against the latest version.

Inside each version directory there are a number of .json files, each containing a collection of related tests. The grouping is often by property under test, but the README says not always. In addition, each version directory contains one or more special subdirectories.

The first is optional/, containing tests that are considered optional. The README is refreshingly candid that this subdirectory currently conflates many reasons why a test may be optional: it may be because the tests within a particular file are genuinely not required by the specification but are still potentially useful, or because they only apply to programming languages with particular functionality, in which case they are not truly optional in that language. The README says the directory structure will be made richer to reflect those differences more clearly in the future. For a validator implementer, the practical rule is that a failure in optional/ is not a conformance failure unless you are in a language the test targets.

The second is proposals/, containing a subfolder for each active proposal to the specification. If the proposal is a keyword, which is generally the case, the subfolder bears that keyword's name. Inside is a series of test files containing amendments to the required test suite should the proposal be incorporated into the specification. The README says these tests should be considered volatile while the proposal is in development, but that implementations claiming to support the proposal are expected to pass its tests. So a validator that advertises support for a draft keyword is expected to pass the proposal tests for it, even though those tests may still change.

Each .json file is a single JSON array of test cases, and the README defines the terminology precisely. A test suite is the whole repository, covering tests for multiple specification releases. A test case is a single schema with a description and an array of tests. A test is a single example with a description, an instance and a boolean saying whether the instance is valid under that schema. A test runner is a program outside the repository, written by whoever is implementing the validator, that executes each test.

An example and how to wire the suite into a validator

The README gives a single test case that shows the whole shape at once:

{
    "description": "The test case description",
    "schema": { "type": "string" },
    "tests": [
        {
            "description": "a test with a valid instance",
            "data": "a string",
            "valid": true
        },
        {
            "description": "a test with an invalid instance",
            "data": 15,
            "valid": false
        }
    ]
}

A schema, an array of instances, and a boolean per instance saying whether it should validate. The runner walks the files, applies the schema to each instance and compares the outcome with the expected boolean.

The formal definition of the contents of a test case is itself a JSON Schema, and the README says it can be found alongside the README in a file named test-schema.json. The repository also carries an output-test-schema.json and an output-tests/ directory, plus a remotes/ directory, an annotations/ directory, a bin/ directory, a tox.ini, a CONTRIBUTING.md and a package.json.

The package.json is minimal, describing the project as a language agnostic test suite for the JSON Schema specifications, with the author given as the json-schema.org site and the licence as MIT. The presence of a tox.ini and a GitHub workflow named Test Suite Sanity Checking, which the badge refers to, is worth noting: it means the test data itself is checked, so a malformed test file is caught before an implementer trips over it.

For a validator implementer, the practical workflow is to add the repository as a submodule, write a runner that iterates the version directories, skipping optional/ unless your language is one it targets, and report failures by directory. The version directories are what tell you whether you have a genuine bug or a draft mismatch, which is the distinction that takes longest to work out by hand.

One thing to expect when you start: some tests will fail for reasons that are not your bug. That is the point of the suite. A specification ambiguity that your implementation resolved one way and the suite resolves another is a finding, and the README's invitation to file issues for bugs encountered in real-world implementations means such a case can be corrected at the source.

Editorial conclusion

Adopt the JSON Schema Test Suite as a conformance gate if you maintain a JSON Schema validator, since it is the closest thing the ecosystem has to an official test of correctness. Skip it if you only consume schemas, because reading it teaches you about validator behaviour rather than about writing schemas. Verify first by cloning it as a submodule and running your validator against the draft version you claim to support, and check the optional/ directory separately since those tests are not required by the specification.

Frequently asked questions

Official sources

1. Issues
2. json-schema-org/JSON-Schema-Test-Suite on GitHub
3. License: MIT
4. README
5. Releases