ASDF and the file format for astronomy data with readable metadata

What ASDF is and what the package provides

The Advanced Scientific Data Format is described as a next-generation interchange format for scientific data, and the repository holds the Python implementation of the ASDF specification. The specification itself lives in separate documentation at asdf-standard.readthedocs.io, which is the distinction to hold on to: the format is a standard with an independent document, and this repository is one implementation of it.

The feature list in the README is short and each item maps to a real design decision. Hierarchical and human-readable metadata in YAML format. Efficient binary array storage with support for memory mapping and flexible compression. Content validation using schemas, using JSON Schema. And native and transparent support for most basic Python data types, with an extension API to add support for any custom Python object.

The audience is a scientist or pipeline developer working with array data, and the repository topics point at the domain precisely: astronomy, astropy, jwst. JWST in particular is the kind of instrument whose data products are large enough that the difference between a self-describing text header and an opaque binary blob stops being a preference.

The package is on PyPI as asdf, the licence is BSD-3-Clause, and the pyproject.toml marks the development status as Production/Stable with Python 3.11 or newer required.

There is also a command line tool, asdftool, exposed as a script entry point in the project metadata, so the format is not limited to the Python API.

The metadata tree and the binary blocks below it

The mechanism that makes ASDF different from other array formats is visible in a single file. The metadata section is a YAML document with a header, and the array data is not in that document at all. It is stored as binary blocks below the metadata section.

The README makes the point explicitly: the numerical array data is not stored in the metadata tree itself, instead it is stored as binary data blocks below the metadata section. That is the whole design in one sentence.

The YAML header looks like this, with the version comments and the YAML directives at the top:

    #ASDF 1.0.0
    #ASDF_STANDARD 1.2.0
    %YAML 1.1
    %TAG ! tag:stsci.edu:asdf/
    --- !core/asdf-1.1.0
    asdf_library: !core/software-1.0.0 {author: The ASDF Developers, homepage: 'http://github.com/asdf-format/asdf',
      name: asdf, version: 2.0.0}

Two things are worth reading there. The first two comment lines record both the format version and the version of the standard the file conforms to, which is what lets a reader know whether it understands the file. The YAML directives set the YAML version and declare the tag prefix, so the custom tags in the document are namespaced rather than free-floating.

Each array in the tree is a tagged node rather than an inline value, for example a squares entry with a source index, a datatype, a byteorder and a shape. The source number is the pointer to the binary block that holds the bytes, and the shape and datatype say how to interpret them.

The payoff is that the header is a tree that mirrors the structure of the data you stored, so what you read with a text editor looks like the dictionary you wrote. That readability is the point of the format, and it is also what makes the format inspectable by a human during an incident.

Writing a file, and the compression option

Creating a file means building a plain Python dictionary and handing it to the AsdfFile constructor. From the README:

    import asdf
    import numpy as np

    # Create some data
    sequence = np.arange(100)
    squares = sequence**2
    random = np.random.random(100)

The dictionary can nest arbitrarily, so the structure of the file is the structure of the tree you built, and the AsdfFile object is constructed from that tree and then written out with write_to.

Compression is applied to the array data at write time, not to the metadata:

    af.write_to("compressed.asdf", all_array_compression="zlib")

The built-in compression algorithms are zlib and bzp2. The lz4 algorithm becomes available when the lz4 package is installed, and the README notes that other compression algorithms may be available via extensions. That last point is a small illustration of the extension mechanism doing real work: the set of compressors is not fixed by the implementation, and the optional extras in the project metadata follow the same pattern, with an lz4 extra and an http extra listed separately.

Compression applies to all arrays at once through that one keyword, which is convenient for a pipeline writing a homogeneous set of arrays and blunt for a file that mixes a compressible array with an incompressible one. The finer control is not described in the README, so anyone needing per-array settings should look at the API documentation rather than assume.

Reading, inspecting and the lazy array behaviour

Reading uses the top-level open function of the asdf package, and it doubles as a context manager:

    with asdf.open("example.asdf") as af:
        ...

The object you get back behaves like a Python dict, and nodes are accessed like any other dictionary entry, so af["name"] returns a value and af["powers"] returns a nested mapping.

The behaviour worth understanding is what happens with the arrays inside. The README states that array data remains unloaded until it is explicitly accessed, and the dictionary view shows the array as unloaded with its shape and dtype:

    >>> af["powers"]
    {'squares': <array (unloaded) shape: [100] dtype: int64>}

Only when you reach into the array does the data get read, and the result is a real numpy array. The README verifies the round trip by comparing against a plain Python list of squares and confirming the comparison is True.

That lazy behaviour is what makes memory mapping worthwhile. An instrument pipeline that wants to look at one array out of a file that holds many does not pay for the whole file, and a consumer that needs random access into a huge array can do it without loading it into memory. It also means that the shape and dtype information in the header is available before any data is touched, which is what lets a tool plan a read.

For a quick overview rather than individual access, the AsdfFile.info() method prints a tree of the file's contents with types and values, showing the root as an AsdfObject, the software and extension history recorded in the file, and each array with its shape and dtype. That tree view is also the fastest way to see what a colleague actually handed you.

Validation, extensions and the entry points

Content validation is one of the four headline features, and it works with JSON Schema, applied to the metadata. A file that does not match the schema for its content is not a valid ASDF file, and the schema can be extended by the extensions an installation registers.

The extension mechanism appears in three places in the project metadata, which is how you can tell it is a first-class part of the design rather than an afterthought. There is an asdf_extensions entry point registering the builtin extension, an asdf.extensions entry point for discovery, and an asdf.resource_mappings entry point that provides the JSON Schema resource mappings used for validation. The metadata of a written file also records the extensions in use, under a history key, so a reader can see which extension produced a given node.

That history is not incidental bookkeeping. When a file is read by a different installation, the recorded extension list tells the reader what it is dealing with, and the schema validation is what catches the case where the data is structurally wrong rather than merely unfamiliar.

The native support covers most basic Python data types, and the extension API is how a custom Python object gets written and read back as something meaningful rather than an opaque pickle. For a project with a bespoke data type, that extension point is the whole reason to pick this format over a generic container.

The dependency list in pyproject.toml is worth a look for the mechanism rather than the versions. Alongside asdf-standard, numpy, pyyaml, jmespath and semantic_version, there is a block commented as being for the vendorized jsonschema, with attrs and an end marker. Bundling the JSON Schema implementation means validation does not depend on what happens to be installed in the wider environment, at the cost of carrying a copy of that code.

The ecosystem, the tooling and the cost of adoption

The repository is a working Python project rather than a script collection, and the layout shows the investment. There is an asdf/ package directory, an integration_tests/ directory, a benchmarks/ directory, a changes/ directory for the changelog fragments, a docs/ directory, a licenses/ directory, and a noxfile.py for session management. The pyproject.toml lists test, benchmark, docs and typing extras, plus a dependency group for S3 integration testing using boto, fsspec and moto.

That last group tells you something real about scope. The http extra pulls in fsspec[http], which means reading an ASDF file from a remote location is a supported configuration rather than a hack, and the S3 test group exists to exercise that path against a mock. For pipelines that stage data in object storage, that is a feature worth knowing about before you assume ASDF is a local-file-only format.

The tooling around the project is also mature. The CI badge points at a ci.yml workflow, coverage is tracked on Codecov, the package is archived on Zenodo with a DOI badge for citation, and the docs are hosted on Read the Docs. The pyproject.toml pins pyrefly for the typing extra, and a.pyrefly-baseline.json file at the root shows the type checker is run against a recorded baseline rather than ignored.

The cost of adoption is therefore mostly in your dependencies rather than in the format itself. Python 3.11 or newer, a numpy version of 1.22 or later, and PyYAML 6.0 are hard requirements, and the version 6.0.0 release on 2026-10-19 sits alongside 5.4.0 from 2026-09-04 and 5.3.1 from 2026-06-08, so the major version moves with some regularity. A major bump is the point at which the standard version recorded in files you write may also move, and that is why the header carries both numbers.

Editorial conclusion

ASDF is a good fit for a pipeline that produces large instrument arrays and needs to carry its metadata with the data in a form a human can read and a machine can validate, because the YAML header plus separate binary blocks gives you both inspectability and memory-mapped access to the arrays. It is a poor fit if you need a format every tool already reads, since FITS and HDF5 remain the safer interchange choices, and a poor fit if you need the array bytes interleaved with the metadata, because the split is the design rather than an option. Before adopting, check the ASDF standard version your readers support, since the file header records both the ASDF version and the ASDF_STANDARD version, and confirm the extension ecosystem for your instrument types, since custom objects are supported through an extension API rather than built in.

Frequently asked questions

Official sources

1. asdf-format/asdf on GitHub
2. License: BSD-3-Clause
3. Project website
4. README
5. Releases