HDF5: a C library, a file format, and an open stewardship

A library that owns a format, not just a format that has a library

What the HDF Group calls HDF5 is two things in one repository: the source code of a high-performance library, and a file format specification implementing the HDF5 data model. Most file formats arrive in that order, a format first and a library later, and the format ends up with several partially compatible implementations. Here the steward of the format also writes the reference implementation, which is the reason the format spread so widely in science and engineering.

The README describes the model as a de facto data management standard in science, engineering and research communities. That claim rests on adoption rather than on a standards body, and the repository reinforces it with a registered media type, application/vnd.hdfgroup.hdf5, plus a DOI for citation.

The problem HDF5 solves is not file format as such. It is the problem of storing hierarchical, heterogeneous, chunked, compressed data with metadata that describes the data, and of reading parts of it without reading all of it. The data model is the point. A flat table of numbers does not need HDF5, and a project that reaches for it because the data is large, rather than because the data is complex, usually ends up paying the complexity cost for nothing.

The audience is scientific and engineering software: instruments, simulations, genomics, climate and physics pipelines. The library itself is written in C, with the surrounding directories showing where the rest of the ecosystem lives.

The C core, the high-level wrapper and the language bindings

The top-level layout shows how the library is layered and who it serves. The src/ directory holds the C core, and hl/ holds the high-level library, a convenience layer over the raw API. c++/, fortran/ and java/ carry the language bindings. HDF5Examples/ contains example programs, test/ and testpar/ hold the test suites, with the parallel variant separated out, and tools/, utils/, bin/ and config/ hold the supporting programs, scripts and configuration.

The distinction between the C core and the high-level wrapper is the one that matters most for a new user. The low-level API is a flat, capability-oriented interface that gives you exact control over datasets, datatypes and file space, and it is unforgiving about details you have to get right yourself. The high-level library wraps the same machinery in a more forgiving style, and it is the usual starting point for code that just needs to read and write.

None of that guidance comes from the repository overview, and it is the first thing a new user will have to work out. The overview points at docs/INSTALL.md for compilation and installation and docs/USING_HDF5_CMake.md for building applications against the installed library, but it does not walk through the API. Expect to make the layer choice yourself, early, and to be wrong once.

The build tooling in the repository root is the other visible part of the structure. CMakeLists.txt is the entry point, with CMakePresets.json for configure presets, and separate files for build options, installation, plugins, testing and volume handling. CTestConfig.cmake drives the test runs, .clang-format and .codespellrc set the house style for C code and spelling, and CITATION.cff plus ACKNOWLEDGMENTS handle how the work is credited.

Building from source with CMake, make or a preset

HDF5 is not installed from a package manager here, and the repository does not present a one line install command. It is built from source, and the README routes readers to separate guides depending on what they are doing. The guide set available in the docs/ directory is:

docs/INSTALL.md
docs/INSTALL_CMake.md
docs/README_HPC.md
docs/INSTALL_Windows.md
docs/INSTALL_Cygwin.md
docs/USING_HDF5_CMake.md
docs/USING_CMake_Examples.md

The first entry is the general compilation and installation guide and the documentation marks it as the starting point. The second covers CMake based builds. The last two are for the consumer side: how to build an application that uses HDF5 with CMake, and how to build and test the bundled examples with CMake.

The root of the repository already contains the CMake machinery, including CMakePresets.json, so a CMake build is the path the project itself is organised around. A .devcontainer/ directory is also present, which gives you a container with the toolchain already assembled if your host is not a Linux distribution the project expects. The presence of CMakeBuildOptions.cmake and CMakeInstallation.cmake suggests the build exposes its options as named switches rather than through hand-edited files, but the exact option names live in those files and in the guides, not in the overview, so read them rather than guessing.

For a first real use, the sensible order is: build the library following docs/INSTALL.md, build the examples with the CMake guide, then write a small program that opens a file, writes a dataset and reads it back. The examples exist precisely so that the second step is not guesswork.

Parallel HDF5 has its own configuration document

Parallel HDF5 is a build configuration that uses MPI and MPI-IO. For CMake builds, the project documents `HDF5_ENABLE_PARALLEL:BOOL=ON`; enabling it requires an MPI toolchain. Consult `docs/README_HPC.md` and the installation guide for the version-specific requirements and configuration details. The repository also maintains a separate `testpar/` directory for parallel tests.

The parallel configuration has interface compatibility constraints. The CMake installation guide says the thread-safe, C++, and Java interfaces are not compatible with the parallel option in the supported configuration; check the guide for the release being built before selecting options. Applications using parallel file access must also configure the MPI file-access driver and an appropriate communicator.

Choose parallel HDF5 when multiple MPI processes need coordinated access to HDF5 files and the workload benefits from it. Dataset size alone does not determine whether it will help: access patterns, process count and the storage system matter, so measure performance for the target workload.

The release model has no calendar, only a branch per year

HDF5 does not follow a regular release schedule. The project states that updates come from the introduction of new features and the resolution of bugs, and that the aim is at least one annual release for each maintenance branch. The README embeds a chart of past releases that have reached end of life, which is the practical version of that policy: branches end, and you need to know which one your data is on.

The recent history matches the policy rather than a quarterly cadence. Version 2.1.1 shipped on 2026-03-23, a pre-release 2.2.0-rc1 on 2026-07-24, and 2.2.0 on 2026-07-29. The repository is not archived, and the last push was on 2026-10-16.

The release tracking is unusual and genuinely useful. Badges report the next release, the latest release in the same major series, and the open issues at critical, high, medium and low priority for the version in development. The next release version is derived from the H5_VERS_MAJOR and H5_VERS_MINOR constants in src/H5public.h, annotated with the target date of the matching GitHub milestone, or TBD when no date is set. That means the version in development is a fact you can read out of a header file rather than a guess, and it tells you the project treats releases as a planned, tracked process even without a calendar.

The practical consequence for an adopter is that pinning to a branch is a decision with a shelf life. Read the end of life chart before you commit, and expect to move to a new maintenance branch periodically, with file compatibility handled by the format itself rather than by your code.

Reading the LICENSE file instead of trusting the badge

The licensing picture needs care. The README displays a BSD badge, and the GitHub metadata for the repository reports a custom licence that GitHub cannot classify. Both can be true of a licence file whose text does not match GitHub's dropdown of common licences. The rule of thumb is simple: the LICENSE file in the repository root is the authoritative text, and the badge is a summary of it.

HDF5's licence carries conditions beyond the permissive defaults, which is why a human needs to read it rather than a licence detector to classify it. Review the LICENSE file, and the ACKNOWLEDGMENTS file next to it, as part of your own review rather than relying on a badge or on this article. This is a technical read of the repository, not legal advice.

The wider governance picture is clear from the repository. The HDF Group is named as the developer, maintainer and steward of HDF5 software, and the same organisation staffs the help desk, runs the forum, publishes the documentation and now sells Priority Support with guaranteed response times, a dedicated technical account manager and other benefits. The free support path is described as community based and handled as time allows, with response times varying by issue complexity and staff availability.

That is the real trade-off in the project. The library and the format are free and open, and the organisation that owns them funds them with a mix of public and commercial support. A user who only needs the library is fine. A user who needs a contractual response time is a paying customer of the same steward, which is a business model rather than a limitation, but it is worth understanding before a deadline lands.

The Java artifacts and the C library are two different shapes of tool

The obvious alternative to using the C library directly is the Java side of the project. The repository carries a java/ directory and the README carries a section on Java Maven Artifacts, which means the same data model and file format are reachable from JVM code through published Maven artifacts rather than through a native library you compile yourself.

The difference in approach is real and practical. The C library is the reference implementation, it is what the file format is defined against, and it is the fastest path to the full feature set, including the parallel configuration. The Java artifacts give you the same on-disk format without a native build step, which is a large win for an application that already runs on the JVM and for a build pipeline where compiling a C toolchain is an obstacle. The cost is that the JVM binding is a translation of the model, and the deepest features are the ones you will reach for last.

The other alternative is not a competitor at all but a different choice of dependency: an HDF5 file can be read by the high-level library rather than the raw C API, which trades control for less boilerplate. For a scientific application that produces and consumes datasets as part of a pipeline, the more common alternative is the file format itself, used through whatever interface the language ecosystem provides, rather than HDF5 as a runtime dependency.

What makes HDF5 hard to displace is not the library quality but the file longevity. Datasets written with it are expected to stay readable across decades, and the end of life chart in the README is the maintenance policy that backs that promise.

Editorial conclusion

Adopt HDF5 when your workload is large, structured, scientific data that has to outlive a single process and a single file format, and you are prepared to build from source against docs/INSTALL.md or docs/INSTALL_CMake.md and to accept the stewardship model of The HDF Group. Do not adopt it for small datasets or when a self-describing format with zero build step matters more than a rich data model, and do not plan around a quarterly cadence, because the project states it has no regular release schedule. Before you commit, read the LICENSE file rather than the BSD badge, check the end of life chart in the README for the maintenance branch you intend to depend on, and confirm the current version constants in src/H5public.h against the release you download.

Frequently asked questions

Official sources

1. HDFGroup/hdf5 on GitHub
2. Issues
3. Project website
4. README
5. Releases