Hail, genomic dataframes from Python over a Scala and Spark engine

A Python API over Scala, Spark and increasingly C++

Hail is an open source, general purpose, Python based data analysis tool with additional data types and methods for working with genomic data. The description sounds like a Python library with a genomic extension, and the readme is careful to say that, but the sentence that follows reframes it. Hail is built to scale and has first class support for multi-dimensional structured data, like the genomic data in a genome-wide association study.

The implementation detail matters for understanding the shape of the tool. Hail is exposed as a Python library, using primitives for distributed queries and linear algebra implemented in Scala, Spark, and increasingly C++.

So the user writes Python, and the work happens in a Scala implementation of query and linear algebra primitives running on Spark, with parts of the hot path migrating to C++. That is a three-language system, and it is a deliberate choice: Spark supplies the distributed execution that a large cohort requires, Scala supplies the performance of the core, and Python supplies the ergonomics that a researcher can work in without a compiler toolchain.

For a user, the practical effect is that Hail is not a pandas replacement with extra columns. It is a distributed execution engine with a Python-shaped interface, and the questions you would ask of a library, is this in memory, how does it scale, what does the query plan look like, have different answers than you would expect.

The documentation is versioned alongside the code, at hail.is/docs/0.2, which is a small but important detail. A genomics project that runs for years pins a documentation version, and having the docs versioned in the same scheme as the releases means the two stay in step.

Multi-dimensional data is the design, not a feature

The distinguishing claim in the readme is first class support for multi-dimensional structured data, and it is worth being concrete about what that means in a genomics context.

A genotype is not a scalar in a cell. It is a variant, an allele, a call, a genotype, a set of phased alleles, associated with a sample, a sample associated with a cohort, a cohort associated with a reference genome build. A typical analysis touches all of those levels at once, and an ordinary table model has to either flatten that structure into columns or nest it in a way that the query engine handles badly.

Hail's answer is a table of nested records with an explicitly typed schema, where the nesting is part of the data model rather than a JSON blob. Operations then work across those levels, which is what makes a stratified analysis a single expression rather than a join.

The linear algebra primitives are the other half of the story, and they are the reason Spark is in the picture. Genomic analyses are matrix operations over variant counts, and a GWAS is essentially a large linear model. The readme mentions distributed queries and linear algebra as the two categories of primitive, and the second one is not something a general purpose SQL engine gives you.

So the design is coherent: a data model that keeps the nesting, a query engine that understands it, and a linear algebra layer for the statistics. The result is a tool where a geneticist can write an analysis that is close to the mathematics of the paper, which is the goal the project states in terms of first class support for multi-dimensional structured data.

The adoption argument: gnomAD and UK Biobank

The community section makes the strongest claim in the readme, and it is a claim about usage rather than about features. Hail has been widely adopted in academia and industry, including as the analysis platform for the genome aggregation database and for UK Biobank rapid GWAS.

Both of those are large. A tool that is the analysis platform for both of them has been operated at a scale that a spreadsheet metaphor does not survive.

The readme points to a page of Hail powered science for the wider list, and the citation is the other sign of adoption: the project asks users to cite the software in published work and provides a machine-readable way to do it. The version string it emits includes the exact build, with a version and a commit hash, which is the level of reproducibility a published analysis should have.

Maintenance is a laboratory rather than a company, and the readme is explicit about it. Hail is maintained by a team in the Neale lab at the Stanley Center for Psychiatric Research of the Broad Institute, together with the Analytic and Translational Genetics Unit at Massachusetts General Hospital. The contact address is a Broad Institute mailbox, [email protected].

That structure has a practical consequence for an adopter. The project is sustained by people whose work is genomic research, so the roadmap follows the science, and the users are researchers. The acknowledgements section names the funders, the Stanley Center, the principal investigators, and the supporting organisations including the National Human Genome Research Institute and the Chan Zuckerberg Initiative, and it thanks Zulip for hosting the chat and YourKit for providing free Java profiler licences for open source development.

Continuous deployment means the version moves under you

Hail uses a continuous deployment approach to software development, which the readme says means they frequently add new features, and the mechanism for telling users is the Discussion Forum, where the readme recommends creating an account so you can subscribe to the updates.

That is a real operational difference from a library with quarterly releases. The version numbers reflect it, with the recent releases at 0.2.137, 0.2.138 and 0.2.139, which is a patch sequence that has been incrementing for a long time rather than a major version that changes the contract.

The readme also states a clear rule for citing a version, and it doubles as advice on how to pin one. The citation is obtained by executing a short piece of code:

import hail as hl
print(hl.citation())

which produces a string naming the Hail team, the version and the release tag. The example output in the readme shows a version of 0.2.13 with a commit hash, and it is tagged as a release on GitHub, so the citation points at something a reader can find.

For a research group, the sensible response to continuous deployment is to pin the exact version, record the citation with the analysis, and subscribe to the forum. A published result that was computed with a specific build is reproducible; one that was computed with whatever was on the branch that week is not.

Development discussion has two channels, and the distinction is sensible. The Zulip chatroom is for talking about development, the day to day of methods and infrastructure, and the development forum at dev.hail.is is for longer form discussions, with the readme noting that the forum is where the development happens.

The repository is a platform with services, not just a library

The root listing is the clearest statement of how large this project is. Alongside the hail/ directory itself there is auth/, batch/, ci/, monitoring/, website/, gateway/, bootstrap-gateway/, internal-gateway/, web_common/, gear/, infra/, services/, admin-pod/, datasets/, dev-docs/, devbin/, docker/, grafana/, prometheus/, tls/ and letsencrypt/.

Read as a list, that is a working platform: an authentication service, a batch job service, a continuous integration system, a monitoring stack with Prometheus and Grafana, a public website, an API gateway and its bootstrap counterpart, the shared libraries those services use, and the infrastructure definitions.

The Makefile confirms it, because it builds images for services rather than just a library. The service list is auth, batch, ci, monitoring and website, and the Makefile derives image names, database names and check targets from that list, and it also handles the special images for Hail itself, the batch worker and certificate handling, along with a set of CI images and the genetics images for Hail and the VEP containers. The Scala side is built with mill, invoked as a shell script, which is the build tool the Scala half of the project uses.

One detail in the Makefile is worth reading twice. The default target does not build anything. It prints a message saying not to use this makefile to build Hail, points at the documentation site for information on how to build Hail, and then fails. That is an unusual thing to find and a good one, because the repository is a monorepo whose Makefile is for developers, and the build instructions for users live in the documentation.

How the code is checked, and what is exempted from the check

The linting configuration in pyproject.toml is a good example of a project that has made its peace with its own history.

Ruff is configured with a line length of 120 and a target version of py310, with the comment that this should always be the lowest version of Python supported. The rule selection is F, E, W, I, PL and RUF, which is pyflakes, pycodestyle, isort, pylint and the ruff specific rules, and the ignore list is long and annotated with a reason for each entry, from ignoring bare excepts to allowing too many arguments in a function call. The formatter runs with preview mode on and quote style preserve, which keeps the existing quoting of the codebase rather than rewriting it.

The per-file ignores are the interesting part. Several directories are excluded from linting entirely with an ALL rule: the hail/src tree, benchmark, datasets, devbin, docker and query, plus the tutorial notebooks. The pyright configuration ignores the same areas and more, including the SQL directory, build and target directories, and the main hail Python package.

That is a deliberate two-tier structure, and a new contributor will meet it immediately. The modern services and tooling are strictly checked, while the older parts of the tree, which are numerous, are exempt so that a linting change does not turn into a diff of thousands of lines. The Makefile reflects the same split with a fast check target that runs ruff on the hail tree, ruff format in diff mode and pyright over the hailtop package.

Pytest is configured with a timeout of 120 seconds, which is a small but meaningful choice for a test suite that talks to a Spark cluster, since a hung job fails rather than occupying a CI slot until the job is killed.

Where Hail is the wrong tool

The first limit is the operational one. Hail runs on Spark, which means a cluster, which means the resource and the expertise. For a research group that has a cluster and a bioinformatics engineer, that is a solved problem. For a group with a laptop and a single cohort, it is a large amount of machinery for a task that a simpler tool would do, and the installation guide in the documentation rather than a pip install of a small package is the first sign of the difference.

The second is the learning curve of the data model. Hail's nested, typed table is the right model for the analyses it was built for, and it is not the model a Python user arrives with. The expressions read differently from pandas, and the debugging cycle involves reasoning about a distributed query plan.

The third is the release cadence, already mentioned. Continuous deployment with 0.2.x version numbers means the API surface moves, and a project that plans to run Hail for years has to plan for upgrades as a recurring cost rather than an occasional event.

The fourth is the fit. The readme calls Hail general purpose, with additional data types and methods for working with genomic data, and the additional part is where the effort went. The primitives are linear algebra over matrices of variant counts and a query engine for nested genomic records. A team with a different kind of large, nested, matrix-shaped data would be using the tool outside the domain its design was shaped by.

The fifth is the surrounding platform. The repository contains a full set of services, a gateway, a website, a CI system and infrastructure definitions, and that machinery is developed for the Broad Institute's own use. A small deployment should not assume it needs any of it, but a user should know that the auth service, the batch service and the website exist, because that tells you what kinds of workflow the maintainers have in mind.

Editorial conclusion

Hail fits a genomics group running genome-wide association studies or building a variant data platform, and the fact that it is the analysis platform behind the genome aggregation database and UK Biobank rapid GWAS is a stronger argument than any benchmark. It does not fit a team that wants a small Python package to import, because the engine is Scala on Spark and the surrounding platform assumes a cluster, and it does not fit a project that needs a stable release, since the project uses continuous deployment and ships frequent changes under 0.2.x version numbers. Before adopting it, read the installation section of the documentation rather than the repository readme, because the Makefile in the repository explicitly refuses to build Hail and points you elsewhere, and check the Discussion Forum, since that is where the project announces changes. The licence is MIT, version 0.2.139 was published on 2026-08-10, and the last push was on 2026-10-19.

Frequently asked questions

Official sources

1. hail-is/hail on GitHub
2. License: MIT
3. Project website
4. README
5. Releases