PUDL: turning US energy agency spreadsheets into one warehouse

Hundreds of gigabytes of agency files, one consistent schema

US energy data is published in forms that resist analysis. The Public Utility Data Liberation Project exists because the raw inputs are spread across spreadsheets, CSV files and databases that different agencies shape differently, and because the people who most want the numbers are often the least able to spend months preparing them. PUDL is an open source data processing pipeline that ingests those original inputs, cleans and integrates them, and writes the result to Apache Parquet files with accompanying metadata stored as JSON. The same processed data is also packaged into a single DuckDB database, and for backwards compatibility a SQLite database as well. The stated audience is broad on purpose: researchers, activists, journalists, policy makers and small businesses that could not otherwise afford access to the same data commercially. The framing is that you should spend your time on new analysis rather than on data preparation.

Raw inputs are archived on Zenodo so a pipeline can be replayed

The part of the design that makes the rest trustworthy is the raw data archive. PUDL archives every raw input it uses on Zenodo, for permanent and versioned access, and the reasoning is concrete: agencies change how they publish data and delete old files. Without an archive, a pipeline built on last year's inputs breaks, and nobody can reproduce the outputs. Each input may have several archived versions, each assigned a unique digital object identifier and made available through the Zenodo REST API. That is not a footnote, it is the foundation. Each release of the PUDL software carries a set of DOIs indicating which versions of the raw inputs it processes, and that is what lets a published figure be traced back to the exact file it came from. The archive code lives in a separate repository, pudl-archiver, and the project documentation has a Raw Data Archives section covering the details.

Parquet and DuckDB are the future, SQLite is on a deadline

The output formats tell you where the project is heading, because one of them has an expiry date attached. The fully processed data is written to Parquet files with JSON metadata, packaged into a DuckDB database, and, for backwards compatibility, into pudl.sqlite. That SQLite output is explicitly deprecated and will no longer be produced starting in 2027, and the guidance for new work is to use the Parquet outputs or pudl.duckdb. If you are starting an analysis now, a SQLite dependency is a liability with a known end date. Parquet is the better default for anything that will be read by tools such as DuckDB or a notebook engine, since it stores columnar data and does not require a running database to query. DuckDB is the convenient middle ground, since it queries the Parquet files and the single packaged database with the same SQL. The choice between them is a real decision for your project, and the project has already made it.

A nightly build refreshes the outputs without installing the pipeline

The Parquet, DuckDB, SQLite and JSON outputs, called PUDL outputs, are updated each night by an automated build process and periodically archived, and the point of that arrangement is that you can use the data without installing and running the data processing system at all. That distinction matters more than it sounds. Running the pipeline means dealing with hundreds of gigabytes of raw inputs, a Dagster home directory, a workspace on disk and the compute to transform one. Downloading the outputs means a copy of the tables. For a journalist writing a story, the second option is the only realistic one. For a researcher who wants to rebuild a table with different assumptions, the pipeline in this repository is the one you need. The outputs are described as a small file-based data warehouse containing hundreds of tables, and the data access documentation explains how to get them.

Local runs are configured through environment variables and a devcontainer

The repository carries the pieces a developer needs to run the pipeline locally. The .env.example file is the starting point, and it shows the shape of a run: three required workspace paths for inputs, outputs and the Dagster home, then logging controls, then a set of optional switches. PUDL_LOGLEVEL and PUDL_COLOR_LOGS handle logging, PUDL_LOGFILE adds a file log alongside console output, and OMP_NUM_THREADS sets the parallelism used by the CPU intensive timeseries imputation step. Several options exist for development rather than production. PUDL_FORCE_FERC_TO_SQLITE forces the full FERC to SQLite conversion even when a compatible database is already cached, and the FERC EQR related variables point at a local input archive, a deployment target config file and a nonstandard archive path for working with local data. The three required paths look like this in the example environment file.

PUDL_INPUT=/absolute/path/to/pudl-input
PUDL_OUTPUT=/absolute/path/to/pudl-output
DAGSTER_HOME=/absolute/path/to/dagster-home

The repository also contains a .devcontainer directory, which matters because the environment is defined in pyproject.toml through pixi rather than pip alone.

Python 3.14, pixi for dependencies, dbt and terraform in the tree

The packaging metadata constrains what you can run this on. The project is named catalystcoop.pudl, it is MIT licensed, and it requires Python 3.14 or later, with an upper bound that excludes the 3.15 alpha series. Dependencies are declared as empty in the project table on purpose, because they are managed through pixi and installed from conda-forge where a package exists, and the workspace targets linux-aarch64, linux-64, osx-64 and osx-arm64. The version scheme is CalVer, a year, month and patch pattern, with the tag pattern restricted to v20 prefixed tags so nightly and other tags are ignored, and a fallback of 0.0.0.dev0. The release names follow that scheme, v2026.10.0 published on 2026-10-14, v2026.9.0 on 2026-09-12 and v2026.8.0 on 2026-08-07. Other directories reveal the pipeline's structure: a dbt directory for transformation models, a terraform directory for infrastructure, a builds directory for the automated builds, plus devtools, notebooks, docs and tests.

Where PUDL is the wrong tool, and what it does not cover yet

Two boundaries are worth stating before anyone plans a project around this. The first is coverage. The list of integrated sources in the project documentation starts with EIA Form 176, and it is marked work in progress, so the set of forms and agencies covered grows over time and some datasets you need may not be there. The second is that PUDL is a batch pipeline over archived snapshots, not a live feed. If your question needs to know what an operator did ten minutes ago, the nightly build cycle is the wrong shape. There is also a cost the documentation is honest about. Local runs need a real machine, since the inputs run to hundreds of gigabytes and the pipeline needs its own disk for inputs, outputs and the Dagster home. A team that only wants the analysis-ready tables should take the outputs and avoid all of that. The alternative to PUDL in that case is the commercial databases, which the project names as the audience that cannot afford them, and which also impose their own licensing terms on derived work.

Editorial conclusion

PUDL suits a researcher, journalist or policy analyst who needs plant-level and utility-level US energy data in one consistent schema rather than a dozen agency spreadsheets, and it suits a team that wants to cite the exact raw inputs behind a number. It does not suit a project that needs live data, since the outputs are refreshed by an automated nightly build and the raw archives are versioned snapshots of what agencies published. Before you build on it, pick Parquet or pudl.duckdb rather than the deprecated pudl.sqlite, which stops being produced in 2027, and check the release DOI set for the version you install so the outputs you cite can be traced back to specific raw input versions.

Frequently asked questions

Official sources

1. catalyst-cooperative/pudl on GitHub
2. License: MIT
3. Project website
4. README
5. Releases