unitedstates/congress, the scrapers that feed public domain legislative data

What the collectors cover, and where the data comes from

This is a community-run project to develop Python tools to collect data about the bills, amendments, roll call votes, and other core data about the US Congress into simple-to-use structured data files. The word structured is doing the work, because the raw material is a set of government websites that were never designed to be queried.

The tools list gives the coverage. There is a downloader for the official bulk bill status data from Congress, which is described as the official source of information on the life and times of legislation, and it converts that data to an easier-to-use format. There are scrapers for House and Senate roll call votes. There is a document fetcher for GovInfo.gov, which holds bill text, bill status and other official documents, and which downloads only newly updated files. And there is a defunct THOMAS scraper for presidential nominations, which the README still lists.

The data types you can request are enumerated in the README, and each links to a wiki page describing it: bills and amendments, votes, nominations, committee_meetings, govinfo for bill text, and statutes.

For practical purposes, the govinfo bulk route is the one to start with, because it is the official bulk file rather than a scrape of pages:

usc-run govinfo --bulkdata=BILLSTATUS
usc-run bills

The bills script outputs bulk data into a top-level data directory, then organized by Congress number, bill type and bill number, and two data output files are generated for each bill, a JSON version and an XML version.

The repository was originally developed by GovTrack.us and the Sunlight Foundation in 2013, and is currently maintained by GovTrack.us and other contributors. For more information about data in Congress, the README points to the Congressional Data Coalition. The documentation on the contents and the schema is in the project wiki.

Running it, and the output shape

The general form of the runner is:

usc-run <data-type> [--force] [other options]

with the data type being one of bills, votes, nominations, committee_meetings, govinfo or statutes. The console script is registered in setup.py as usc-run=congress.run:main, and there are also shell scripts for bills, statutes, votes and a voteview script.

The output is deliberately doubled. Two bulk data output files are generated for each object, a JSON version and an XML version, and the XML version attempts to maintain backwards compatibility with the XML bulk data that GovTrack.us has provided for years. There is a flag for that: add the --govtrack flag to get fully backward-compatible output using GovTrack IDs, and otherwise the source IDs used for legislators are used.

That flag is a compatibility shim for existing consumers, and it is the kind of detail that tells you the project has real users whose pipelines depend on the identifier scheme. Getting the same bill with a different identifier is not an abstract concern if you are joining against another dataset.

The caching model is described too. The script will cache downloaded pages in a top-level cache directory and output bulk data in a top-level data directory, and the --force flag applies to all data types and suppresses use of the cache for network-retrieved resources. The combination of a cache directory and a force flag is what makes it practical to re-run the collectors without hammering the government sites, and also what makes the non-determinism of scraping visible: the cache is the state of what the tool saw.

The remaining options are about operations. Debugging messages are hidden by default, and to include them you run with --log=info or --debug, and to hide even warnings you run with --log=error. To get emailed with errors, you copy config.yml.example to config.yml and fill in the SMTP options, and the script automatically uses the details when a parsing or execution error occurs. That last point is a real design choice for a scraper, since silent failures in a long collection run are how you end up with a plausible but incomplete dataset.

The project is dedicated to the public domain. The README spells it out: the project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication. All contributions to the project will be released under the CC0 dedication, and by submitting a pull request you are agreeing to comply with that waiver.

What a scraper against government sites has to survive

There is a reason the dependency list looks the way it does, and it is worth reading as a description of the problem rather than a shopping list.

The sources are government websites with no API, no rate limit contract and no stability guarantee. A bill status page is rendered from something, and when the rendering changes the scraper silently returns nothing. The bulk data route exists precisely because the bulk file is a stable contract and the HTML is not, and the README's ordering of the two commands says so: run the bulk data first, then the individual collectors.

The GovInfo fetcher is the most carefully built of the three, and its design reveals the approach. It downloads only newly updated files, which means it is tracking state across runs rather than re-downloading a corpus, and that state is what makes the tool survivable. A scraper that re-fetches everything cannot tell you what changed, and a scraper that re-fetches only what changed can.

The THOMAS scraper for presidential nominations is listed as defunct, and the honest reason is visible in the name. The README lists it as defunct rather than deleting the entry, which is itself a form of documentation.

That pattern of one target disappearing is the honest reason to treat any of this as a snapshot of how the data looked while the sources held. It is also why the caches and the force flag matter, and why the project publishes a schema and a wiki rather than just a script. A downstream user who relies on a field is relying on the documented schema, and when a source changes, the schema is the contract that survives.

The other thing the project does that a one-off script would not is test itself. The contributing section says unit tests are strongly encouraged and links an example test file, and the test runner is a single./test/run. For a project whose output is a data file that people cite, a regression in the parser is not a nuisance, it is silently wrong data.

The Python and system dependencies, and the container

The setup is documented for both Linux and macOS, and the dependency list is a good indicator of the era and the approach.

On Ubuntu you need wget, pip and some support packages:

sudo apt-get install git python3-dev libxml2-dev libxslt1-dev libz-dev python3-pip python3-venv

On OS X you need the developer tools installed, which is Xcode, and wget, which is brew install wget.

The Python side recommends a virtualenv, and the sequence is:

python3 -m venv env
source env/bin/activate

Then pip install . pulls in the Python dependencies automatically. The setup.py lists them: beautifulsoup4, cssselect, iso8601, lxml, mechanize, mock, rtyaml, python-dateutil, pytz, pyyaml, scrapelib, xmltodict and packaging. The presence of mechanize and scrapelib alongside lxml and beautifulsoup4 is the interesting part, because it shows the collectors mix session-based browser automation with plain HTTP parsing, which is what a scraper against a government site with session handling actually needs.

The Dockerfile does the same thing in a container, and its comment explains the intent: this Docker image will create a minimal environment to run the Congress scrapers in, which provides isolation from the host and allows testing in an environment that's as close to production as you can. The base is debian:bullseye, and it installs the same system packages the README lists, then pip3 installs the project, creates a symlink from python to python3 and sets a .pth file so the package is importable, with the working directory at /congress.

The documented usage shows the output directory being mounted from the host, and the comment recommends writing it to the /srv/ tree with a path such as /srv/io.unitedstates/congress/. The point of the container is that a long-running scrape is isolated and reproducible, and the data lands wherever you want it rather than inside the image.

The build and run are the two commands in the comment:

docker build --rm -t unitedstates/congress .

and then a docker run with the output directory mounted and the data type as the arguments.

For the tests, the README gives a single entry point, ./test/run, and the contributing section asks that unit tests be strongly encouraged, with a link to an example test file.

Editorial conclusion

unitedstates/congress suits a researcher, newsroom or civic technologist who needs legislative data in a structured form with a documented schema and no licence encumbrance, since the whole project is CC0 and the schema is published in the wiki. It is the wrong choice if you want current data out of the box, because the collectors run against live government sources and a scraper that works today can break when a site changes, and it is the wrong choice if you want a hosted API rather than files, because the output is bulk data on disk. Before running it, read the wiki pages for the specific data type you need, and check the bulk bill status route first, since the README suggests starting with the official bulk data before running the individual collectors.

Frequently asked questions

Official sources

1. Issues
2. License: CC0-1.0
3. Project website
4. README
5. unitedstates/congress on GitHub