juriscraper: metadata extraction from American court websites

Metadata only, and the caller does the rest

Juriscraper is described as an API to scrape American court websites for metadata, and that word choice is the most important thing about the design. It is not a document archive and it is not a search engine. It is a scraper library that gathers judicial opinions, oral arguments and PACER data, and it returns metadata about them.

The architecture is explicitly two-part. The second part is your code, which calls Juriscraper, and that code is responsible for calling a scraper, downloading and saving the results. A reference implementation of the caller is in use at CourtListener.com, and the repository also ships a basic sample caller, sample_caller.py, which the documentation says can be used for testing or as a starting point when developing your own.

The separation is deliberate and it has consequences. One of the stated design goals is no need for a database, and another is to return all metadata available on court websites so the caller can pick what it needs. Juriscraper decides what to fetch and how to parse it; you decide what to keep, where to put it, how to schedule the calls and what to do when a site changes. For a project building a searchable corpus, that is the right split, because the storage and deduplication decisions are the ones your project has opinions about.

There is a second design goal that deserves attention: friendly as possible to court websites. That is an unusual thing for a scraper to optimise for, and it reflects the fact that the source sites are public infrastructure.

What the scrapers actually cover

The documented coverage is stated as a list. Juriscraper can scrape a variety of pages and reports within the PACER system, opinions from all major appellate Federal courts, opinions from all state courts of last resort except for Georgia, typically their Supreme Court, and oral arguments from all appellate federal courts that offer them.

The Georgia exception is the kind of detail that tells you the list was assembled empirically, one court at a time, and that it is accurate as of the documentation rather than aspirational. Read the list as a description of what has been built and tested, not as a promise about the whole court system.

The PACER coverage deserves a separate note, because PACER is the federal case management system and it is the most heavily guarded source in the list. A PACER account is required to reach much of it, and access carries per-page fees. The repository topics list pacer, courts, government and scraping, which tells you where the difficulty is concentrated.

The design goals also mention extensibility to support video and oral argument audio, and extensibility to support geographies such as US, Cuba, Mexico and California. Those are stated as goals for the architecture rather than as shipped features, and they are the reason the project is described as having a generalized architecture with minimal code repetition. A scraper library that is going to be extended to a court in Cuba has to be structured very differently from a set of one-off scripts.

XPath and lxml, and the mime type detection layer

The scraping technique is stated directly: XPath-based scraping powered by lxml's html parser. That is a deliberate choice over regular expressions, and it is the right one for court sites, which are often generated by document management systems with deeply nested tables and inconsistent markup. XPath lets a scraper select on structure, and lxml's HTML parser is forgiving of the malformed markup that real court sites contain.

The dependency list confirms the modern stack. pyproject.toml requires lxml>=6.1, plus html5lib as a second parser, httpx with http2 support for the HTTP client, feedparser for RSS, dateparser and python-dateutil for the date handling that court metadata demands, pydantic>=2.13.5 for validation, chardet and charset-normalizer for character encoding detection, nh3 for HTML sanitisation, cssselect for CSS-to-XPath translation, geonamescache for place names, and certifi for certificate authorities.

The mime type identification through magic numbers goal explains the chardet and charset-normalizer entries. Court documents arrive as PDF, as Word files, as HTML, as plain text and as audio, and the library detects what a file really is by its leading bytes rather than by what the server or the extension claims. For a corpus builder, that matters, because a misidentified type produces a broken extract downstream.

The dependency on pydantic is a more recent addition and tells you the library now validates its own output rather than handing callers loose dictionaries.

Installing it on Linux, macOS and Windows

The documented install starts with Python 3.11 or newer, and the project metadata confirms that floor with requires-python set to >=3.11, with classifiers for 3.11 through 3.15 and for both CPython and PyPy.

There are native dependencies to satisfy first, because lxml and the XML and YAML libraries need system headers. On Ubuntu based distributions and Debian Linux the documentation gives:

    sudo apt-get install libxml2-dev libxslt-dev libyaml-dev

On Arch based distributions the equivalent is:

    sudo pacman -S libxml2 libxslt libyaml

And on macOS with Homebrew:

    brew install libyaml

After that, the library itself is a normal pip install:

    pip install juriscraper

One configuration detail is worth setting before you run anything, and the documentation is unusually specific about it. You can set an environment variable for where you want to stash your logs, and the default is /var/log/juriscraper/debug.log if that path exists on the filesystem. Setting it explicitly is the better choice on a machine where that path does not exist, and it matters operationally because the project lists clear log levels, DEBUG, INFO, WARN and CRITICAL, and a scheduled scrape that starts failing at three in the morning is only diagnosable if the log went somewhere you chose:

    export JURISCRAPER_LOG=/path/to/your/log.txt

Running a first scrape with the sample caller

The first real use of the library is the sample caller, sample_caller.py, which the documentation says can be used for testing or as a starting point when developing your own. Running it against a single court before writing any pipeline is the right sequence, because it tells you two things at once: whether the court site is reachable from your network and whether the current scraper still matches the current markup.

The second step is the reference caller used at CourtListener, in cl/scrapers/management/commands/cl_scrape_opinions.py in that repository. Reading it tells you how a production system organizes the calls, and it is the more useful of the two files for anyone who intends to run this on a schedule, because the sample caller is explicitly a starting point rather than a finished scheduler.

Two operational points follow from the design goals and the repository layout. The project has no database, so somewhere in your code the saving happens, and a corpus builder will want idempotent writes and deduplication on document identifiers, which the metadata is designed to support since the caller can pick the fields it needs. And the log levels are meant to be used, so a scraper that starts returning fewer documents than expected should be detectable through the logs rather than through a silently smaller database.

The repository also has a tox.ini, a .pre-commit-config.yaml, a CHANGES.md and a tests/ directory, and the contributing guide points to it for instructions on setting up an environment and running tests. The pyproject.toml also lists a pyrefly baseline and a pyrefly.toml, so the project has a type checker in its toolchain alongside the pre-commit configuration.

Release history, licence and the fragility you inherit

The version numbers in the project metadata and the release tags in the repository do not line up in the way a casual reader would expect. pyproject.toml declares version 4.0.1, while the listed releases are 1.4.15 from 2017-03-21, 1.4.3 from 2017-02-15 and 1.3.2 from 2017-01-27, the last of which is named for a new scraper, ohioctcl_beginningofyear. The repository is not archived and the last push was on 2026-10-22, so the code is being worked on, and CHANGES.md is the file that records what changed.

The practical guidance is to treat the changelog, not the tag list, as the release history, and to check it before upgrading, because the scraper set is where behaviour changes.

The licence is the permissive BSD licence, specified as BSD-2-Clause in pyproject.toml with the licence text in the LICENSE file, and the README describes it as the permissive BSD license. For a project that fetches public court documents, that licence choice matters in a practical way: you can build a commercial legal data product on it.

The fragility is inherited by design. A scraper library for court websites is a collection of assertions about third-party HTML, and every one of those assertions can be invalidated by a site redesign without warning. The mitigation is the extensibility goal, the generalized architecture and the test suite, but a user of this library should plan on scrapers breaking and budget for reacting to that, rather than treating the library as a stable API over stable upstreams.

Where Juriscraper is the wrong tool

The first case is a one-off question. If you need the docket for a single case, running a scraping library and building a storage pipeline is absurd overhead, and a direct fetch of the court site or a paid research service is the answer.

The second case is documents rather than metadata. Juriscraper gathers opinions, oral arguments and PACER data and returns metadata about them. If you need the full text of every opinion as a searchable archive, this library gives you the index and leaves the document acquisition to you, which is a substantial second project involving PDF extraction, OCR quality issues and the per-page cost of PACER.

The third case is a pipeline that cannot tolerate upstream change. Court sites redesign. A parser breaks. This library will not prevent that, and while the sample caller and the test suite give you a way to detect a breakage, you still need an alerting story. Teams that need continuous court coverage with a contractual service level should look at a commercial provider instead, where the scraping is somebody else's problem.

The alternative worth understanding is not another scraper library, it is a different strategy for acquiring the same data. Some court systems publish bulk data or RSS feeds, and the library depends on feedparser for exactly that reason, so a feed-based path exists for at least part of the coverage. Where a feed exists, subscribing to it is more reliable and cheaper than scraping the HTML behind it. And where a commercial bulk source exists for a court, the right approach is to take the data rather than to re-derive it. Juriscraper is strongest in the gap it was built for, the courts where no machine-readable feed exists, and it is worth using precisely there rather than everywhere.

Editorial conclusion

juriscraper is worth adopting for a legal data project that needs court metadata rather than court documents, especially if you plan to run it on a schedule and store the results in your own system, since the library deliberately has no database and returns everything the site exposes. It is the wrong tool for a one-off lookup, for anyone who needs the document files as well as the metadata, and for a production pipeline that cannot tolerate a site redesign breaking a parser. Before building on it, run the included sample_caller.py against one court to see the shape of the returned data, read the CHANGES.md file for the scrapers added since the release tags, and set JURISCRAPER_LOG to a path you control so a failing court scrape leaves a trace rather than disappearing.

Frequently asked questions

Official sources

1. freelawproject/juriscraper on GitHub
2. License: BSD-2-Clause
3. Project website
4. README
5. Releases