datamule: a Python package for pulling SEC filings at scale

What the package does, and the two entry points you get

datamule is a Python package for working with SEC filings at scale, developed by John Friedman, and the package description adds that it incorporates the datamule endpoints, which is the hosted service behind it.

The interface is small at the top and gets deeper as you go into the examples directory. The quickstart is three lines:

from datamule import Portfolio

portfolio = Portfolio('amzn')
portfolio.download_submissions(ticker='AMZN',submission_type='10-K')

A Portfolio is constructed with a ticker, and then you ask it for submissions filtered by ticker and submission type. From there the package gives you the filing documents themselves, and the examples show what you do with them once you have them.

That separation matters for how you should read the repository. The top level listing is a .github directory, a .gitignore, the LICENSE, a contributors.md, a readme.md, and then two directories that do the work: the datamule/ package itself and an examples/ directory with about twenty scripts and notebooks.

The examples are the documentation in practice, and their names are a good index of what the package actually supports. There is parse_xbrl.ipynb and parse_fiscal_year_inline_xbrl.ipynb for the structured financial data, parse_tenk.py and parse_items_from_apple_10k.py for the narrative sections, extract_mda_from_s1.py for the management discussion and analysis, exec_comp_tables.py and convert_information_table_to_csv.py for the tables in a filing, getitem502.py for a specific item, get_8k_cik_date.py for current reports, and get_public_float_from_ticker.py.

There is also a fundamentals.ipynb notebook, a get_detected_time.ipynb, a process_data_from_websocket.py, and a construct_accession_cik_crosswalk.py, which is the kind of utility that only matters once you have filings from many companies and need to map them back to identifiers.

The rate limit, and the paid provider that removes it

The paid integrations section is the commercially significant part of the README, and it explains the reason the package exists in a form rather than as a toy.

Most of the package is free and open source, but that can be slow. For convenience, datasets built using the package have been uploaded to the cloud, with a link to the products page. The example given is the SEC archive, where filings can be downloaded without rate limits at a cost of 1 dollar per 100 thousand downloads.

The comparison in the code comment is the clearest statement of the trade-off:

# Using the SEC w/ rate limit 5/s ~= 10 days of downloads
# Using datamule ~= 1 hour
portfolio.download_submissions(submission_type='4',provider='datamule-tar')

The provider argument is what switches between the two paths. Passing provider='datamule-tar' routes the request through the hosted archive rather than the SEC directly, and the difference in wall clock time for the same job is two orders of magnitude.

The reason is the SEC's own rate limit, which the comment states as 5 per second. At that rate, a large download of historical filings takes days rather than hours, and the practical effect is that most people who want a real archive of filings give up and use a paid source. This package lets you start free and switch to the paid path only for the runs where the time cost actually bites.

So the architecture is worth being explicit about. The open source package is the interface and the local parsing, and the hosted endpoint is a data source that happens to be more convenient than the SEC for bulk work. If you never pay anything, you still get the whole package and you are limited by the SEC.

Production use, and the AWS footprint behind the service

The production section makes a claim that is worth reading carefully, because it is the kind of statement that either means the package is solid or that the author is describing his own tooling.

The claim is that the package is suitable for production, and that it can run on small machines such as an AWS t4g.nano. The t4g.nano is the smallest and cheapest Graviton instance, so that is a specific claim about the memory and CPU footprint of the parsing work, not a vague one.

Then the detail that makes the claim credible. The datamule AWS infrastructure, described as concurrent ECS Fargate instances and constantly running EC2 instances, is built with this package as its core. In other words, the same code that parses a filing on your laptop is the code that fills the hosted archive that the paid provider serves from.

That is a meaningful quality signal for a library that most people would only use in a notebook. If the code is parsing millions of filings continuously on a fleet of instances, it is not going to fall over on a slightly unusual filing, because the archive has already ingested everything the SEC has published.

It also explains the rest of the commercial model. The paid product is not a wrapper around someone else's data service. It is a hosted version of the same pipeline the author already runs, with the rate limit and the waiting removed.

The MIT licence matters here. The package is free to read, to modify and to self-host, which is what lets you run the same pipeline on your own infrastructure if you have the patience or the hardware, and the paid path exists for the people who would rather not.

What the package directory holds, and what a filing is made of

The package sits in a single datamule/ directory at the root of the repository, and the README is not specific about its internal structure, so the useful reading comes from the shape of the domain rather than from a module listing.

An SEC filing is a document with an identity and a set of documents attached. The identity is the accession number, and the documents are the individual files a company filed, which is why the package takes a ticker and a submission type and returns a set of submissions rather than a single document. The example directory names make the taxonomy visible, since they refer to specific filing forms: 10-K for the annual report, S-1 for a registration statement, 8-K for a current report, and the SEC's own item numbers within them.

The word submissions in the method name is the key to the design. A submission is the envelope, and the documents inside it are what you parse. That distinction explains why there is a construct_accession_cik_crosswalk.py example, because once you have filings from many companies you need a reliable way to map an accession back to the company that filed it, and the CIK is the identifier that makes that mapping possible.

The submission type filter is also more useful than it looks, since SEC form types are how the data is organised. Filtering on a form type rather than on a date range or a document count is the right abstraction for most research tasks, because the form determines what is inside the envelope and therefore which parser you want.

The whole set of examples is built on that same idea. parse_tenk.py and parse_items_from_apple_10k.py work on the documents of a submission rather than on the submission itself, and the fundamentals notebook works from the structured data extracted from them. So the pipeline is download submissions, pick the documents out of the envelope, and then apply a parser that knows the form.

The documentation gap, and how to work around it

The disclaimer is the most useful thing in the README for anyone deciding how to rely on this package, and it is unusually blunt.

The package almost certainly solves your use case or will solve it soon, but the docs are incomplete. The author invites you to post a GitHub issue or email him for clarification, says he replies quickly, and explains that this helps him improve the documentation.

That last clause is the part that changes how you should use the repository. The documentation is incomplete because the author is asking questions of users and writing the answers down as he goes, which means the examples directory is effectively a growing set of worked answers to questions people actually asked.

The links confirm the shape of the resource. There is the documentation site, the quickstart page, which points at a specific example, and the project website. The quickstart link goes to a path under the examples directory for a portfolio quickstart, so the documented entry point and the runnable code are the same thing.

Given the documentation gap, the examples directory is where you should actually spend your time. The names are specific enough to find the one you need. If you want the structured numbers, there are the XBRL notebooks. If you want the prose, there is the MD&A extraction from an S-1 and the Apple 10-K item parsing. If you want the tables, there is the exec_comp_tables and information table to CSV pair.

And if none of them covers what you need, the email address is the fastest route to an answer, and the author says he replies quickly. For a package with this much capability and this little documentation, that is a reasonable trade, and one the author has made explicit rather than leaving you to discover.

Editorial conclusion

Adopt datamule if you are pulling SEC filings in bulk and have hit the SEC rate limit, since that is the specific pain it addresses and the paid provider is the direct answer. Do not adopt it expecting a complete financial data library, because the repository scope is filings and their contents rather than a priced dataset. Before you rely on it, read the disclaimer, because the documentation is stated to be incomplete and the author invites issues and email for clarification, and consider the paid provider if you need volume, since the free path is rate limited to roughly 5 requests per second.

Frequently asked questions

Official sources

1. Issues
2. john-friedman/datamule-python on GitHub
3. License: MIT
4. README