# fundus: a Python news crawler for building article corpora

> fundus is a static news crawler from Humboldt University of Berlin that extracts headlines, body text and images from live publisher sites or the CC-NEWS archive. It is aimed at corpus builders, and its publisher list, not its code, is the thing you have to check first.

**flairNLP/fundus** — A very simple news crawler with a funny name

- Repository: https://github.com/flairNLP/fundus
- Stars: 480 · Forks: 113
- Language: Python
- License: MIT
- Published: 2026-09-17 · Updated: 2026-09-17 · Language: en
- Canonical page: https://hysenlabs.com/projects/flairnlp-fundus

## The problem fundus solves: per-publisher extraction that you do not have to write

News scraping is mostly maintenance. Every publisher lays out articles differently, and a generic readability pass will pull in navigation, cookie banners or comment threads. fundus takes the other route: it ships a collection of publisher definitions, and you select the ones you want by name. The README describes it as a static news crawler that works from live websites or the CC-NEWS dataset, and the package metadata files it under web scraping and crawling keywords.

The intended user is someone building a text corpus, not someone monitoring a single site. The README's first example crawls two articles from publishers based in the US and the second narrows to The New Yorker, which tells you the unit of selection is the publisher object rather than a URL list. If your sources are already in the collection, the extraction work is done for you. If they are not, fundus gives you nothing that a plain requests call plus a parser would not.

## How the crawler is structured: publishers, sources and the CC-NEWS path

The public surface is small. PublisherCollection is the catalogue, Crawler is the live-site crawler, and CCNewsCrawler is the archive crawler. You construct a crawler with one or more publisher collections and iterate over crawl(). Each iteration yields an article object carrying a title, body text, URL, source name and a set of images, which is what the sample console output in the README shows.

The live crawler does not have one discovery mechanism. The README shows a restrict_sources_to parameter taking Sitemap, which implies articles are normally found through more than one route, and the project's topics list both sitemap and rss. Restricting to sitemaps is presented as the low-bandwidth option for large corpus creation, which is a useful signal: the default discovery path pulls more than sitemaps do.

The CC-NEWS path is architecturally different. Instead of fetching publisher pages, it reads from the CommonCrawl news crawl, and the README points at a separate tutorial before you use it. That distinction matters for planning, because the archive path depends on what CommonCrawl captured. The README states plainly that not all publishers are included in the CC-NEWS crawl, especially US-based ones. A publisher present in PublisherCollection can therefore be absent from your CC-NEWS run, and that mismatch is silent.

## Installing fundus and crawling your first two articles

The README gives a single install command and states that fundus requires Python 3.8 or newer. The pyproject.toml sets requires-python to >=3.8 and pins a dependency set that includes lxml, feedparser, requests, fastwarc and curl-cffi.

```bash
pip install fundus
```

After installation, the quick start is four lines of Python. Import the publisher collection and the crawler, build a crawler over US publishers, and iterate with a cap on article count. The README's example prints each article object.

```python
from fundus import PublisherCollection, Crawler

# initialize the crawler for news publishers based in the US
crawler = Crawler(PublisherCollection.us)

# crawl 2 articles and print
for article in crawler.crawl(max_articles=2):
    print(article)
```

What you should see is a short block per article: the number of images found, a Title line with the headline, a Text line with the opening of the body, the URL, and a From line naming the publisher and a timestamp. If the Text line contains navigation or a cookie notice, that is your first sign the publisher definition is stale for that site.

Narrowing to one publisher is the same shape with a different first argument, which is the quickest way to check whether a specific outlet works before you commit to a large run.

```python
from fundus import PublisherCollection, Crawler

# initialize the crawler for The New Yorker
crawler = Crawler(PublisherCollection.us.TheNewYorker)

for article in crawler.crawl(max_articles=2):
    print(article)
```

## Bandwidth, process count and the cost of a million-article crawl

The README is unusually direct about scale, and the numbers are worth reading before you plan anything. The crawler uses all available CPU cores by default, and the README recommends setting the processes parameter manually instead, with a rule of thumb of one process per 200 Mbps of bandwidth. That is a bandwidth-bound design, not a CPU-bound one, which shapes the machine you rent.

The reference run in the README crawls from the entire PublisherCollection in roughly seven hours on a 1000 Mbps connection with a Core i9-13905H, 64GB of RAM and Windows 11, with printing disabled. The README adds that the estimate can vary substantially by publisher and bandwidth. Treat that as an upper-bound sketch from the maintainers, not a guarantee, and note that printing to a console during a large crawl is itself a cost the example avoids.

The low-bandwidth alternative is the sitemap-restricted live crawler, which the README says requires significantly less bandwidth and is suited to large corpus creation. That is the lever to reach for when the CC-NEWS route does not cover your publishers, which, per the README, is common for US outlets.

```python
from fundus import PublisherCollection, Crawler, Sitemap

# initialize a crawler for us/uk based publishers and restrict to Sitemaps only
crawler = Crawler(PublisherCollection.us, PublisherCollection.uk, restrict_sources_to=[Sitemap])

for article in crawler.crawl(max_articles=1000000):
  print(article)
```

## Images, and the AI-training disclaimer you have to read yourself

Image extraction is on by default. The README says fundus tries to parse the images included in every crawled article, and the fourth example iterates article.images and prints each one. The article object reports an image count in its printed form, so you can spot articles where extraction found nothing.

That default has a cost. Parsing images means more work per article and more failure surface, and if you only want text you are paying for a feature you do not use. The README does not document a switch for disabling image parsing, so plan on the default or on filtering after the fact.

The larger issue is licensing, and fundus is explicit that it is not the authority here. The README carries a disclaimer stating that the indication of whether a publisher has objected to AI training on its data must be verified independently before the content is used, and points to a section on filtering publishers for AI training. Read that as the project telling you the per-publisher flag is a hint, not a clearance. The package itself is MIT licensed, per pyproject.toml and the README badge, but the MIT licence covers the crawler code. It says nothing about the articles you download, and the README does not claim otherwise.

## Where fundus is the wrong tool

The publisher list is the boundary. fundus works by having a definition for each supported outlet, and the README links a separate document listing supported publishers rather than claiming general coverage. A site outside that list is not handled by a fallback heuristic as far as the README describes, so the honest answer for an unsupported domain is that fundus is not the tool.

The second boundary is dynamic content. fundus is described as a static crawler, which means it works from what the server returns rather than from a rendered page. Sites that build article bodies in the browser are a poor fit, and the README does not present a headless-browser mode as an option.

The third is scope. This is an article extractor, not a site mirror. It returns title, text, images, URL and source. If you need comments, structured data, or a full HTML archive, you are outside what the examples show, and the README does not document those fields.

Finally, CC-NEWS coverage gaps are a real failure mode rather than an edge case. The README states that not all publishers are in the CC-NEWS crawl, especially US ones. A run that returns far fewer articles than expected may be a coverage problem in the archive, not a bug in your code.

## Alternatives: general-purpose extractors and hand-written scrapers

The obvious alternative is a readability-style extraction library, typically paired with requests or a headless browser. The difference in approach is where the site knowledge lives. A readability extractor applies one heuristic to any page and hopes the main content wins. fundus moves that knowledge into per-publisher definitions, so accuracy on a supported site is a matter of whether the definition is current, and accuracy on an unsupported site is not attempted. That trade is the whole design.

The second alternative is writing your own scraper per outlet. That gives you full control over fields and update timing, and it is the right call when you need one site and need it exactly. fundus pays off in the opposite situation: many publishers, a standard article shape, and a willingness to depend on someone else's definitions staying current.

A third option is to take a ready-made corpus rather than crawl. If your research question tolerates an existing dataset, crawling is work you can skip entirely. fundus is for the case where you need a specific publisher mix or a specific time window that no packaged corpus gives you.

## Maintenance, releases and what a version bump costs you

The repository is not archived, and the last push was on 2026-09-14. The most recent release listed is v0.5.7 on 2026-08-03, following v0.5.6 on 2026-05-05 and v0.5.5 on 2026-02-02. That cadence is roughly quarterly, which fits a project whose main ongoing work is keeping publisher definitions aligned with live sites. It also means a broken publisher can sit until the next release unless you patch it yourself or track master.

Upgrade cost is mostly dependency churn, and pyproject.toml shows the shape of it. Dependencies carry upper bounds, including fastwarc>=0.14, <0.16, and a comment explains the ceiling: from version 0.16, FastWarc requires Python 3.10 or newer. That is the kind of constraint that decides your interpreter version for you. If you are pinned to Python 3.8 or 3.9, the fastwarc ceiling is what keeps the install working, and lifting it later is a Python upgrade, not a one-line change.

On licensing, the code is MIT, which is permissive and imposes no obligations on your own source. The content you crawl is a separate question, and the README's disclaimer directs you to verify publisher positions independently. That is a factual statement about what the project does and does not assert, not legal advice.

## Conclusion

Adopt fundus if you need article text plus images from named publishers and want to write Python instead of maintaining per-site selectors. Do not adopt it if your sources are not in the publisher list, or if you are crawling a site whose terms you have not read. Before committing, open docs/supported_publishers.md and confirm your domains are covered, then run the two-article example against one of them and inspect the extracted text yourself.

## FAQ

### How do I install fundus and crawl my first articles?

Install with pip install fundus, which requires Python 3.8 or newer. Then import PublisherCollection and Crawler, build a crawler over a collection such as PublisherCollection.us, and iterate over crawl(max_articles=2), printing each article.

### What is fundus?

It is a static news crawler written in Python and developed at Humboldt University of Berlin. It extracts article titles, body text, URLs, source names and images, either from live publisher websites or from the CC-NEWS dataset.

### Which publishers does fundus support?

The README links a separate document listing supported publishers, and the crawler is built around per-publisher definitions rather than a general heuristic. Publishers outside that list are not covered by the examples shown in the README.

### Can fundus crawl a million articles?

The README shows a CCNewsCrawler example with max_articles=1000000 reading from the CC-NEWS archive, and reports a reference run of about seven hours on a 1000 Mbps connection with printing disabled. It notes that not all publishers are included in CC-NEWS, especially US-based ones, so a sitemap-restricted live crawl is the stated low-bandwidth alternative.

### Does fundus extract images as well as text?

Yes. The README states that fundus tries to parse the images included in every crawled article by default, and the fourth example iterates article.images and prints each one.

### Can I use the articles fundus crawls to train an AI model?

The README carries a disclaimer stating that its indication of whether a publisher has objected to AI training must be verified independently before the content is used. The MIT licence covers the crawler code, not the articles you download.

## Sources

- [flairNLP/fundus on GitHub](https://github.com/flairNLP/fundus)
- [Issues](https://github.com/flairNLP/fundus/issues)
- [License: MIT](https://github.com/flairNLP/fundus/blob/master/LICENSE)
- [README](https://github.com/flairNLP/fundus/blob/master/README.md)
- [Releases](https://github.com/flairNLP/fundus/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/flairnlp-fundus