news-please: an integrated news crawler and article extractor

What news-please is for

Gathering news articles at scale means two jobs: crawling sites to find articles, and extracting the actual content, title, body text, authors, publication date, from pages cluttered with navigation and ads. news-please combines both into one tool that aims to work on arbitrary news websites without you writing per-site scraping rules. It can crawl a set of news sites and extract structured data from each article, and it can also read the CommonCrawl CC-NEWS archive, a large public collection of crawled news, so you can gather historical articles without crawling the live web. The audience is researchers, data scientists and developers building news datasets or monitoring coverage who want structured article data out of many sources with minimal per-site effort. It is a Python package, usable as a library or a crawler, oriented to producing clean article records as JSON.

Crawling plus generic extraction

The mechanism pairs a crawler with a generic extractor. For live sites, news-please builds on Scrapy-style crawling to discover article URLs and fetch pages, then applies extraction that pulls the article fields from the HTML without site-specific templates, using the heuristics and libraries that have proven to generalize across news layouts. For historical data, it reads the CommonCrawl news dataset directly, so you can pull articles from the archive rather than re-crawling. Output is structured, each article as a record with its title, text, authors, dates and metadata, and it can store results as JSON files or into a database such as Elasticsearch. The design goal in the project's own words is that it just works: point it at sites or the archive and get article records, rather than maintaining a scraper per publisher, which is where naive news scraping usually breaks down.

Installing and a first extraction

news-please is on PyPI and installs with pip:

pip install news-please

The simplest use is its library API for a single article, importing NewsPlease and calling it on a URL to get a structured article object, which the README demonstrates with from newsplease import NewsPlease. For larger jobs you configure the crawler with the sites to follow and where to store output, and to work with the archive you use its CommonCrawl mode. A Dockerfile and a cc-docker setup ship in the repository for containerized runs, including the CommonCrawl workflow. The first real use is calling NewsPlease on a single news article URL and inspecting the returned title, text, authors and date, which confirms extraction works before you scale up to crawling many sites or processing the archive.

Where generic extraction falls short

The limitations are inherent to generic news extraction. Because it uses site-agnostic heuristics rather than per-site rules, extraction quality varies: on unusual layouts, paywalled pages, or heavily dynamic sites, it can miss the body, grab boilerplate, or mis-parse the date and authors, so extracted fields should be validated for the sources you care about. Crawling the live web brings the usual concerns, respecting robots directives, rate limits and site terms, which are your responsibility to honor. The CommonCrawl archive is large, so processing it is a real compute and storage exercise, not a quick query. And as with any extractor, it produces best-effort structured data rather than guaranteed-correct fields. These are the trade-offs of a tool that aims to generalize across all news sites rather than being tuned to a few, and they mean you verify output quality per source.

news-please versus a bare extractor or a raw crawler

The alternatives split the two jobs. A bare article extractor such as newspaper3k or trafilatura pulls fields from HTML you already have, but does not crawl, so you supply the URLs and fetching yourself. A general crawling framework such as Scrapy fetches at scale but leaves extraction to you, so you write the parsing. news-please's difference is that it integrates both and adds CommonCrawl access, so you get crawling, generic extraction and an archive path in one tool aimed at just working. The trade is less control than assembling your own crawler plus a tuned extractor. Choose a standalone extractor when you already have the pages and want the cleanest single-purpose parser; choose Scrapy when you need custom crawling logic; and choose news-please when you want an integrated crawl-and-extract pipeline for news, especially if the CommonCrawl archive is part of your plan.

Apache-2.0 and standing

news-please is Apache-2.0 licensed, a permissive license with a patent grant, so it is freely usable including commercially, and it has academic standing with a citable DOI, which suits its heavy use in research for building news datasets. The last push was on 2026-04-14, and it is distributed on PyPI with Docker and CommonCrawl tooling in the repository. Adopt it when you need structured article data from many news sources or the CommonCrawl news archive without writing per-site scrapers, install it with pip, test extraction on a few representative article URLs to judge field quality for your sources, and honor robots directives, rate limits and site terms when crawling the live web. For historical work, plan for the storage and compute the CommonCrawl archive requires.

Editorial conclusion

Adopt news-please if you need to crawl news sites and extract structured article data, or pull historical articles from the CommonCrawl news archive, without writing per-site scraping rules. Do not assume perfect fields: generic extraction varies by layout, so validate output for your key sources, and honor robots directives and site terms when crawling live. Install it with pip install news-please, call NewsPlease on a sample article URL to check the extracted title, text, authors and date, then scale to crawling or the archive.

Frequently asked questions

Official sources

1. fhamborg/news-please on GitHub
2. Issues
3. License: Apache-2.0
4. README