Newspaper4k keeps the newspaper3k API alive and adds the pieces news pipelines actually break on

A fork that promised API compatibility and delivered it

newspaper3k stopped being updated in September 2020, and Newspaper4k exists to change that. The stated goal of the fork was to keep the project alive, add features and fix bugs, and the compatibility promise is specific: all classes and methods from the original project were kept and the new features sit on top of them, so existing API calls still work as expected.

That framing matters more than any individual feature, because a scraper is embedded in pipelines that nobody wants to rewrite. If you have `import newspaper` in a codebase today, the migration is a dependency change. The project also duplicates the original open issues, and the README notes that of more than 400 issues inherited at fork time, about 180 still need verification as of version 0.9.3, with many already fixed but tedious to confirm.

The package metadata describes the goal in one line: simplified python article discovery and extraction. The topic list covers articles, crawler, datasets preparation and scraping, which tells you the intended users are people assembling news datasets rather than people building a general purpose web scraper.

One article at a time, then the metadata stack on top

The simplest entry point is a single URL. The `newspaper.article` call fetches a page and gives you an object with authors, publish date, body text, top image and a movies list, and it also carries the NLP side with keywords and a summary.

import newspaper

article = newspaper.article('https://edition.cnn.com/2023/10/29/sport/nfl-week-8-how-to-watch-spt-intl/index.html')

print(article.authors)
print(article.publish_date)
print(article.text)
print(article.top_image)
print(article.movies)

article.nlp()
print(article.keywords)
print(article.summary)

Two details in the documented output are worth pausing on. The `publish_date` comes back as a timezone aware timestamp, not a string, so a pipeline that stores it needs to handle a datetime. And `article.nlp()` is a separate call: the keyword and summary fields are not populated by the constructor, which means a bulk job that forgets to call `nlp()` on each object will produce empty keyword columns.

The summary in the documented example is a list of image captions and broadcast listings rather than a written abstract, which reflects how the summariser treats that particular page. Read the summary output for your own sources before you build an assumption about its format.

Source objects crawl a site front page and then download in threads

The `Source` class is the difference between fetching one URL and treating a news site as a publication. Building a source parses the front page, detects category links where possible, picks up any RSS feeds the site publishes and produces a list of article links. Nothing is downloaded at build time.

import newspaper

cnn_paper = newspaper.build('http://cnn.com', number_threads=3)
print(cnn_paper.category_urls())

article_urls = [article.url for article in cnn_paper.articles]

article = cnn_paper.articles[0]
article.download()
article.parse()
print(article.title)

The download step is where the cost lives. `download_articles()` fetches articles in a multithreaded fashion using `ThreadPoolExecutor` from the `concurrent` package, and the thread count is set through `Configuration.number_threads` or passed to `build()`. The README is blunt that it can take a significant time.

There is also a second path for bulk work, where articles are pulled from a site in one pass, and the documentation attaches a warning to that snippet: it could take a long time and could get your IP blocked by the news site. That warning is the single most important operational fact in the project, and the reason `number_threads=3` in the examples is a small number rather than a large one.

Google News arrives as a separate extra and a separate class

From version 0.9.3 onwards, Google News is supported through a dedicated source object rather than by scraping a Google URL. The class lives in its own module, and it takes a country, a period and a result limit at construction time, then builds a list of links.

pip install newspaper4k[gnews]

The extra exists because the feature relies on the separate Gnews package, so it is not pulled in by a plain install.

from newspaper.google_news import GoogleNewsSource

source = GoogleNewsSource(
    country="US",
    period="7d",
    max_results=10,
)

source.build(top_news=True)

print(source.article_urls())

Note the shape difference from the ordinary `Source`: `build()` takes a `top_news` flag, and the result is accessed through `article_urls()` as a callable rather than through an `articles` attribute. Code written against the plain source class will not transfer directly, which is a small but real source of confusion when both are used in the same project.

The project also keeps a README section describing an experimental ChatGPT helper bot, linked as a shared conversation for Newspaper4k. It is labelled experimental in the project itself, so treat it as a demonstration rather than a supported interface.

The CLI writes one article to a JSON file

A command line entry point ships with the package, which is the fastest way to check whether a given site will parse the way you expect before you write any pipeline code.

python -m newspaper --url="https://edition.cnn.com/2023/11/17/success/job-seekers-use-ai/index.html" --language=en --output-format=json --output-file=article.json

The flags are straightforward: the target URL, a language, an output format and an output file name. With the command as written, the result lands in `article.json` in the current directory, and the language flag is the hook for the per language processing described below.

This is the command to reach for during evaluation. Run it against two or three representative articles from your target site, open the JSON, and check whether the body text, the byline and the publish date are what your downstream model expects. Scraping problems are almost always site specific, and this command surfaces them in seconds rather than after a scheduled job has failed quietly for a week.

Language extras decide which keyword tools you get

Keyword extraction is where language support becomes concrete, and the packaging makes the choice explicit. The base install carries the extraction stack, while each language extra installs the segmenter or tagger for that script.

The extras listed in `pyproject.toml` cover Chinese with `jieba`, Thai with `pythainlp`, Japanese with `tinysegmenter`, and Bengali, Hindi, Nepali and Tamil with `indic-nlp-library`. A separate `nlp` extra pulls in `nltk`, and a `cloudflare` extra adds `cloudscraper`. The `gnews` extra follows the same pattern, installed with `pip install newspaper4k[gnews]`.

Two observations. First, the language handling is about tokenisation for keyword work, not about translating content. Second, `cloudscraper` tells you something about the operating environment: the project anticipates sites sitting behind Cloudflare protection, and the `cloudscraper` extra is the supported way to reach them. Whether you should be circumventing that protection is a decision for you and whoever owns the site, and the licence and the robots rules of the target site are the inputs to that decision rather than the presence of a dependency.

The base dependencies are ordinary and well known: `beautifulsoup4`, `lxml[html_clean]`, `requests`, `feedparser`, `tldextract`, `python-dateutil`, `brotli`, `w3lib` and `pyyaml`, each pinned to a version range. A compiled lock file also exists in `requirements.txt`, generated with `uv pip compile`.

robots.txt handling arrived in 0.9.5 and changes your crawl posture

The release notes for 0.9.5 list two things: improvements to Google News and honouring robots.txt. That second item is a behavioural change for anyone who had the old defaults, and it deserves attention rather than a passing mention.

A scraper that starts respecting robots.txt by default will visit fewer pages and will visit different pages. For a researcher building a dataset that is the correct default, since the site has expressed a preference. For a pipeline that previously collected a category listing overnight and now returns a thin set, the change is visible as a data loss bug rather than as a policy change, and the fix is to understand the new behaviour before re-enabling anything.

The related risk is the one the documentation already names: bulk downloading can get your IP blocked by the news site. Rate limits, robots rules and site terms are three separate things, and respecting the first two does not settle the third. A block also takes down every other consumer behind the same egress address, which is a detail that surprises people running shared office NAT.

The project is MIT licensed, so the main obligation is keeping the copyright and licence notice with the code. The licences of the content you extract are a different question, and the MIT grant on the software says nothing about the articles you collect with it.

Release rhythm, Python floor and what a migration costs

The last push to the repository was on 2026-08-24, and the recent releases are spread across the year. Version 0.9.6 came on 2026-07-19 with PyInstaller support, small fixes, more documentation and some reworks. Version 0.9.5 landed on 2026-02-28 with the Google News improvements and the robots.txt change. Version 0.9.4.1 on 2025-11-18 added Python 3.14 support.

The Python floor is 3.10, stated in the README and enforced by `requires-python = ">=3.10"` in `pyproject.toml`. That is a real cost for anyone still on 3.9, and it is also a sign of a modern dependency posture. The `Pillow` requirement is conditional on the interpreter version, with a different floor for Python 3.14 and above, which is the kind of detail that bites when a lock file is generated on one machine and run on another.

For a migration, the practical path is narrow. Replace the pinned `newspaper3k` dependency with `newspaper4k`, keep the existing `import newspaper` calls, then check the two places where behaviour changed: robots.txt handling and the fields that now need `nlp()` to be called explicitly. The repository layout supports a deeper look if you need one, with a `newspaper` package directory, `tests/`, an `evaluation/` directory, `stubs/`, `mypy.ini`, `pytest.ini`, a pre-commit configuration and a `CHANGELOG.md`.

The version number itself carries a signal worth respecting. The project classifies itself as a Beta under development status, and the remaining issue backlog inherited from newspaper3k has not been fully triaged, so a bug you hit may be a carry-over rather than a regression.

Editorial conclusion

Adopt Newspaper4k if you already know newspaper3k and want the same imports with a maintained dependency set, or if you need its language extras for Chinese, Thai, Japanese or Indic keyword extraction. Do not adopt it as a high volume crawler: the documentation itself warns that bulk downloads can take a long time and get your IP blocked by the news site, and the robots.txt handling added in 0.9.5 is a signal about how sites view automated access. Before you commit, run `python -m newspaper --url=... --output-format=json` against your target site, confirm the extracted text matches what you need, and check the terms of the site you intend to crawl.

Frequently asked questions

Official sources

1. AndyTheFactory/newspaper4k on GitHub
2. License: MIT
3. Project website
4. README
5. Releases