oxylabs/google-news-scraper: the documented variable name is not the one the Makefile reads

The tutorial passes a variable the Makefile never reads

The scraping step is documented as one command, and the command does not work as written. The text says to run make scrape with a topic identifier passed as TOPIC_ID, and gives a full example with a long identifier after an equals sign. The build file checks a different name:

scrape:
	@if [ -z "$(TOPIC)" ]; then \
		echo "Error: An ID of a topic to scrape is required. Use make scrape TOPIC=<topic_id>"; \
		exit 1; \
	else \
		poetry run python -m google_news_scraper --topic=$(TOPIC); \
	fi

Passing TOPIC_ID leaves TOPIC empty, the guard fires, and the error message tells the reader to use TOPIC. So a reader who follows the tutorial exactly is corrected by the tool, which is the right behaviour for a guard and the wrong behaviour for documentation. Everything else in the flow is consistent: the guard, the error text and the command that runs the package all use the same name.

Installing pins Poetry to one exact release through pip

The install target is two lines, and the first one installs the build tool before using it:

install:
	pip install poetry==1.8.2
	poetry install

So a contributor without Poetry gets a single exact release, not the one already on their machine or the current one, and the repository also carries a lock file and a build backend declaration for Poetry. That combination is self-consistent for reproducibility and awkward in practice: pip installs into whatever environment is active, so a developer with a newer Poetry elsewhere on the machine now has two, and the pinned one is the one this target will use. The second line then resolves the project's dependencies from the lock file, which is the part that makes the run reproducible. The prerequisites section separately asks for Python 3.11, while the project file asks for 3.11 or newer.

One dependency is pinned to a 0.0 shim while the rest are real ranges

Nine dependencies are declared and eight of them use a caret range against a real version: the async HTTP client, the validation library, the command line parser, the dataframes library, the browser automation library, its driver manager, and the settings companion to the validation library. The ninth is different:

[tool.poetry.dependencies]
python = "^3.11"
bs4 = "^0.0.2"
httpx = "^0.27.0"

The package named bs4 is a shim distribution whose own version has never left the 0.0 series and whose job is to pull in the real parser. A caret constraint on 0.0.2 resolves to that patch and nothing else, so the declared dependency can never float forward the way the others can. It works, because the shim's job is to exist rather than to be used directly, but it is the one line in the list a reader has to think about, and the honest declaration would be the library the code actually imports.

Two fetching stacks are installed and only one is documented

The declared dependencies contain two unrelated ways of fetching a page: an HTTP client with a version range, and a browser automation library with a driver manager that downloads browser binaries on demand. The documented workflow uses neither explicitly. It describes copying a topic identifier out of a browser address bar, passing it to the command, and getting a CSV file, which reads as a plain HTTP request against the news endpoint. So one of the two stacks is doing work the documentation does not account for, or one of them is left over from an earlier approach. The distinction matters for anyone running this on a machine, because a driver manager will fetch and execute a browser binary the first time it runs, which is a considerably larger action than making an HTTP request.

The output file lands wherever you happened to be standing

There is no output path in the interface. The command takes one variable, the topic identifier, and the documentation says that when the tool finishes you will find a file named articles.csv in the directory you were running the tool from. That is a fixed filename written into the current working directory, so running the tool twice from two different directories produces two files with the same name in different places, and nothing in the repository lets you redirect or name the output. For a tool intended to be run repeatedly while tracking topics, that is the constraint most likely to annoy, and it is the kind of detail that a single output argument would fix.

The news product is the search endpoint with one context flag

The commercial half is documented with a request sample, and the sample is more informative than the prose around it. It posts to a queries endpoint on a realtime hostname, authenticates with a username and password pair, and sends a payload whose source is the general search target, whose query is a search term, and whose parse flag asks for structured output rather than markup. The news specificity is one entry in a context list:

payload = {
    'source': 'google_search',
    'query': 'adidas',
    'parse': True,
    'context': [
        {'key': 'udm', 'value': '12'},
    ],
}

So the news API is the search API with a vertical selector, which is worth knowing before you model it as a separate product. Three output formats are offered, structured data, markup, and markdown, and the trial is described as up to two thousand results.

The declared homepage is an affiliate link with a transaction id

The homepage recorded for the project is not a documentation page. It is the vendor's product page with campaign parameters appended: a source number, the word affiliate, a group identifier, a content slug naming this repository, and a transaction identifier that looks like a per-visitor token. The same URL is the first thing in the readme, as the banner link above the title, and it appears again as the destination of the badge row. So the three places a reader is sent first are one tracked link rather than the documentation. The repository itself is otherwise lean: a lint configuration, a pre-commit configuration, the Makefile, the readme, the lock file, the project file and a source directory, with a single image committed at the root under a filename that contains spaces.

Editorial conclusion

Read the terms question first, because it is the one neither half of this repository answers. Scraping a public news index is ordinary work, and this repository makes no attempt to state what the news platform permits, how long its data may be kept, or what a compliant request rate looks like. That judgement is yours and it belongs before any code runs. Three practical points follow. The documented make variable does not match the one the build file reads, so follow the error message rather than the tutorial. The free tool is the small half: a topic identifier, an HTTP call, a CSV in your current directory, with a browser automation stack carried in the dependencies for a workflow the documentation never describes. And the second half is a commercial product, so if you take the free path for evaluation you have not evaluated the paid path.

Frequently asked questions

Official sources

1. Issues
2. oxylabs/google-news-scraper on GitHub
3. Project website
4. README