How oxylabs/how-to-scrape-google-scholar turns a query into JSON

Three files wrapped around one paid HTTP endpoint

Only `README.md`, `main.py` and one banner image named Google-Scraper-API-1090x275.png sit in the repository, and that is the whole code base. Two functions in the script post a payload to https://realtime.oxylabs.io/v1/queries with basic auth, then flatten the parsed answer into a list of plain dictionaries. No HTML parsing happens anywhere in it, and nothing runs without credentials, because the parsing is done by a hosted service that has to be paid for. Registration on dashboard.oxylabs.io comes with a 1-week free trial, so the signup, not the code, is where a first run begins. One naming detail trips up anyone arriving from a search result: the repository description advertises the Oxylabs SERP Scraper API, while both the README text and the code talk about Web Scraper API and post to the realtime endpoint. Confirm which product your account covers before copying a line.

The google_scholar source replaced a URL and a custom parser

4 rows of a before and after table carry the whole migration. Older code passed the `google` source with a url parameter holding a full `scholar.google.com` link, and getting structured output meant writing your own Custom Parser instructions while depending on `requests` plus `beautifulsoup4`. Today the source is `google_scholar`, the target is a plain search term in `query`, and structured output comes from `"parse": true`. Two consequences follow. Your own selectors disappear from the project, since the vendor returns the fields instead, and the dependency list drops to `requests` alone, which is why the Requirements section asks for `pip install requests` and nothing more. A note under the same table retires the `domain` parameter for Google sources and pushes localization onto `geo_location`, so anything written against the old URL-based call has to move that key too.

Eight payload keys, and the two main.py never sends

8 keys appear in the payload table: `source`, `query`, `start_page`, `pages`, `parse`, `geo_location`, `render` and `callback_url`. Of those, 3 carry a printed default, and `parse` is the interesting one, since it defaults to `false` and that default returns HTML rather than structured JSON. `start_page` and `pages` both default to 1. Four keys describe scope rather than behaviour: `query` is the search term and is documented as UTF-8 encoded, `geo_location` sets the country the search is based on, `render` takes `html` to switch JavaScript rendering on, and `callback_url` names a URL the result is sent to. Only 5 keys reach the service from this script, which leaves `render` and `callback_url` unset, so rendering stays off and the answer comes back in the response body. A minimal request is short enough to paste by hand:

{
    "source": "google_scholar",
    "query": "global warming",
    "start_page": 1,
    "pages": 2,
    "parse": true
}

Five keys in, 6 keys out: the conditional `geo_location` line is the only route by which a 6th key arrives.

pip install requests, then edit credentials and run main.py

Setup is Python 3.8+ and one library, with no install step beyond the pip line:

pip install requests

No requirements.txt or pyproject.toml sits in the repository, so that command is the entire dependency step and somebody has to type it. Credentials are 2 constants, `USERNAME` and `PASSWORD`, next to a hardcoded `API_URL`, and basic auth carries them over the wire. The function that uses them builds the payload in a single dict:

def scrape_google_scholar(query, start_page=1, pages=1, geo_location=None):
    """Scrape Google Scholar via the dedicated `google_scholar` source."""
    payload = {
        "source": "google_scholar",
        "query": query,
        "start_page": start_page,
        "pages": pages,
        "parse": True,
    }
    if geo_location:
        payload["geo_location"] = geo_location

The request then posts that dict as JSON, with `timeout=180` on the call and `raise_for_status()` before `response.json()["results"]` hands the list back. Swap both constants for real credentials and run the file:

python main.py

The main block at the bottom calls the function with the query `global warming`, `start_page=1` and `pages=2`, so a single run asks for 2 pages, then prints position, title, authors, cited by, URL and the first PDF link whenever one exists.

How extract_articles flattens the organic array into dictionaries

A second function walks the list of results, opens the `content` object on each one and iterates the `organic` array inside it. Three lookups happen before any record is written, and the 3rd sits 1 level deeper than the other 2:

def extract_articles(results):
    """Flatten parsed results into a simple list of articles."""
    articles = []
    for result in results:
        content = result["content"]
        for item in content.get("organic", []):
            publication_info = item.get("publication_info", {})
            inline_links = item.get("inline_links", {})
            cited_by = inline_links.get("cited_by", {})

`cited_by` is not a sibling of `inline_links`, it lives inside it, and those 3 lines are where the citation count comes from. Everything downstream uses `.get` with a fallback, so a record with no title or no summary yields None rather than an exception, while `cited_by_count` falls back to 0 whenever the citing block is missing. PDF links come from a comprehension over `resources` that keeps entries matching `resource.get("file_format") == "PDF"`, so a record hosting a PDF contributes that address and one merely linking a PDF contributes nothing.

Author ids, result ids, and cite urls arrive on every record

One worked output entry shows how much identity a single result carries. It sits at position 5, reports a `result_type` value of pdf, and holds a `result_id` of `r_g8gsmjLJgJ`. Its author object carries an `author_id` of `Of8dNP0AAAAJ`, embedded in a `scholar.google.com/citations?user=` link with `hl=en` and `oi=sra`, a profile page you can follow without a second request. The `cite_url` embeds that same `result_id` in an `info:` query and adds `output=cite` with `scirp=4`, so 2 identifiers are the whole glue between a record and Scholar's own pages for it. A 2013 conference paper supplies the sample title, `Success with style: Using writing style to predict the success of novels`, hosted as a PDF at aclanthology.org, which is why `resources` holds 1 entry and `pdf_links` resolves to that address.

The 180 second timeout and what the repository leaves unstated

`timeout=180` is a ceiling high enough to tell you the vendor expects slow responses, and `raise_for_status()` turns any 4xx or 5xx into an exception. No try block, retry or backoff exists anywhere in the file, so 1 refused request ends the run. `geo_location` is added only when it is truthy, which means passing an empty string drops the key silently instead of sending a blank location.

Other gaps matter before you build on this. Nothing documents rate limits, how many `pages` 1 account may request, what a parsed call costs, or how concurrent requests behave, and the script never sets `render`, so JavaScript rendering stays off while `parse` stays on. There is no tagged build to install and no GitHub releases to pin against; the last push was on 2026-08-20.

Where the free route still wins over the hosted parser

A real alternative sits in the project's own before column, and it is not a rival product: same vendor, same account, older call. You pass the `google` source, a url holding a full scholar.google.com search address, and you write Custom Parser instructions for the fields you want. The bill does not move, but the work does, because you keep the HTML and the selectors.

For anyone not paying Oxylabs, the free route is `requests` plus `beautifulsoup4` against the same search addresses, which is exactly the pair the table drops. That route has no hosted parser, no `geo_location` switch and no `callback_url`, and layout maintenance lands on your own code. Decide by how far you push `pages` and by who absorbs the rate limits, since the parser lives inside a paid account rather than inside the 3 files in this repository.

Editorial conclusion

Adopt this guide if an Oxylabs Web Scraper API subscription already exists for other work, because for one short extraction main.py is the shortest route to clean Scholar JSON. Without that subscription, keep `requests` and `beautifulsoup4` and parse the HTML yourself. Check three things first: the price per page for the `google_scholar` source, whether a 180 second timeout fits your own retry policy, and the fact that no rate limits are documented anywhere in the repository. None of the 3 files answers those questions.

Frequently asked questions

Official sources

1. Issues
2. oxylabs/how-to-scrape-google-scholar on GitHub
3. Project website
4. README