chatgpt-scraper documents a hosted endpoint rather than shipping a library

The repository holds documentation and nothing else

The top level listing is short enough to enumerate. There is the readme, three image files whose names describe them as a repositories banner and two product graphics, one example output file, and a directory of code examples for other programming languages. There is no source directory, no manifest, no test suite and no release history. The repository is recorded as a Java project, which reflects the language of the samples in that examples directory rather than anything that compiles. The licence field reads as unknown, and there are no tagged releases, so the project is versioned by whatever the vendor's own service happens to be doing. Read as a whole it is a landing page for a commercial product, which is why the readme carries affiliate parameters in its very first link.

import requests
from pprint import pprint

# Structure payload.
payload = {
    'source': 'chatgpt',
    'prompt': 'best supplements for better sleep',
    'parse': True,
    'geo_location': "United States",
    'callback_url': "https://your-server.com/oxylabs-callback"
}

One POST to one endpoint with seven parameters

The interface is a single POST to a versioned queries endpoint with basic authentication taking a username and a password. Two parameters are mandatory. The source parameter sets the scraper target and the documented value for this product is chatgpt. The prompt parameter carries the text, capped at 4000 characters, and the table describing it contains a typo that drops a letter from the word prompt. Five more are optional. search turns on web search and defaults to false. parse asks for structured results and also defaults to false. geo_location names a country to route the request from. callback_url is an endpoint of your own to receive results at. browser_instructions takes custom instructions for the headless browser, which is the parameter that exposes how the page is being driven rather than merely fetched.

parse is what turns an answer into an envelope

With parsing enabled the response arrives as JSON with a results array, and each element carries a job identifier, a status code, the URL that was visited, and a content object. Inside content the submitted prompt comes back, along with the specific model that produced the answer and two renderings of the response: a plain-text version and a Markdown version. A third field, markdown_json, holds a structured representation of that Markdown, where each item has a type and a children array. That last field is the one to reach for when the answer has to be fed into another tool rather than displayed, because it turns prose into something traversable without a second parse. The readme points at a complete example output file in the repository, and it also notes that a Markdown extraction is available for integration workflows involving AI tools.

Six fields only exist when the answer contains them

The response table marks several rows as conditional, returned only when that content is in the model's response, and they are the fields that make the payload unpredictable in shape. A citations array carries source references, each with a title, a url, text, a description and a section. A search_queries array holds the queries the model used, as plain strings. A links array holds inline source details as a url and a text pair. A shopping_products array carries product details including price, title, rating, currency, a formatted price string and a thumbnail. An ads array carries url, title, image_url, description and a nested advertiser_info object that itself holds a url, a name and an image_url. Separately, parse_status_code is documented with one value that means success and anything else meaning the parser failed to extract some or all structured fields.

The shape varies with the device and with the prompt

Two separate caveats in the documentation warn that a caller cannot assume a fixed response. One says the composition of elements may vary depending on whether the query was made from a desktop or a mobile device, which means the vendor does not fully control which surface answers and therefore does not guarantee the same field set twice. The other says the number of items and fields for a specific result type may vary depending on the submitted prompt. Taken together with the six conditional fields, that is three independent sources of variation in a single response, and a consumer that indexes into the JSON by position rather than by key will break. The readme also labels a JSON code block as an HTML example, which is a labelling slip rather than a difference in the data.

The feature claims are all about not being stopped

The section arguing why to choose the vendor is built entirely on the infrastructure the buyer would otherwise have to run. It claims the API handles proxy management, IP rotation and bot traffic management systems, so no engineering time goes to maintenance or adapting to website changes. It claims high success rates and a consistent data flow on that infrastructure. And it names the advanced features as custom browser instructions for a headless browser to emulate real user behaviour, automatic handling of CAPTCHAs, and geo-targeting for localised responses. Read plainly, those three are the reason the browser_instructions parameter exists and the reason the geo_location parameter is offered, and they are also what makes the terms of the target service the relevant question for anyone using it.

Three use cases, and all three are measurement

The stated applications fall into two families. The first is building training datasets, described as collecting diverse real-world conversational data at scale for training or fine-tuning large language models. The other two are measurement rather than training. SEO and competitor analysis means monitoring how competitors' brands and keywords are represented in AI-generated search results. Brand presence management means tracking brand mentions and content rankings to improve visibility strategies. Both measurement cases are the same query run repeatedly, which fits the API design: one prompt, one structured answer, a job identifier to correlate runs with, and an optional callback so a long series of queries does not need a client holding the connection open.

Editorial conclusion

chatgpt-scraper is useful to a developer who already has credentials for the service and needs to know the exact parameter names and the shape of the response, since the documentation is complete enough to skip the dashboard. It is not useful to anyone looking for something they can run locally, because there is nothing to install. Before building against it, read the vendor's own FAQ about legality, decide whether the paid endpoint and its account terms suit your use, and handle the conditional response fields defensively since their presence depends on the answer.

Frequently asked questions

Official sources

1. Issues
2. oxylabs/chatgpt-scraper on GitHub
3. Project website
4. README