oxylabs/chatgpt-scraper: a Java repository for sending prompts through Oxylabs Web Scraper API

What oxylabs/chatgpt-scraper actually is, and who it is for

This is not a scraper you run. It is a collection of request examples and an output sample for a hosted target on Oxylabs' Web Scraper API. The README states the goal plainly: send prompts to ChatGPT and collect both conversational responses and structured metadata, with parsed JSON returned and no need to manage proxies, browsers, or server-side defenses. The repository's top-level entries are a Code examples directory, the README, two banner images, an HTML output screenshot, and output-chatgpt-scraper.json. That layout tells you what to expect: reference material, not a packaged library.

The audience is narrower than the topics suggest. If you are doing SEO monitoring, tracking how a brand appears in AI answers, or building a dataset of model responses for analysis, and you already have Web Scraper API credentials, this saves you the work of writing a browser automation stack. If you want to call ChatGPT yourself with your own OpenAI key, this is the wrong repository entirely, because the request goes to data.oxylabs.io, not to OpenAI.

The request mechanism: one POST, a source key, and parsed content

The data flow is a single HTTP POST to https://data.oxylabs.io/v1/queries with basic authentication. The payload carries source set to chatgpt, the prompt, and optional flags. According to the README's parameter table, source and prompt are mandatory, prompt is capped at 4000 characters, and parse defaults to false, which means you get the structured JSON envelope only if you explicitly set parse to true. That default is worth noticing: a first request without it returns something less convenient, and the README does not describe what the unparsed form looks like beyond the HTML screenshot in the repository.

The response envelope is results[], and each entry has job_id, status_code, url, and a content object. The ChatGPT-specific fields inside content are prompt, llm_model, response_text, markdown_text, markdown_json, citations, search_queries, and parse_status_code. markdown_json is an array where each item has a type and children, so you can render the answer without writing a Markdown parser. citations is an array of objects with title, url, text, and description. The README notes that the number of items and fields can vary with the prompt, and that element composition may differ between desktop and mobile queries. Treat the schema as a shape, not a guarantee.

Installing and sending your first prompt

There is no package to install. The README points to a free trial through the Oxylabs Dashboard for credentials, and the repository ships code examples in a Code examples directory. The README's own sample is Python, even though the repository is listed as Java, so the fastest path is copying that sample and substituting your username and password. The block below is the README example with the placeholders left as they appear there.

import requests
from pprint import pprint

payload = {
    'source': 'chatgpt',
    'prompt': 'best supplements for better sleep',
    'parse': True,
    'geo_location': "United States",
    'callback_url': "https://your-server.com/oxylabs-callback"
}

response = requests.request(
    'POST',
    'https://data.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'),
    json=payload,
)

pprint(response.json())

Run it and you should see the envelope described above: a results array with a job_id, a status_code of 200 on success, and a content object containing response_text alongside markdown_text and citations. Note that the sample includes callback_url, which pushes the result to your own endpoint rather than returning it inline; if you want the answer in the response body, drop that key. To enable web search, add search set to true, since the README lists its default as false. For a Java version, look in the Code examples directory rather than the README, which only shows Python.

Where this breaks down: prompt limits, plan gating, and schema drift

The 4000-character prompt cap is a hard constraint for anyone trying to feed long documents or multi-turn context into the request. There is no documented mechanism in the README for continuing a conversation: each request carries one prompt, and nothing describes passing a prior response back in. If your use case needs back-and-forth dialogue, this API target does not model it.

The second constraint is commercial rather than technical. Every request goes through Oxylabs with your credentials, and the README points to a free trial through their Dashboard. That means per-query cost, account limits, and availability all sit with the vendor. There is no offline mode, no local fallback, and no way to run this against a self-hosted model. The README also does not document rate limits, retry semantics, or what happens when a job fails, so error handling is guesswork until you test it against a live account.

Third, the output is described as variable. The README says the number of items and fields may vary depending on the prompt, and that desktop versus mobile queries can produce different element compositions. Code that assumes citations is always present, or that markdown_json always has the same nesting, will break on some prompts. Validate defensively.

How it compares to Bright Data's ChatGPT scraper

The related search people use is a comparison against Bright Data, and the difference is mostly in packaging rather than concept. Both route prompts through a hosted scraping API with credentials and return parsed output, so neither is a self-hosted tool. What distinguishes this repository is its shape: oxylabs/chatgpt-scraper is documentation plus a handful of language examples, with the README sample written in Python and the repository tagged as Java. Bright Data's equivalent offering is a separate product with its own API surface and its own documentation set, and this repository contains no code, adapter, or comparison against it. If you are choosing between them, the decision rests on which vendor's API contract, pricing, and geographic routing you already use, not on anything in this repository, because there is nothing here that would port to the other vendor without rewriting the request layer.

Maintenance, licensing, and what the repository does not tell you

The repository is not archived, and the last push was on 2026-08-26, which is recent enough that the examples track the current API. That matters more than usual here, because the API target's fields are the product: if Oxylabs changes the content schema, the output sample and the README tables are what go stale first.

The licence is not stated in the repository metadata. That is a real gap for a repository you might copy code from into a commercial codebase. Without a licence file or an SPDX identifier, the default position is that no permission has been granted, and the README does not address it. This is not legal advice, and the practical answer is to ask Oxylabs directly before vendoring any of the example code. The output-chatgpt-scraper.json file is useful as a fixture for testing your parser, but it is a single sample, and the README already warns that field composition varies by prompt.

Editorial conclusion

Adopt it if you already pay for Oxylabs Web Scraper API and want parsed ChatGPT answers plus citation metadata without maintaining browsers or proxies; the repository itself is example code, not a library, so there is nothing to version-pin beyond the API contract. Do not adopt it if you need a self-hosted scraper, offline operation, or a documented open source licence, since none of those are present. Before writing production code, verify three things against the live API: the exact auth format for your account, whether your plan includes the chatgpt source, and whether callback_url delivery fits your infrastructure.

Frequently asked questions

Official sources

1. Issues
2. oxylabs/chatgpt-scraper on GitHub
3. Project website
4. README