oxylabs/ai-crawler-py: a prompt-driven crawl agent behind the oxylabs-ai-studio package

What the AI-Crawler actually removes from your workload

The recurring cost in scraping is not the HTTP request. It is the selector layer: the CSS or XPath expressions that break when a site ships a redesign, and the per-site scripts that grow around them. The README frames the project against exactly that, saying that unlike traditional scrapers which rely on static selectors, the AI-Crawler uses natural language prompts and AI-assisted URL selection. You describe what you want in a prompt, and the crawl agent decides which pages on the domain are worth visiting and what to pull out of them. The intended audience is stated plainly: developers and data scientists who would rather spend time on analysis than on maintaining custom scrapers. The repository itself is small in scope. It is the Python entry point to a hosted app, so the interesting logic lives on Oxylabs' side, and what you get here is the client surface plus the parameter contract.

The AiCrawler object and the two calls that matter

The whole documented API is two methods on one class. You import AiCrawler from oxylabs_ai_studio.apps.ai_crawler and construct it with an API key. The first method, generate_schema, takes a natural language prompt such as "want to parse name, platform, price" and returns a schema you can then pass back into the crawl call. That is a convenience wrapper: instead of hand-writing an OpenAPI schema, you describe the fields and let the service produce one. The second method, crawl, does the work. It accepts a starting URL, a user_prompt, an output_format, the schema, a render_javascript flag, a return_sources_limit and a geo_location. Two of those are mandatory, url and user_prompt, and the table marks schema as mandatory when output_format is json. Everything else has a default. The data flow is one-directional: your process sends the request, Oxylabs crawls and parses, and you receive a result object whose .data attribute holds the output. In the README example that output is a list of objects, each pairing a data payload with a src field naming the page it came from.

Choosing between markdown and schema-bound JSON

output_format defaults to markdown, which is the lower-commitment path. You get page content as Markdown and do your own parsing afterwards. That suits archiving, feeding text into another model, or any case where you do not yet know the field list. The json path is stricter and more useful for pipelines: you supply a schema, and the service returns records shaped to it. The README's sample output shows three product entries, each with name, platform and price, and each carrying its own src URL. That src field is the part worth noticing, because it lets you trace a parsed record back to the page it came from, which matters when a downstream consumer disputes a number. The schema requirement is also the main friction. If you pass output_format="json" without a schema, you are outside the documented contract. generate_schema softens this, but the README does not describe how faithful the generated schema is to your intent, so treating its output as a draft to review is the safer reading.

Getting it running: the real commands and keys

Installation is a single pip command, pip install oxylabs-ai-studio, and the README states Python 3.10+ as the requirement. You also need an API key, with a free trial offering 1,000 credits mentioned in the registration link. The README's example constructs the client as AiCrawler(api_key="your_api_key") and then calls crawl with url set to https://sandbox.oxylabs.io/products, user_prompt set to "Find all Halo games for Xbox", output_format="json", render_javascript=False, return_sources_limit=3 and geo_location="US". Two of these deserve attention before you copy them. return_sources_limit defaults to 25, so the example's value of 3 is a deliberate narrowing, not the norm. And render_javascript defaults to False, which means JavaScript-rendered content is not executed unless you opt in. On a site that builds its product grid client-side, the default will return less than you expect, and the failure will look like a thin result rather than an error. geo_location takes an ISO2 code and routes through a proxy location, which is how you see region-specific pricing or content.

Where this is the wrong tool

The documented parameter set is the boundary. There is no crawl depth, no page budget, no rate limit, no robots.txt toggle, no include or exclude pattern list, and no scheduling. If your requirement is "crawl this domain but never more than 500 pages and never touch /admin", the README gives you nothing to express that with. return_sources_limit caps how many sources come back, but the README describes it as the maximum number of sources to return, not as a crawl budget, so it is not a substitute for depth control. There is also no stated concurrency or retry behaviour, and no offline mode: every call goes to Oxylabs' service and consumes credits. That rules it out for air-gapped environments, for teams with data residency constraints that forbid sending page content to a third party, and for anyone whose compliance review requires knowing exactly which URLs were fetched and when. The FAQ section in the README is truncated mid-question, so questions about site coverage are not answered in the material available.

How it differs from a general-purpose crawling framework

Scrapy is the obvious alternative and the difference is architectural, not cosmetic. Scrapy runs in your process, gives you spiders, middlewares, an item pipeline, autothrottle, a scheduler and a robots.txt policy you configure yourself. You own the crawl graph and you can inspect every request. The AI-Crawler inverts that: you hand over a URL and a sentence, and the service decides the graph. You gain the ability to skip selector maintenance and the ability to describe extraction in English. You lose introspection and control. For a one-off job against a handful of sites, the trade is usually good. For a recurring crawl across thousands of domains where you need deterministic page selection, reproducible request logs and the ability to debug a single bad extraction, a framework you host will be easier to reason about. The honest summary is that these solve different problems: Scrapy is a crawler you build, and this is a crawl you commission.

Licence, maintenance and what the repository does not tell you

The repository metadata supplied here lists no licence and no primary language, and no releases were retrieved. That is a real gap for anyone doing adoption review. Without a licence file you cannot state the terms under which the client code is distributed, and without releases there is no changelog to read for breaking changes in the parameter contract. The last push date is recent, so the project is active, but activity is not the same as a published versioning policy. The practical consequence is that you should pin the oxylabs-ai-studio dependency to a specific version in your requirements file rather than tracking latest, because a change to the crawl response shape or to a default such as return_sources_limit would surface as a silent behaviour shift in your pipeline. On the service side, credit consumption is the cost model, and the README does not break down how many credits a crawl of a given size consumes. That number is what you need before you can budget a recurring job, and it is not in the material.

Editorial conclusion

Adopt oxylabs/ai-crawler-py if your extraction target is described more easily in a sentence than in CSS or XPath, and if you accept that crawling runs on Oxylabs infrastructure behind an API key. Do not adopt it if you need to control crawl depth, rate, robots handling or queue behaviour, because the documented surface exposes none of those. Before committing, verify three things against your own site: whether render_javascript=True is needed for your pages, what the default return_sources_limit of 25 does to a catalogue with hundreds of matching pages, and whether the generated schema from generate_schema survives review by whoever consumes the JSON downstream.

Official sources

1. Issues
2. oxylabs/ai-crawler-py on GitHub
3. Project website
4. README