LLM Scraper: structured data from any webpage via Playwright and Vercel AI SDK

What problem it solves and who it is for

LLM Scraper addresses the mess of traditional scraping: you write selectors, handle page structure changes, and parse HTML into JSON by hand. This library replaces that with a single call: give it a Playwright page and a Zod schema, and it returns structured data matching that schema. The intended user is a TypeScript developer who already uses Playwright for browser automation and wants to extract, say, product listings or news headlines without maintaining fragile CSS paths. It is not for someone scraping thousands of pages on a budget, because each extraction runs through an LLM, which costs money and takes seconds. The README's example pulls top Hacker News stories with fields like title, points, and by, which shows the typical use case: semi-structured pages where the layout changes often but the semantics stay stable.

How it works: Playwright pages, AI SDK providers, and Zod schemas

The core mechanism is a scraper instance that holds an LLM provider. You launch a Playwright browser, navigate to a page, define a Zod schema for the output, and call run(page, Output.object({ schema }), options). The library takes the page content, formats it according to the chosen mode, sends it to the model, and parses the response against the schema. The Output.object wrapper comes from the Vercel AI SDK, which means the schema is not just for validation; it is also used to constrain the model's response format. The README lists six formatting modes: html for pre-processed HTML, raw_html for unprocessed HTML, markdown, text (via Mozilla's Readability.js), image for screenshots on multimodal models, and custom where you supply your own function. That variety lets you trade fidelity against token usage: raw_html preserves everything but inflates the prompt, while text or markdown shrinks it at the cost of lost layout information.

Getting it running: install, provider setup, and a minimal script

The README gives a concrete path. First install three packages: zod, playwright, and llm-scraper. Then install the provider package for your model; for example npm i @ai-sdk/openai for OpenAI, @ai-sdk/anthropic for Anthropic, @ai-sdk/google for Google, or ollama-ai-provider-v2 for local Ollama models. Groq uses the OpenAI-compatible endpoint via createOpenAI with a custom baseURL. Next, create an LLM instance, for instance openai('gpt-4o'), and pass it to new LLMScraper(llm). The example script launches chromium, opens a page, defines a schema with z.object, and calls scraper.run(page, Output.object({ schema }), { format: 'html' }). The result is an object with a data property, which TypeScript infers from your schema. Note that you need to install the browser binary separately with Playwright's own setup command, though the README does not mention it explicitly.

Streaming and code generation: two modes that change the workflow

Beyond the basic run call, the library offers stream and generate. The stream method returns a partial object stream, letting you consume fields as they arrive from the model. The README shows a for-await loop over the stream. This is useful for long extractions where you want to show progress or act on early fields. The generate method is more interesting: it produces a reusable Playwright script that you can evaluate directly on the page, without another LLM call. The README shows const { code } = await scraper.generate(page, Output.object({ schema })), then page.evaluate(code), then schema.parse(result). That means you can pay the LLM cost once to write scraping code, then run it for free on subsequent pages with the same structure. That is a genuine cost saver for repeated extractions, though the generated code's reliability depends on the model and the page's DOM stability.

A real limitation: cost, latency, and the blind spot of dynamic content

The biggest failure mode is economic. Every run sends a full page's content to an LLM. Raw HTML for a typical news site can be hundreds of kilobytes, which blows up token counts and response times. The text and markdown modes mitigate that, but they lose information. Image mode only works with multimodal models and is even more expensive. There is also no built-in scheduling, retry logic, or rate limiting in the README; you must handle those yourself. A second limitation is that the library depends on Playwright's ability to render the page. If a site requires login, has anti-bot measures, or loads content via complex client-side interactions, you must script those steps before calling run. The README does not address such cases. Finally, the schema must fit the model's output constraints; asking for an array with .length(5) as in the example works, but larger or deeply nested schemas may exceed the model's context or produce invalid JSON that Zod rejects.

Alternatives: how it differs from selector-based scrapers and other LLM tools

The obvious alternative is a traditional scraper like Cheerio or Puppeteer with hand-written selectors. That approach is deterministic, fast, and cheap, but brittle: any class or ID change breaks the code. LLM Scraper trades that brittleness for nondeterminism and cost. A closer alternative is something like Firecrawl or ScrapeGraphAI, which also use LLMs to extract structured data. The difference is architectural: LLM Scraper is a library you embed in your own Playwright session, giving you full control over navigation, cookies, and page interactions. Firecrawl, by contrast, is a hosted API that manages its own browser. If you need to scrape behind a login or handle a complex multi-step flow, the library approach is more flexible. If you want a zero-infrastructure REST call, a hosted service wins. Another difference is that LLM Scraper ties you to the Vercel AI SDK's provider ecosystem, so any model not available there is out of reach.

Maintenance, upgrade cost, and licence implications

The repository is MIT-licensed, so you can use it in commercial projects without paying a fee, but you must retain the copyright notice. The last push was September 2026, and the README notes that version 2.0 brought Vercel AI SDK 6 support and updated examples. That indicates an active maintenance cadence, but it also means the API changed between versions. If you adopted version 1.x, upgrading to 2.x may require changes to how you construct the output schema, since the new version uses Output.object from the AI SDK. The README does not provide a migration guide, so you should check the changelog or the examples folder before upgrading. The dependency on Playwright and the AI SDK means your upgrade cost is not just this library; you must track those upstream releases as well. The project has no recent releases listed, so you may need to install from the main branch to get fixes. Overall, the maintenance picture is positive but requires vigilance.

Editorial conclusion

Adopt LLM Scraper if you already work in TypeScript, need schema-validated extraction from live pages, and want to avoid writing per-site CSS selectors. Do not adopt it for high-volume scraping where cost and latency per page matter, or where the target site blocks headless browsers. Before committing, verify that your chosen model provider is supported through the Vercel AI SDK, that your schema fits the model's context window, and that the output format you need (raw_html, markdown, image, etc.) matches the page structure. The project's last push in September 2026 suggests active maintenance, but confirm the current version's API against the README before upgrading.

Official sources

1. Issues
2. License: MIT
3. mishushakov/llm-scraper on GitHub
4. README