Lightfeed Extractor: LLM Structured Data Extraction with Schema Recovery
Use LLMs to robustly extract web data
At a glance
- What is it?
- A TypeScript library that converts HTML to LLM-ready markdown and extracts typed data against a Zod schema. The JSON recovery layer is the interesting part, and it tells you what kind of reliability problem the authors were actually solving.
- Who is it for?
- Adopt it if you are already running LangChain providers in a TypeScript pipeline and your schemas are nested enough that raw JSON mode keeps failing validation. Skip it if you need deterministic output without per-page LLM cost, or if you want a hosted extraction API.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 102 days ago.
- What is it written in?
- Mainly TypeScript, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem Lightfeed Extractor targets is schema conformance, not fetching
Most scraping libraries solve retrieval. They get bytes off a server and hand you a DOM. The gap this project addresses sits after that point: you have HTML, you have a target shape in mind, and you need the two to meet without writing a selector for every site variant. The README frames the library as being for "robust web data extraction using LLMs" where you "use natural language prompts to extract structured data from HTML, markdown, or plain text." The audience is TypeScript developers building data pipelines, and the topics list names ecommerce scraping, ETL, RAG and crawlers as the intended neighbourhood. That framing matters because it sets the failure cost. A selector-based scraper that breaks produces nothing. An LLM-based extractor that breaks produces something plausible-looking and wrong, which is a worse outcome in a pipeline that does not validate downstream. The library's answer to that is schema validation through Zod plus a recovery step for malformed JSON, and that pairing is the actual product. If you only needed text out of a page, a readability port would do.
HTML to markdown conversion runs before the model sees anything
The pipeline has two distinct stages and the first one is not an LLM call. Content arrives as HTML, markdown or plain text; the format argument tells the library which. When the input is HTML, the library converts it to markdown with options the README names directly: extractMainHtml, includeImages and cleanUrls. The cleanUrls option strips tracking parameters, and the README also describes handling relative URLs, dropping invalid ones, and repairing markdown-escaped links. This ordering is the design decision worth noting. By reducing a full page to markdown first, the library cuts the token count before the model is invoked, which is where the "great token efficiency" claim in the overview comes from. It also means the LLM never sees the raw DOM, so anything that only exists as an attribute or a script payload is gone by the time extraction runs. For a product listing where price and title are visible text, that is fine. For a page where the price is rendered client-side into a data attribute, the markdown conversion is the wrong front door and you would need a different path. The library does not claim otherwise, but the README's emphasis on markdown as the LLM-ready format makes the trade-off easy to miss.
Zod schemas drive the extraction call and JSON recovery patches the gaps
The extract function takes an llm instance, the content, a format, an optional sourceUrl, a Zod schema, and htmlExtractionOptions. The README states that the library uses LLMs in JSON mode to extract data according to the input Zod schema, and that token usage limits and tracking are included. The schema is not decoration. Field-level .describe() calls become part of the prompt, which is how the model learns that a given number is a price rather than a rating. Nested arrays and optional fields are where this gets hard, and the README is explicit that the JSON recovery feature exists for exactly that: "Sanitize and recover failed JSON output. This makes complex schema extraction much more robust, especially with deeply nested objects and arrays." Read that as an admission. If the provider's JSON mode were reliable enough on its own, a recovery layer would not be the headline feature. The practical consequence is that you should expect a nonzero fraction of calls to return text that fails initial parsing, and you should decide in advance whether a recovered result is trustworthy for your use case or whether it needs a second pass. The library gives you the recovery; it does not give you a confidence score on the recovered object.
Installation pulls in LangChain as a peer dependency you must pin yourself
Setup is two npm commands plus a provider package. The base install is npm install @lightfeed/extractor @langchain/core, and then you add one of @langchain/openai, @langchain/google-genai, @langchain/anthropic or @langchain/ollama depending on where inference runs. The README flags @langchain/core as a required peer dependency shared between this library and every provider package, with an explicit instruction to install it directly rather than letting it resolve transitively. That instruction is worth following literally. Peer dependency resolution across a library and several provider packages is a common source of duplicate-instance bugs where types line up but runtime identity does not. The Ollama option is the one to note if you have data residency constraints, since it is the only listed provider that can run without sending page content to a third party. The README does not state which models are supported per provider or what context window is required, so that is something you would determine from the provider packages themselves.
The Walmart example shows the intended shape of a real extraction call
The README's worked example launches Chromium through Playwright, navigates to a Walmart grocery category page, waits for networkidle with a 10000 ms timeout wrapped in a try/catch that logs and continues on failure, grabs page.content(), and closes the browser before extraction. Then it calls extract with a ChatGoogleGenerativeAI instance configured for gemini-2.5-flash at temperature 0, passing the HTML, ContentFormat.HTML, the source URL, a product catalog schema, and htmlExtractionOptions set to extractMainHtml true, includeImages true, cleanUrls true. Two details in that sequence are load-bearing. The networkidle timeout is caught rather than thrown, which means the library is designed to run against pages that never fully settle; the extraction has to tolerate partial DOM. And the schema uses z.number() for price with a separate optional originalPrice, which is the pattern you want if you plan to diff prices over time. The README shows an expected output block with a named product, a price of 3.98, an originalPrice of 4.57, a rating and a review count. That block is documentation of intended shape, not a benchmark, and the README does not report accuracy rates or per-page token counts. Anyone evaluating this for a production pipeline should treat the example as a starting point to reproduce, not as evidence of hit rate.
Where the LLM-in-the-loop design is the wrong tool
Three cases argue against this library. First, high-volume deterministic extraction. If your target pages share stable markup and you control the selectors, a parser costs nothing per call and returns identical output for identical input. Routing that through a language model adds latency, per-page cost, and a class of failure that selectors do not have: confident wrong answers. The README's own token tracking feature exists because that cost is real enough to measure. Second, pages where the data is not in the visible text. Because the pipeline converts HTML to markdown before the model sees it, content that lives only in attributes, JSON-LD blocks, or hydration payloads may not survive the conversion. The includeImages option suggests images are handled as URLs rather than as vision input, so a page whose price is baked into a rendered image is out of scope. Third, any workflow where you need to explain a result. A selector-based extractor can point at the rule that produced a value. An LLM extraction produces an object and, per the README, no stated rationale. If your domain requires auditability of each extracted field, that gap is not something the JSON recovery layer closes.
Playwright and browser-agent sit at different layers of the same stack
The closest comparison inside this project's own ecosystem is @lightfeed/browser-agent, and the README treats them as complements rather than alternatives. Browser-agent handles navigation: searching, clicking through pagination, dismissing popups, driven by natural language commands. Extractor handles the read step after the page is in the right state. The README's guidance is that you pair them "for pages that require interaction before extraction." The difference in approach is therefore about which problem you have. If your target page loads and shows everything at once, Playwright plus extractor is sufficient and adding browser-agent only adds a second model call. If the data is behind a search box or a paginated list, the extractor alone cannot reach it, because it reads content you hand it and does not drive the page. The README does not describe how the two libraries coordinate state, so the integration surface is something you would read from the browser-agent repository. Outside that ecosystem, the meaningful alternative is a plain HTML parser such as Cheerio or a readability port, which is faster and free but requires you to write and maintain per-site rules, and which will not generalize when a site changes its markup.
Version cadence, licence and what to check before you depend on it
The project ships under Apache-2.0, which permits commercial use and modification and includes an explicit patent grant. That is a permissive licence, and it is the same family most infrastructure libraries use. It is not legal advice; if you are redistributing the library or embedding it in a product, read the LICENSE file in the repository and get your own counsel on notice requirements. On maintenance, the release history shows v0.5.2 and v0.5.3 both dated 2026-04-07, roughly an hour apart on the same day, followed by v0.5.4 on 2026-06-06. Two releases in one day usually means a fix shipped immediately after a release, which is worth knowing if you pin versions. The version numbers are still in the 0.5.x range, so the API has not reached a 1.0 commitment and minor releases may carry breaking changes. The upgrade cost is dominated by the peer dependency chain rather than by the library itself: a bump to @langchain/core can force provider package bumps, and the README's own warning about installing @langchain/core explicitly is the mitigation. Budget for re-running your extraction tests on every minor bump, because the JSON recovery behaviour is the kind of thing that changes quietly between versions.
Editorial conclusion
Adopt it if you are already running LangChain providers in a TypeScript pipeline and your schemas are nested enough that raw JSON mode keeps failing validation. Skip it if you need deterministic output without per-page LLM cost, or if you want a hosted extraction API. Before committing, run the repository's own test:browser script against two or three of your real target pages and count how often the JSON recovery path fires; that ratio, not the README's Walmart example, is what determines whether this library saves you work.
Community notes