x-crawl: a Node.js crawler library with optional OpenAI and Ollama extraction

What x-crawl actually solves for Node.js teams

The README frames the problem in terms of maintenance cost. A crawler that locates data through fixed class names or DOM structure breaks when a site is redesigned, and the extraction logic has to be rewritten. x-crawl's answer is to let an AI model read the HTML and return the elements you asked for, so the selector is a sentence rather than a CSS path. The repository description calls it a flexible Node.js AI-assisted crawler library, and the README splits it into two parts: a crawler API that works without AI, and an AI layer that integrates ollama and openai.

That split matters for who this is for. The library targets developers who write TypeScript or JavaScript, run Node.js on their own machines or servers, and are comfortable with Puppeteer-style page objects. It is not aimed at people who want a graphical scraping tool. The README states that the crawler half can work normally even without relying on AI, so a team can adopt the scheduling and retry machinery and ignore the model integration entirely.

How the crawl API, retries and proxy rotation fit together

The entry point shown in the README is createCrawl, which returns an application object carrying configuration such as maxRetry and intervalTime. From that object you call crawlPage, crawlFile, or the interface and static-page equivalents the feature list names. crawlPage resolves with a data object containing page and browser, which means the library hands you the live Puppeteer page rather than a parsed document. You then call page.waitForSelector and page.$eval yourself, exactly as you would with Puppeteer directly.

That design decision is the honest core of the library. It does not hide the browser; it manages the browser's lifecycle and the surrounding policy. The feature list names fixed and random intervals, a configurable retry count, automatic proxy rotation triggered by failed retries with custom error counts and HTTP status codes, and a priority queue that lets a single target jump ahead of others. Device fingerprinting is listed as either zero configuration or custom configuration. The README does not document how the fingerprinting is implemented, so treat that as a claim to verify against the source rather than a described mechanism.

AI sits beside this, not inside it. createCrawlOpenAI takes clientOptions with an apiKey and a defaultModel with a chatModel, and exposes parseElements, which takes an HTML string and an instruction written in natural language. In the README example the instruction is to get the image link, not source it inside, and de-duplicate it. The result is an elements array that the example then feeds into crawlFile as targets.

Installing x-crawl and running a first AI-assisted crawl

The repository is published on npm as x-crawl, and the package.json engines field requires Node.js 24.16.0 or newer. Install it alongside nothing else; puppeteer, openai and ollama are listed as dependencies of the package, so they arrive with it.

npm install x-crawl

The README example reads the OpenAI key from an environment variable rather than inlining it, and passes a model name to defaultModel. The example below is the setup block from the README, with the key read from process.env.

import { createCrawl, createCrawlOpenAI } from 'x-crawl'

const crawlApp = createCrawl({
  maxRetry: 3,
  intervalTime: { max: 2000, min: 1000 }
})

const crawlOpenAIApp = createCrawlOpenAI({
  clientOptions: { apiKey: process.env['OPENAI_API_KEY'] },
  defaultModel: { chatModel: 'gpt-4-turbo-preview' }
})

From there the README chains crawlPage, waits for a selector, pulls the inner HTML with page.$eval, passes that HTML and a natural-language instruction to parseElements, closes the browser, and sends the returned URLs to crawlFile with storeDirs pointing at a local folder.

crawlApp
  .crawlPage('https://www.example.cn/s/select_homes')
  .then(async (res) => {
    const { page, browser } = res.data

    const targetSelector = '[data-tracking-id="TOP_REVIEWED_LISTINGS"]'
    await page.waitForSelector(targetSelector)
    const highlyHTML = await page.$eval(targetSelector, (el) => el.innerHTML)

    const srcResult = await crawlOpenAIApp.parseElements(
      highlyHTML,
      `Get the image link, don't source it inside, and de-duplicate it`
    )

    browser.close()
  })

What you should see is a Puppeteer-controlled browser navigating to the target, then a result object whose elements array holds whatever the model returned. The README's own tip warns that sending an entire page to the model consumes a large number of tokens and that a more precise location description improves accuracy, so the practical first step is to narrow the HTML with a selector before calling parseElements, as the example does.

Where x-crawl is the wrong tool

The AI path is nondeterministic. Two runs over the same HTML can return different element sets, because the extraction is a model response rather than a parsed selector. If your pipeline needs byte-identical output on every run, or if the extracted values feed something that cannot tolerate a field appearing and disappearing, the AI half is the wrong choice and the plain crawler API is the one to use.

Cost is the second boundary. The README itself notes that whole-page extraction consumes a lot of tokens, and createCrawlOpenAI requires credentials for OpenAI or a running Ollama instance. A crawl that runs on a schedule multiplies that cost per run. There is also no distributed mode described anywhere: the configuration covers intervals, retries, proxies and priority within one process. If you need work spread across a fleet of workers with a shared frontier, this library does not describe that.

Finally, the runtime floor is high. Node.js 24.16.0 or newer is a stricter requirement than most Node libraries impose, and Puppeteer means a Chromium download. Environments that cannot fetch or run headless Chromium, or that are pinned to an older LTS line, will not get past installation.

x-crawl against writing Puppeteer by hand

The obvious alternative is Puppeteer directly, which x-crawl already depends on and exposes through the page object it returns. Writing Puppeteer yourself gives you the full API with no wrapper, and no opinionated retry, interval or proxy layer. You also get no AI extraction step, so you write and maintain the selectors.

The difference in approach is where the policy lives. With Puppeteer alone, retry logic, proxy rotation, request spacing and priority are things you build around each script, and they tend to be rebuilt per project. x-crawl moves those into the createCrawl configuration, so maxRetry and intervalTime are declared once and apply to every crawl the application performs. The trade is that you are now inside someone else's abstraction for scheduling, and the README does not document how to reach past it if its retry or rotation behaviour does not match what you need.

A second alternative is a hosted scraping service, which removes the browser and proxy management entirely but adds a vendor dependency and per-request pricing. x-crawl keeps everything in your process, which means your proxy credentials, your API keys and your crawl targets never leave your infrastructure. For teams with data-residency constraints that is the deciding factor; for teams without them, the operational burden of running Chromium is the deciding factor against.

Maintenance, licence and the cost of upgrading

The repository is not archived, and the last push was on 2026-09-14. The most recent release listed is v10.1.0 from 2025-04-06, with v10.0.2 from 2024-07-21 and v10.0.1 from 2024-04-10 before it. The gap between the latest release and the latest push means development activity is visible in the repository even though no release has been tagged since April 2025. If you pin to a published version, that is the version you are getting.

The package version in package.json matches v10.1.0, and the package declares type module, so it is ESM-only. That is a real upgrade consideration: CommonJS projects need a dynamic import or a build step. The engines field also means a Node upgrade may be a prerequisite before a library upgrade, and Puppeteer is pinned to an exact version, 24.43.1, rather than a range, so Chromium behaviour changes only when x-crawl changes it.

The licence is MIT, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are included. That is a permissive licence with no copyleft obligation and no source-disclosure requirement. This is a description of the licence text, not legal advice; if the licence interacts with your own distribution terms, that is a question for your counsel.

Editorial conclusion

Adopt x-crawl if you are already writing Node.js scripts around Puppeteer and want retries, proxy rotation, interval control and a priority queue in one place, or if you want to try AI extraction without committing to a separate pipeline. Do not adopt it if you need distributed crawling across machines, if you cannot run headless Chromium in your environment, or if you want a no-code scraping tool. Before committing, verify that your Node runtime satisfies the engines field, that a headless Chromium download works in your CI, and that your AI provider is reachable from the machine that runs the crawl, since the README shows the API key being read from an environment variable at runtime.

Frequently asked questions

Official sources

1. coder-hxl/x-crawl on GitHub
2. License: MIT
3. Project website
4. README
5. Releases