XXL-CRAWLER, a lightweight annotation-driven Java crawler

The idea: annotations instead of parsers

XXL-CRAWLER is described as a lightweight web crawler framework for Java, and the promise in the README's introduction is a single line of code to develop a multi-threaded crawler, with a fully annotated way to collect page data into Java objects.

The annotation approach is what distinguishes it. You declare the fields you want on a Java object with annotations pointing at CSS selectors, and the framework does the extraction and wraps the results into the object for you. The README states that the underlying extraction and packaging of the PageVO object is done automatically, and that a single page can yield one or more PageVO objects.

That is a real difference in approach from a hand-written parser. The usual Java scraping code is a sequence of document queries with null checks, and the mapping from selector to field is expressed in imperative code that has to be read to understand. Here the mapping is declared as data on the class, so the shape of the extraction is visible in one place and the boilerplate disappears.

The lightweight claim is about the dependency footprint. The README says the underlying implementation depends only on jsoup, which is worth believing for a Java crawler: the most common heavyweight mistake in this space is pulling in a browser automation stack and a concurrency framework and a logging stack, when the actual work is fetching HTML and selecting nodes.

The release history shows steady work: v1.5.0 in August 2025, v1.6.0 in December 2025 and v1.6.1 in June 2026, with the last push on 2026-08-08. The artifact is on Maven Central as com.xuxueli/xxl-crawler, the licence is Apache-2.0, and the documentation is Chinese, with the homepage at xuxueli.com/xxl-crawler.

The feature list as a checklist of crawler problems

The README's eighteen numbered features read as a list of the problems a crawler has to solve, and it is worth taking them in that order because each one is a decision you would otherwise make.

The first four are design qualities rather than features: an intuitive API that is quick to learn, a light implementation that depends only on jsoup, a modular structure that is easy to extend, and the annotation-driven extraction. Then come the operational ones.

Multi-threading runs on a thread pool to improve collection efficiency. Site-wide expansion starts from an existing URL and spreads to crawl the whole site. JS rendering is handled by extending the PageLoader module, with built-in implementations for Jsoup, which is not JS rendering and is faster, and Selenium with ChromeDriver, which is JS rendering with high compatibility, and room for other implementations.

Proxy support is described as a way to counter anti-collection strategies and WAFs, and the dynamic proxy support lets you adjust the proxy pool at runtime and customise the routing strategy through it. Failed requests are retried with a configurable retry count. Both synchronous and asynchronous running modes are supported. Idempotent de-duplication prevents crawling the same thing twice. URL filtering supports allow-list regular expressions per page.

Then the customisation surface: custom request information such as parameters, cookies, headers, user agent rotation and referrer, and dynamic adjustment of request parameters at runtime. Timeout control for crawler requests. And active pausing, where a crawler thread deliberately waits after processing a page to avoid being blocked for being too frequent.

The last one is worth noting because it is the feature that most crawlers leave to the operator. An intentional delay between requests is the single most effective way to be a good citizen of someone else's server, and having it in the framework rather than in a wrapper sleep call means it is applied consistently.

Distributed operation comes through extending the RunUrlPool module and combining it with Redis or a database to share run data, with a LocalRunUrlPool as the default single machine implementation.

The sample crawlers and the two ways to fetch a page

The README lists seven sample crawlers, and they are worth reading because each one isolates a different capability.

The first is a CNBlog article crawler that demonstrates the annotation extraction, paging through a site and wrapping the results into PageVO objects. The second crawls the same site but downloads the raw HTML to local files instead of extracting data, which is a mode people forget exists when they need an archive rather than a dataset.

The third downloads images from a news site. The fourth crawls a JSON interface rather than a web page, which is the case where a crawler framework is overkill but you still want the threading, the retries and the de-duplication.

The fifth is the interesting one: an e-commerce price crawler that uses Selenium with ChromeDriver because the price is rendered asynchronously. That is the scenario the JS rendering support exists for, and the sample shows the cost. A browser per thread is expensive, and a framework where the page loader is pluggable lets you use the fast path for most pages and the slow path only for the ones that need it.

The sixth uses a proxy to crawl, described as a way to break through access restrictions and protect data security. The seventh is the distributed one, using Redis so that several crawler instances share a RunUrlPool, coordinate on which URLs to expand and consume the tasks to be crawled, and raise the collection throughput.

The seventh sample is the clearest statement of the distributed model. It is not a cluster of nodes coordinating through consensus; it is a shared queue. Each crawler instance pulls URLs from the shared pool and pushes new ones back, and the work divides itself. That is simpler than a real distributed system and considerably more useful than a single machine crawling everything.

The README notes the sample code is for learning and testing only and asks that people get in touch if there is an infringement, which is the usual framing for a Chinese crawler framework whose samples target live sites.

Building on the framework: what to check first

Before pointing the framework at anything that matters, four checks will tell you most of what you need to know.

The first is the extraction fidelity. Run one of the sample crawlers against a page you control, dump the extracted objects and compare them field by field with what you expect. Annotation-driven extraction is fast to write and easy to get subtly wrong, because a selector that matches the wrong element still returns a value rather than an error. This is the failure mode of the approach: silent wrong answers instead of exceptions.

The second is the thread count and the pace. The framework runs on a thread pool, and the active pause feature exists precisely because a crawler that hammers a server gets blocked. Before you point it at a third party site, work out what request rate your thread count implies, and use the pause rather than relying on the target's tolerance.

The third is the proxy and user agent behaviour. Rotation of user agents and dynamic proxy routing are supported, and both change what the target server sees. If you are crawling sites you have permission to crawl, the honest configuration is a single stable user agent and no proxy rotation at all, because pretending to be many clients is what turns a crawler into something a site has reasons to block.

The fourth is the de-duplication and the allow-list filtering working as you expect. Idempotent de-duplication prevents the same page being crawled twice, and the allow-list regular expressions filter which URLs get expanded. Both are configuration you set once and then forget, and both are the settings that decide whether the crawler stays inside the site you meant.

One practical note about the samples. The README says the sample code is for learning and testing only and asks people to get in touch if there is an infringement, and the samples target live Chinese sites whose markup changes without notice. Treat them as documentation of the API shape rather than as something to run in production, and write your own crawlers against your own targets.

Where this fits against the alternatives

The first alternative is jsoup on its own. For a single site with a handful of pages, a jsoup script with a handful of selector calls is shorter, has no framework to learn, and has nothing to break on upgrade. The framework earns its place when the crawling is repetitive enough that the per-page code dominates, and when the threading, proxying, retry and de-duplication logic would otherwise be copied into every new crawler you write.

The second alternative is a general purpose scraping platform, of which there are several commercial ones with hosted browsers, scheduling, proxy pools and structured extraction. Those are the right answer when non-developers need to run the scrapes, or when you need a browser at scale without operating one. The trade is that the data leaves your infrastructure and the pricing follows volume.

The third alternative is a headless browser pipeline, where you drive Chrome directly with Playwright or Puppeteer and write extraction in the page's own JavaScript. That is the modern answer for sites that render heavily, and it handles the cases where Selenium plus ChromeDriver in a Java framework feels dated. It is more expensive per request and gives up the raw HTTP speed that makes jsoup fast.

What XXL-CRAWLER offers in that space is a middle path with a specific shape: a Java API, jsoup-speed for ordinary pages, annotation-driven extraction that removes the imperative parsing code, and the operational features already assembled. If your team writes Java and your targets are ordinary pages, that is a coherent package.

The honest limitations are that the documentation is Chinese, the sample set targets live Chinese sites which may change, and the project is Java only, so a polyglot data platform would have to call it across a service boundary.

Editorial conclusion

Adopt XXL-CRAWLER if you write Java and want a crawler where page extraction is declared with annotations rather than hand-written parsing, and you want threading, proxies and retries without assembling those yourself. Skip it if you need a language-agnostic crawler, since the framework is Java specific, or if your targets render heavily and you would rather use a headless browser pipeline than Selenium. Verify first by running one of the sample crawlers against a site you control, and confirm that the extracted objects match what you expect before pointing it at anything with rate limits.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. xuxueli/xxl-crawler on GitHub