Gecco for lightweight Java web crawling

Gecco's target problem and Java design

Gecco addresses the repeated code that appears when a Java application needs to collect data from HTML pages. Instead of manually managing HTTP requests, parsing documents, and mapping selected values to business objects, the framework defines crawl rules through annotations or a runtime configuration builder. The intended users are Java developers who already have an application structure and want to add focused extraction without adopting a large crawler platform.

The README describes the project as a lightweight web crawler integrated with jsoup, httpclient, fastjson, Spring, HtmlUnit, and Redis-related extensions. Its core design is selector-oriented. A crawl bean can declare URL patterns, request parameters, HTML fields, and an output pipeline, while jQuery-style CSS selectors identify the elements that become fields. This is a useful fit for structured pages where the data appears in predictable markup.

Gecco also covers several operational details that often sit outside a small extraction library. It supports asynchronous Ajax requests, JavaScript variable extraction, random User-Agent selection, and random proxy selection. Redis support is described as distributed crawling through a separate gecco-redis project. Spring and HtmlUnit are also separate extensions rather than features that are necessarily present in the base artifact. That separation matters when estimating deployment size and upgrade work.

How annotation rules become crawl objects

The annotation model places the crawl definition beside the Java object that receives the extracted values. In the GitHub example, the bean implements HtmlBean, declares a URL pattern with Gecco, and uses RequestParameter fields for path variables. HtmlField then associates each property with a CSS selector and an extraction mode such as text or HTML. The engine starts from a matching URL, sends the request, extracts the selected content, and passes the bean to the configured console pipeline.

The example also shows that URL matching can include named parameters. The pattern contains user and project placeholders, while the engine's initial URL supplies a concrete repository path. This separates the crawl rule from the entry URL and makes the bean reusable for multiple pages. Setters and getters are present because the extraction process needs to populate the bean rather than merely return a string map.

The engine configuration controls basic crawl behavior in the README example. The classpath identifies the package containing crawl beans. The thread setting limits concurrency to one, interval sets a delay between requests, loop allows continued processing, and mobile selects whether a mobile-oriented request mode is used. These controls are explicit rather than hidden behind a global configuration file. However, the README does not document retry behavior, response status handling, encoding defaults, politeness rules, or persistence semantics, so those areas need direct verification before production use.

Installing Gecco and running the GitHub example

The README gives Maven coordinates but uses a placeholder version rather than naming a concrete release. The artifact is published under the com.geccocrawler groupId. Replace the placeholder only after selecting a version available in the repository you use; the README itself does not identify the intended version for this snippet.

<dependency>
    <groupId>com.geccocrawler</groupId>
    <artifactId>gecco</artifactId>
    <version>x.x.x</version>
</dependency>

After the dependency is available to the Java project, create a bean based on the documented quick-start class. The class below is copied from the README's example, including its selectors and engine settings. It is a Java source example rather than a shell command, so it should be placed in the application source tree under a package matching the configured classpath.

public class MyGithub implements HtmlBean {
    @RequestParameter("user")
    private String user;

    @RequestParameter("project")
    private String project;

    @Text
    @HtmlField(cssPath=".pagehead-actions li:nth-child(2) .social-count")
    private String star;

    @Html
    @HtmlField(cssPath=".entry-content")
    private String readme;

    public static void main(String[] args) {
        GeccoEngine.create()
            .classpath("com.geccocrawler.gecco.demo")
            .start("https://github.com/xtuhcy/gecco")
            .thread(1)
            .interval(2000)
            .loop(true)
            .mobile(false)
            .start();
    }
}

The omitted accessors in this shortened block would need to be included in a compiling bean, or the class can be copied in full from the README. The documented selectors expect a particular GitHub page structure, so a change in the site's markup can produce missing or misleading fields even when the crawler starts normally. The `consolePipeline` named in the original example also needs to exist in the application's configured pipelines.

DynamicGecco and runtime rule generation

DynamicGecco provides an alternative to annotation-based beans. According to the README, it generates a SpiderBean at runtime through bytecode programming and uses a custom GeccoClassLoader for rule deployment. The practical difference is that the extraction rule can be registered from application code instead of being fixed in a compiled bean class. That can help applications that receive crawl definitions from configuration, administration screens, or another service.

The builder still uses the same underlying concepts as annotations. It registers a URL template, a request field, request parameters, and fields extracted with CSS selectors. The example also selects an attribute through href, showing that the builder is not limited to visible text. After registration, the engine is started against a concrete URL and can be run once rather than placed in the annotation example's looping mode.

DynamicGecco.html()
    .gecco("https://github.com/{user}/{project}", "consolePipeline")
    .requestField("request").request().build()
    .stringField("user").requestParameter("user").build()
    .stringField("project").requestParameter().build()
    .stringField("star").csspath(".pagehead-actions li:nth-child(2) .social-count").text(false).build()
    .stringField("fork").csspath(".pagehead-actions li:nth-child(3) .social-count").text().build()
    .register();

GeccoEngine.create()
    .classpath("com.geccocrawler.gecco.demo")
    .start("https://github.com/xtuhcy/gecco")
    .run();

This approach reduces source changes, but it introduces runtime generation and class loading as additional operational concerns. The README does not explain how generated classes are unloaded, how rule conflicts are resolved, or how errors are reported after registration. Those details matter more for long-running services than for a one-off crawl, and should be checked against the selected release and source code.

JavaScript pages, Redis, and optional extensions

Gecco is not limited to static HTML extraction. The feature list explicitly includes asynchronous Ajax requests and extraction of JavaScript variables, while HtmlUnit support is available through gecco-htmlunit. This makes the framework relevant to pages whose useful content is assembled after the initial response. The README does not specify which JavaScript engine is used by default, which browser events are awaited, or how unsupported browser APIs are handled, so Ajax support should not be treated as equivalent to a general browser automation platform.

The Redis extension is described as a way to realize distributed crawling. Distributed work commonly requires shared task state, coordination, duplicate control, and failure recovery. The README names the capability but does not document the Redis data model or the consistency guarantees of the separate extension. A team considering multiple crawler instances would need to inspect gecco-redis directly before assuming that adding Redis automatically provides safe coordination.

Spring integration is similarly optional. The project points to gecco-spring for developing business logic, which suggests that pipelines can coordinate with an application context. Proxy selection and random User-Agent selection are built into the listed capabilities, but the README does not state how proxy failures are retried or whether User-Agent selection is per request, per bean, or per engine. These are configuration-sensitive details for sites that reject automated traffic.

Maintenance, licensing, and the right alternative

The repository is not archived, and its last push was on 2026-10-09. The release list shown in the README records 1.3.0 on 2017-09-15, 1.2.10 on 2017-08-24, and 1.2.5 on 2016-11-03. A recent push does not by itself establish that the published Maven versions are current, so adopters should check whether the last push changed source, packaging, or documentation without creating a new release.

Gecco is licensed under MIT. That permits use and modification under the license terms, but choosing an extension means reviewing that extension's own license and dependencies as well. The README does not provide a release policy, migration guide, supported Java version matrix, or compatibility statement for newer JDKs.

The closest conceptual alternative is a conventional crawler application built directly on jsoup and an HTTP client. That approach gives the developer explicit control over requests, parsing, retries, storage, scheduling, and browser automation, but it requires assembling those pieces yourself. Gecco reduces that assembly for selector-based extraction and pipelines. In return, teams accept a framework-specific model, optional extension repositories, and undocumented edge behavior around failures and runtime rules. For a small, well-shaped extraction job Gecco can be more concise. For a crawler requiring proven retry semantics, large-scale scheduling, or detailed browser behavior, the direct library approach may be easier to reason about.

Editorial conclusion

Adopt Gecco when you want a compact Java crawler with selector-based extraction and optional Redis, Spring, or HtmlUnit integrations. Do not choose it solely for JavaScript-heavy sites or large distributed workloads without first checking the exact version, extension behavior, and operational requirements. Verify the published artifact, selector assumptions, pipeline output, proxy handling, and the behavior of dynamic JavaScript requests before relying on it for production crawling.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. xtuhcy/gecco on GitHub