# SeimiCrawler: writing business logic while the framework runs the crawl

> A Java crawler framework that borrows Scrapy's pipeline ideas, parses HTML with XPath by default and puts distributed queues on Redis through Redisson, aimed at teams that want a crawler inside an existing Spring Boot service.

**zhegexiaohuozi/SeimiCrawler** — 一个简单、敏捷、分布式的支持SpringBoot的Java爬虫框架;An agile, distributed crawler framework.

- Repository: https://github.com/zhegexiaohuozi/SeimiCrawler
- Website: http://seimicrawler.org
- Stars: 1,987 · Forks: 682
- Language: Java
- License: Apache-2.0
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/zhegexiaohuozi-seimicrawler

## What SeimiCrawler takes off your plate

SeimiCrawler is a Java crawler framework whose stated aim is to lower the cost of building a usable crawler system, and its design lineage is explicit: the README says it was inspired by Scrapy and then fused with Java and Spring conventions. The practical effect is that you write one class per site. That class declares where to begin and hands back responses to callback methods, while request scheduling, concurrency, retries of the scheduling layer and result plumbing belong to the framework.

The audience is Java developers rather than data analysts. The project is built around a Spring Boot application, with a standalone mode for when you want a separate process, and it ships on Maven Central under the group id cn.wanghaomiao. Two sibling projects extend it: JsoupXpath, which is the default HTML parser, and SeimiRender, a Chromium based rendering service that the documentation positions as the answer for pages that need JavaScript to reveal their content.

One thing to note about the README itself. Its opening sections carry a paid advertisement for a residential proxy vendor, with pricing and coverage figures for that commercial service. Those numbers describe the proxy product, not SeimiCrawler, and nothing in the framework's own documentation depends on that vendor. Judge SeimiCrawler on what the code does: it schedules and parses, and it leaves proxy choice entirely to you.

## The crawler class is the whole programming model

A SeimiCrawler site is a class extending BaseSeimiCrawler, annotated with a name that later selects it at start time. Two methods matter. startUrls returns the entry points, and start receives the response for each of them. From there, push enqueues follow-up requests, each with a callback method reference that receives the response for that URL.

The parsing default is XPath. The response object exposes a JXDocument, and the doc.sel call evaluates an XPath expression against it, returning a list of matched nodes. The README's example crawler pulls link attributes from an anchor class, pushes one request per link with the method reference Basic::getTitle, and in the callback reads a heading element from the fetched page. Method references rather than string callback names are a V2 feature the README highlights explicitly.

Because extraction is expressed as XPath rather than as a code-side DOM walk, writing a selector is close to writing a query. The trade-off is that XPath has no natural place for the small amount of cleanup that messy real pages tend to need, so that logic ends up in Java anyway. The README acknowledges this by noting that data processing may use other parsers if you prefer, which keeps the default sensible without locking you into it.

## Two configuration shapes: SeimiConfig and application.properties

Running standalone means constructing the framework yourself. The README shows a SeimiConfig instance holding the SeimiRender host and, commented out, the Redis single server address, then a Seimi object created from that config:

```java
SeimiConfig config = new SeimiConfig();
config.setSeimiRenderHost("127.0.0.1");
//config.redisSingleServer().setAddress("redis://127.0.0.1:6379");
Seimi s = new Seimi(config);
s.goRun("basic");
```

Running inside Spring Boot moves the same knobs into standard properties. The keys below control which crawlers start on a start request, where the rendering service lives, and whether the distributed queue is on:

```properties
seimi.crawler.enabled=true
seimi.crawler.names=basic,test
seimi.crawler.seimi-render-host=xx
seimi.crawler.seimi-render-port=xx
seimi.crawler.enable-redisson-queue=true
```

That split is a real design decision with a real cost. The same setting has two spellings depending on the deployment style, so a project that starts standalone and later moves to Spring Boot has to translate its configuration. The properties above are also the place where the two Bloom filter knobs live, seimi.crawler.bloom-filter-expected-insertions and seimi.crawler.bloom-filter-false-probability, both documented as optional with a default false positive probability of 0.001.

## Distributed mode runs on Redis through Redisson

The V2 rewrite moved the distributed queue onto Redisson while keeping Redis as the store underneath, and the reason for the change was deduplication. Rather than keeping a growing set of visited URLs in Redis, the framework filters them through a Bloom filter, which the README frames as a space efficiency improvement. A Bloom filter cannot confirm a URL was seen, it can only tell you it might have been, which is why the false positive probability is configurable rather than fixed.

The operational consequence is that distributed mode has an external dependency you must run and size. A single-node Redis is the simple case, and the commented config line shows the address format the framework expects. Beyond that, the README points to an online Bloom filter tuning simulator with example parameters, which tells you the intended workflow: work out how many URLs you expect to insert and what error rate you can live with, then set the two properties to match.

There is no documented setting for a Redisson cluster in the README beyond the mention that Redis cluster information is configurable through SeimiConfig, so a multi-node Redis topology is something you would configure rather than something the documentation walks you through.

## Packaging for distribution with the Maven plugin

A Spring Boot project packages the standard way, with mvn package. For standalone delivery, SeimiCrawler offers a dedicated Maven plugin that produces a runnable distribution directory. You add it to the pom with a build execution bound to the package phase:

```xml
<plugin>
    <groupId>cn.wanghaomiao</groupId>
    <artifactId>maven-seimicrawler-plugin</artifactId>
    <version>1.2.0</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>build</goal>
            </goals>
        </execution>
    </executions>
</plugin>
```

Running mvn clean package then produces a directory with a bin folder holding run.sh for Linux and run.bat for Windows, and a seimi folder split into classes for your crawler code and configuration and lib for the dependencies. The plugin has its own repository and documentation, and the README notes that the configuration block can override the output directory, which defaults to the target directory.

This plugin is versioned separately from the framework, at 1.2.0 against a framework release of 2.1.5. Two version lines to track means an upgrade can touch both your pom and your application at once, which is worth accounting for before you pin either.

## Where SeimiCrawler is the wrong choice

The clearest limitation is the rendering story. SeimiCrawler does not execute JavaScript itself. That job belongs to SeimiRender, a separate Chromium based service with its own host and port in the configuration, and the README describes it as the modern successor to SeimiAgent with an MCP protocol for agent access. So a crawler for a single page rendering application needs two services, an extra port and a dependency on the render protocol, rather than one jar. Projects that only need static HTML pay for that dependency in setup complexity even if they never use it.

The second limitation is the JDK baseline. The documentation states that JDK 1.8 or newer is required. On a team still standardising on a current long term support release, a framework whose floor is Java 8 means the artifact runs on your JVM but is not built for it, and you get no newer language features inside crawler classes.

The third is scope and cadence. The repository contains the framework module seimicrawler, a spring-boot-example and a standalone-example, and documentation beyond that lives on the project homepage and in a ChangeLog file. Published versions are 2.1.5 on 2026-07-29, then 2.1.4 on 2023-04-25 and 2.1.3 on 2023-02-08. That three-year gap between 2.1.4 and 2.1.5 tells you releases are not on a fixed cadence, so you should read ChangeLog.md when you upgrade rather than assume a sequence of incremental fixes. The last push on 2026-10-15 shows the tree is still moving.

Finally, the framework does not solve access. It schedules requests and parses documents, and the harder half of large scale collection, obtaining pages that do not want to be fetched, remains a proxy and browser problem you solve outside it.

## Licence and what you inherit

SeimiCrawler is released under Apache-2.0, which is what the licence badge in the README points at and what the project's listing states. That is a permissive licence with an explicit patent grant, and the same terms apply to the parts you copy into your own project. This is not legal advice, and it does not cover the separate proxy vendor the README advertises or the terms of the sites you crawl.

The dependency picture is worth reading before you commit. The framework sits on Redisson for distributed coordination, JsoupXpath for parsing, and optionally SeimiRender for dynamic pages, so your production footprint is larger than the single Maven artifact suggests. The main dependency on Maven Central is published under cn.wanghaomiao, and the README asks you to treat the latest central version as the source of truth, showing 2.1.5 as the example.

## Conclusion

SeimiCrawler fits a Java team that already runs Spring Boot, wants crawl logic as plain classes, and is willing to operate Redis for distributed mode. It does not fit a project that needs a stable upstream contract, a modern JDK baseline, or a crawler that handles anti-bot defences on its own. Verify first that the artifact version you resolve from Maven Central matches your Spring Boot generation, since the release history jumps from 2023 to 2026, and confirm your page set can be expressed as XPath before committing to it.

## FAQ

### What does a SeimiCrawler crawler class have to implement?

A class extending BaseSeimiCrawler, annotated with @Crawler and a name, overriding startUrls to return the entry URLs and start to handle the first response. Follow-up requests are queued with push, passing a request URL and a callback method reference such as Basic::getTitle.

### How do I enable the distributed queue in SeimiCrawler?

In Spring Boot mode you set seimi.crawler.enable-redisson-queue=true and point the framework at Redis, for example through the commented redisSingleServer().setAddress("redis://127.0.0.1:6379") line in SeimiConfig. The distributed queue is implemented on Redisson over Redis.

### Does SeimiCrawler handle pages that need JavaScript rendering?

Not on its own. Rendering is delegated to the separate SeimiRender service, and you configure its host and port with config.setSeimiRenderHost or the seimi.crawler.seimi-render-host and seimi.crawler.seimi-render-port properties.

### Which parser does SeimiCrawler use for HTML, and can I change it?

The default parser is JsoupXpath, a separate project, and extraction is written as XPath expressions evaluated through the JXDocument returned by response.document(). The README notes that data processing can use other parsers if you prefer.

### How do I package a SeimiCrawler project for standalone deployment?

Add the cn.wanghaomiao:maven-seimicrawler-plugin:1.2.0 plugin to the pom with a build goal bound to the package phase, then run mvn clean package. The resulting directory contains bin/run.sh and bin/run.bat scripts plus a seimi folder holding classes and lib.

## Sources

- [License: Apache-2.0](https://github.com/zhegexiaohuozi/SeimiCrawler/blob/master/LICENSE)
- [Project website](http://seimicrawler.org)
- [README](https://github.com/zhegexiaohuozi/SeimiCrawler/blob/master/README.md)
- [Releases](https://github.com/zhegexiaohuozi/SeimiCrawler/releases)
- [zhegexiaohuozi/SeimiCrawler on GitHub](https://github.com/zhegexiaohuozi/SeimiCrawler)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/zhegexiaohuozi-seimicrawler