StormCrawler turns Apache Storm topologies into a web crawler

A crawler assembled from Storm bolts rather than a crawler application

Apache StormCrawler is an open source collection of resources for building low-latency, scalable web crawlers on Apache Storm, and it is written mostly in Java. The word that matters is collection. You are not handed a standalone crawler application; you are handed the bolts, spouts and protocol implementations, and you wire them into a topology.

That design makes sense for a specific kind of user. If you already operate an Apache Storm cluster, you already have the machinery for continuous, fault tolerant stream processing, and a crawler is naturally a stream processing problem: URLs arrive, you fetch them, you parse them, and you emit new URLs. StormCrawler supplies the domain logic for each of those stages, so you inherit Storm's supervision and distribution model instead of building a scheduler with a thread pool.

The repository layout matches this. The top level holds core/ with the main library, archetype/ with the Maven archetype and its resources, external/ for additional modules, docs/ for documentation, and a top level pom.xml plus assembly.xml. The build also relies on standard Apache project files such as.asf.yaml, CONTRIBUTING.md, RELEASING.md, SECURITY.md, NOTICE and LICENSE, which are the markers of a project that has to satisfy Apache release and governance process rather than move at the speed of a single maintainer.

The project description calls it a scalable, mature and versatile web crawler based on Apache Storm, and the project website hosts a page listing companies that provide commercial support, which is a reasonable signal of a codebase with production users.

crawler.flux and crawler-conf.yaml are the two files that define a crawl

The archetype does the wiring, and the files it drops in are the real interface of the project. The README points at crawler.flux, at crawler-conf.yaml, and at the files under src/main/resources/, and makes the point that these are all that is needed to run a crawl topology, since all the other components come from the core module.

That division is the architecture. The flux file describes the topology, in a declarative form that names the bolts and the streams between them. The YAML file carries the configuration values that the bolts read at runtime. Everything else, the fetching logic, the parsing, the protocol handling, the politeness and robots handling, the status handling, arrives from the core module as a library dependency. Writing a crawler with StormCrawler therefore means editing two files rather than writing a program.

That is also the main thing to understand about its limits. Because the topology is described rather than coded, a requirement that the shipped bolts do not cover means writing a new bolt, which is a Java component you have to compile and ship separately. The archetype project contains a POM with the core dependency, a default CrawlTopology class, a configuration file and the default resource files, so the generated project is a normal Maven project you can extend.

If you would rather not use the archetype, the README says you can simply copy the files from the archetype-resources directory, which is the escape hatch for people who want to keep their own project structure and only borrow the flux and YAML files.

Generating a project with the Maven archetype and a current version placeholder

The quick start assumes Apache Maven is installed, and it requires Apache Storm 3.1.0 to be installed to run the crawler. The generator is a standard Maven archetype invocation, and the version is a placeholder you have to fill in:

mvn archetype:generate -DarchetypeGroupId=org.apache.stormcrawler -DarchetypeArtifactId=stormcrawler-archetype -DarchetypeVersion=<CURRENT_VERSION>

The README asks you to replace the placeholder with the latest released version, which you can find on search.maven.org for the stormcrawler-archetype artifact. The archetype is published to Maven Central, so the version has to be one that actually exists in the repository, not the version of the source tree you are reading.

After the command runs, you are prompted for a groupId, for example com.mycompany.crawler, an artefactId, for example stormcrawler, a version, a package name, and details about the user agent to use. The user agent prompt matters more than it looks. A crawler that identifies itself badly gets blocked, and the archetype makes you think about the value before the topology ever starts.

The generated directory is the one named after the artefactId, and the README asks you to enter it and follow the instructions in its own README file. At that point you have a POM, a CrawlTopology class, a crawler.flux file and a crawler-conf.yaml, and the next step is submitting the topology to your Storm cluster.

Java 25 and a local Docker environment are the build prerequisites

Building from source has a short requirements list: JDK 25+, Apache Maven 3, and Docker if you want to run tests. The build itself is one command:

mvn clean install

Two of those requirements deserve a note. The first is that StormCrawler requires Java 25 or above, which is a steep floor for a Java library and means a mixed-version build fleet needs separate configuration for this module. The second is that executing tests requires a locally installed and working Docker environment, since the test suite spins up real dependencies rather than mocking them. You can build without Docker, you just cannot run the tests.

The project also asks contributors to format code before submitting a pull request, using a git code format plugin:

mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false

Pre-commit format hooks can be enabled by running the install with formatting active:

mvn clean install -Dskip.format.code=false

The repository also contains checkstyle.xml, an .editorconfig, and a DISCLAIMER-BINARIES.txt file. That last one exists because the project uses binary files for testing advanced crawler functionality, located exclusively in the src/test directories of the respective modules. It is a small but honest detail: some crawler behaviours are only meaningfully testable against real binary responses, and the project has flagged those files as such rather than quietly shipping them.

The core module supplies the bolts, so custom behaviour means writing Java

The central trade-off in StormCrawler is that the library owns almost all of the crawler logic while the user owns almost none of the code. A stock topology will fetch, parse, and discover links using the components in the core module. That is the point of the design, and it is why the archetype can produce a working crawl from two configuration files.

The cost appears when a site needs something the shipped bolts do not do. A site that requires a specific authentication handshake, a form submission, a JavaScript-rendered listing page or a non-standard protocol will not be served by configuration alone. The documented escape route is to write a custom bolt in Java and place it in the topology, and there is an external/ directory in the repository for modules that sit alongside core.

Compare that with the alternative of a small, polite, single-host crawler. A tool that fetches a sitemap, respects a crawl delay and writes results to disk can be a few hundred lines of Python and needs no cluster. StormCrawler earns its complexity when the workload justifies fault tolerant distribution and continuous operation at a scale where a single process falls over. Below that threshold, the flux and YAML model is overhead, and the archetype's own value, that you get a full topology in one command, works against you because the generated project is built to be extended in code rather than configured away.

For support, the README points at the stormcrawler tag on StackOverflow, the discussions section on GitHub, and the project documentation at stormcrawler.apache.org/docs.

Three point releases in 2026, with a Java 25 floor and Apache governance overhead

The release record shows point releases spread through the year: stormcrawler-3.7.0 on 2026-07-29, stormcrawler-3.6.0 on 2026-05-22 and stormcrawler-3.5.1 on 2026-01-17. The repository is not archived and the last push was on 2026-10-19, so work is continuing on the main branch.

The licence is the Apache License 2.0, with LICENSE and NOTICE files at the repository root and the licence badge in the README. For an Apache project this is more than a choice of licence: the NOTICE file and the THIRD-PARTY.properties and THIRD-PARTY.txt files mean dependency licensing is tracked as part of the release process, and RELEASING.md documents how a release is cut. This is a licensing statement rather than legal advice, but the practical effect for an adopter is clear: no copyleft obligations attach to your own crawler code that uses the library.

The upgrade cost is a build environment cost. The current requirement is JDK 25 or above, and the archetype version is coupled to the library version, so a crawler project tracks both. If you generate a project against one archetype version and later upgrade the core dependency, the flux file and the configuration keys are the places where a breaking change is most likely to appear, since those are the contract the library exposes to user code.

Editorial conclusion

Adopt StormCrawler if you already run Apache Storm 3.1.0, have a Java 25 toolchain, and need crawling throughput that justifies a distributed topology, and you can live with the archetype workflow and the flux configuration model. Do not adopt it for a small, polite, single-host crawl where a simple fetcher and a scheduler would do. Before you start, confirm your Storm version, your JDK is 25 or above, and that a local Docker environment is available if you intend to run the test suite, since the README states that executing tests requires it.

Frequently asked questions

Official sources

1. apache/stormcrawler on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases