NetDiscovery swaps the crawler library for a pluggable queue and downloader stack

The unit of extension is a module, not an interface you implement

The design of NetDiscovery is visible in the module names, and they are worth reading as a list of decisions. The core is split into netdiscovery-core-core, netdiscovery-core-engine and netdiscovery-core-engine-monitor. The queue layer has netdiscovery-queue-kafka, netdiscovery-queue-rabbitmq, netdiscovery-queue-redis and netdiscovery-queue-rocketmq, plus a Disruptor option named in the feature list. The downloader layer has netdiscovery-downloader-httpclient, netdiscovery-downloader-okhttp, netdiscovery-downloader-htmlunit and netdiscovery-downloader-selenium. The pipeline layer has netdiscovery-pipeline-elasticsearch, netdiscovery-pipeline-mongo, netdiscovery-pipeline-redis and netdiscovery-pipeline-couchbase. Each of those is a separate Maven artifact that you add to your build, so the composition of your crawler is decided by your dependency list rather than by configuration. The feature list also says the framework supports your own implementations, which is the escape hatch when none of the shipped options fit. For a team whose queue is already Kafka, that is the difference between a day of work and a rewrite.

RxJava 2 gives the multithreading and the thread pool isolation

The concurrency model comes from RxJava 2 rather than from a custom scheduler, and that is stated as the reason the framework is multithreaded and asynchronous. Two consequences follow, and they matter more than the wording suggests. First, the pipeline stages are not tied to one execution model, so a parser or a pipeline step can run in its own thread pool, which the feature list calls thread pool isolation. That is what keeps a slow Elasticsearch write from blocking the download loop, and it is a property you would otherwise have to build yourself on top of a plain blocking HTTP client. Second, because the scheduling is reactive, adding a new stage means composing another observable rather than introducing another executor. The price is a dependency on a reactive library in every project that uses the framework, and the project itself notes that support for RxJava 3.x is on the TODO list, so the reactive generation is pinned to the 2.x line.

The SpiderEngine is the piece that makes it a service

A crawler can be a library you call from a test, or a service you deploy. NetDiscovery supports both, and the README says a Spider can run on its own while a Spider can also be handed to a SpiderEngine to control it. The engine integrates Quartz, which is how scheduling is expressed, and it exposes both http and RPC interfaces, so other systems can drive a crawl rather than only observing it. For distributed work, the engine can register itself with Etcd or Zookeeper before it runs, and the monitor module then watches those registered engines. The feature list also mentions that the agent module reports CPU and memory of the current server in real time, which is the kind of signal an operator needs when a crawl starts consuming the box. The engine-monitor module is a separate artifact, so the monitoring path is optional rather than something every deployment inherits.

Rate control exists at four different levels

The most operationally interesting feature in the list is multi-dimensional speed control. The framework lets you throttle crawling at the Pipeline level, at the Request level, at the Download level and at the Domain level, and each of those is a different question. A per-request delay slows one page, a per-domain delay spreads load against a single host so you do not hammer one origin, a download-level limit caps concurrent fetches, and a pipeline limit keeps the processing side from outrunning storage. That last distinction is the one people usually get wrong when they build this themselves, because a crawler that downloads fast and persists slowly will exhaust memory or disk long before it exhausts the target's patience. The framework also includes a proxy pool, a user agent pool and a cookies pool, a failure retry mechanism, and a debug mode where a request can be switched to use a local cache so the same page is not fetched twice while you are investigating it.

URL deduplication uses a Bloom filter, and requests can be added at runtime

Deduplication is a Bloom filter, which is a deliberate memory trade rather than an exact set. A Bloom filter gives you a bounded structure that never produces a false negative, so a URL judged as seen is treated as seen, while a small fraction of new URLs will be wrongly rejected. In exchange the memory cost is constant and small, which is what you want when the visited set spans a large crawl. The feature list also says that requests can be added dynamically to the queue of a running crawler, so a crawl can be steered while it runs rather than being fully specified upfront, and that deep crawling is supported by starting new requests from inside a pipeline. Together those two features turn the framework from a fixed list of pages into something that can follow a discovered graph, which is the case most crawls eventually need.

Resolving the artifacts is the first problem you will hit

The installation instructions in the documentation are the weakest part of this project, and they will cost you time. For a Gradle build, the README says that because jcenter() is not used by default, you have to configure it in the relevant module's build.gradle, and it gives the repositories block to add.

repositories {
    jcenter()
}

The Maven equivalent points at a bintray URL inside the repositories declaration.

<repositories>
    <repository>
        <id>central</id>
        <name>bintray</name>
        <url>https://jcenter.bintray.com</url>
    </repository>
</repositories>

JCenter and Bintray have both been discontinued, so these instructions describe a path that no longer resolves. The version table at the top of the documentation links to bintray for the latest version of each module, which means the version numbers are not readable from the documentation either. Before writing a line of crawler code, confirm where the netdiscovery artifacts are published and whether they are still available, and treat that as a go or no go question rather than a setup detail.

Where NetDiscovery is the wrong choice

The first case is a greenfield crawler with no legacy constraints, because the framework's value is the breadth of its plug-in ecosystem and the pipeline isolation, and both of those are worth much less when you could write a small crawler on a modern HTTP client with a few hundred lines. The second case is a team that does not want RxJava or Vert.x in its dependency graph, since both come along by design and neither is optional. The third case is a project that needs JavaScript rendering beyond what the shipped downloaders give, because Selenium and HtmlUnit are in the list but the TODO list still includes adding Chromium support and enhancing the HtmlUnit module, so a modern headless browser is not part of the current capability set. The fourth case is a legal or compliance sensitive crawl where proxy pools and user agent rotation are a liability rather than a feature. The README itself asks learners to follow the law, which is a reasonable warning, and the project also carries a sponsored section promoting a commercial residential proxy service.

Editorial conclusion

Java teams that need a crawler whose queue, downloader and pipeline can be swapped for the components they already run, and who can live with an RxJava 2 and Vert.x 3 base, are the people NetDiscovery serves. It is the wrong tool for a team starting a new crawler today, because the documented Gradle and Maven setup points at jcenter and at a bintray URL, and that hosting path no longer works, so the first task is finding a live repository for the artifacts. It is also the wrong tool if you need a maintained ecosystem around it, since the repository has no GitHub releases, the last push was on 2026-06-05, and the TODO list still ends mid-sentence. Check three things before you invest: that you can resolve the netdiscovery modules from a repository that is still up, that your JVM and Kotlin tooling matches the Vert.x 3 and RxJava 2 generation the code targets, and that the queue implementation you need exists, since the shipped choices are Disruptor, Redis, Kafka, RabbitMQ and RocketMQ rather than a general plug-in contract. The licence is Apache-2.0.

Frequently asked questions

Official sources

1. fengzhizi715/NetDiscovery on GitHub
2. Issues
3. License: Apache-2.0
4. README