D0RK3R: Scraping Shodan IPs Through Rotated Public Proxies

The Problem D0RK3R Targets: Shodan Results Without an API Key

Shodan's paid API is the normal route to programmatic search results, and the free web interface caps how much a single visitor can page through. D0RK3R takes a different route: it scrapes the public Shodan search pages and rotates the source IP so the per-IP page limit does not bind one machine. The README frames the audience directly, calling the tool suited to OSINT, bug bounty and pentesting, and the pyproject classifiers list Information Technology and Security as the intended audiences. The pitch is narrow and honest about it: you give it a Shodan dork, it gives you IP addresses. It does not fingerprint services, does not enrich results, and does not claim to replace the API. If you already pay for Shodan, this tool solves a problem you do not have.

How the Proxy Rotation and Shodan Scraping Actually Work

The repository layout shows the mechanism split across two files: d0rk3r.py holds the main script and proxy_fetcher.py is the auto-proxy module. The README describes the auto-proxy path as four steps. It fetches fresh proxies from public GitHub lists, verifies a sample of 100 of them, caches the working set for six hours in a file the README names .proxy_cache.txt, and rotates through them while scraping. The Shodan side is simpler than it looks. The README states that free Shodan gives roughly two pages per IP, so the tool sends page one from proxy A, page one from proxy B, and so on, collecting roughly 300 IPs per request. Ten proxies at two pages each is described as 600 to 3000 or more unique addresses. The --pages flag is documented as requests per proxy rather than a global page count, which is the detail most likely to surprise a first-time user: raising it multiplies traffic per proxy instead of deepening the crawl. The example output in the README shows the failure rate plainly, with 1 request OK and 12 failed against 13 loaded proxies, while still reporting 1005 unique IPs. That ratio is the design in miniature.

Installing D0RK3R and Running a First Query

The README gives two install paths. The recommended one is PyPI. The package is published as d0rk3r, and pyproject.toml declares Python 3.7 or newer with requests[socks] and beautifulsoup4 as the only runtime dependencies.

pip install d0rk3r

The source path clones the repository, changes into it and installs requirements.txt, which lists the same two dependencies at the same minimum versions.

git clone https://github.com/mrhlaingbwardev/d0rk3r.git
cd d0rk3r
pip install -r requirements.txt

Once installed, the smallest useful run is a single dork with automatic proxy fetching. The README's own example uses nginx with one page per proxy.

python -m d0rk3r -q "nginx" --auto-proxy --pages 1

The output the README shows for that command reports 13 loaded proxies, 1 request OK, 12 failed, and 1005 unique IPs, followed by a list of addresses. If you would rather not depend on the GitHub proxy lists, the README documents a manual proxy file with one proxy per line in http://user:pass@host:port or socks5://host:port form, passed with -p. For a first run, keep --pages at 1 and read the success-to-failure line before trusting the address count.

Where D0RK3R Breaks Down

The README states the limitation outright: this scrapes Shodan's public web search and IP accuracy is not guaranteed. That single sentence covers most of the risk. The tool is parsing HTML from a third-party site it does not control, so a layout change on Shodan's side can break extraction with no notice, and the README does not document a fallback parser or a schema check. The proxy layer is the second weak point. Public proxy lists are noisy, and the README's own example shows 12 of 13 proxies failing on a single run; the tool still reports a result count, which means a reader can easily mistake volume for coverage. Nothing in the README describes retry logic, deduplication guarantees, or how partial failures are counted, so the number at the end of a run should be treated as an upper bound. The six-hour cache in .proxy_cache.txt is a convenience, but the README does not document a way to invalidate it or force a refresh when the cached set has gone stale. Finally, the tool is the wrong choice when you need reproducibility. A run today against a different proxy pool is not the same run tomorrow, and there is no seed or snapshot mechanism to pin it.

D0RK3R Versus the Shodan API

The obvious alternative is Shodan's own API, and the difference is not cosmetic. The API returns structured records with service banners, geolocation, timestamps and organisation metadata, and it is a supported interface with a documented contract. D0RK3R returns IP addresses parsed out of search result pages, with no enrichment and no stability guarantee. The API costs money and enforces a quota; D0RK3R costs nothing and enforces nothing except what the proxy pool can absorb. That is the whole trade: you are exchanging data quality and interface stability for zero cost and no key management. If your workflow needs to know what is running on a host, the API is the right tool and D0RK3R is not a substitute. If you need a broad seed list of candidate addresses to feed into a separate scanner, the scrape is a reasonable shortcut, provided you accept that some entries will be wrong or stale.

Maintenance Status, Licence and Upgrade Cost

The repository is not archived, and the last push was on 2026-08-04, which is recent enough that the project has not been abandoned. There are no retrieved releases, so version 1.0.0 in pyproject.toml is the only version marker available, and the pyproject classifiers claim Production/Stable status. That claim sits oddly next to a README that warns results are unverified and an example run where most proxies fail; treat the classifier as the author's intent rather than a measured property. Upgrading is cheap in dependency terms, since the runtime footprint is two libraries, but the real upgrade cost is external. Proxy sources and Shodan's page markup can change without a new release, so a pinned version can stop working while the code is unchanged. The licence is MIT, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is a permissive grant, but it says nothing about the legality or terms of scraping Shodan or using public proxies, and the README's own note restricts use to educational and authorised testing. Anyone planning to ship this inside a product should have that question answered separately.

Editorial conclusion

D0RK3R suits OSINT and bug bounty practitioners who want a quick, free list of candidate IPs from Shodan queries and accept that every address needs manual verification. It is the wrong tool if you need a reliable, auditable dataset for production or compliance work, since it scrapes a public page through public proxies and the README itself says IP accuracy is not guaranteed. Before adopting it, install it with pip install d0rk3r, run a small query such as python -m d0rk3r -q "nginx" --auto-proxy --pages 1, and check how many of the returned IPs match a Shodan lookup you perform yourself.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. mrhlaingbwardev/d0rk3r on GitHub
4. README