Darkdump OSINT Search and Deep Web Scraping

Darkdump's OSINT Workflow and Intended Users

Darkdump addresses the problem of collecting and inspecting dark web search results through one interface. A user supplies a query, chooses a search engine, sets a result limit, and can then request deeper extraction from each matching site. The project targets legitimate security research and open-source intelligence investigations, particularly investigators who need to search Tor-hosted indexes and collect public page data for later analysis.

The tool covers a broader path than a conventional search interface. It can retrieve search results, request site content through Tor, extract metadata, links, email addresses, documents, and images, and stream findings to a terminal or browser page. It also offers a separate breach and credential leak intelligence scan mode. That feature is aimed at investigation, but its availability means authorization and target scope matter throughout the workflow.

Darkdump does not present every index as equally selective. Ahmia is the default and does not require Tor. Not Evil offers a broader index but has only partial filtering. Tordex, Tor66, OnionLand, and Excavator are described as unfiltered engines that require confirmation. Even those searches are checked against Ahmia's public abuse blacklist. The blacklist reduces exposure to known abuse entries, but the project explicitly says that this strategy is not perfect.

How Darkdump Moves Queries Through Search and Scraping

The visible architecture has two presentation paths built around a shared set of OSINT operations. The `darkdump-cli` command provides flags for search, proxy routing, scraping, image collection, deduplication, output, debugging, and breach intelligence. The `darkdump` launcher starts a local browser interface, while `python3 darkdump-web/app.py` provides a direct entry point to that application. Both routes offer access to the search and extraction functions rather than making the CLI an isolated command-line wrapper.

A basic search begins with `-q` and uses `-a` to set the number of results. The engine is selected with `-e`, and Ahmia is selected when the flag is omitted. Tor routing is a separate decision made with `-p`. Deep scraping is enabled with `-s`; adding `-i` includes images but requires scraping to be active. This separation makes the cost of a search explicit in the command rather than silently applying site-level extraction to every result.

The output path is also configurable. `-o` saves results and infers the format from `.json`, `.csv`, or `.txt`. In the web interface, the documented features include live streaming, an engine selector, a Tor proxy toggle, a live exit IP display, metadata-based deduplication, breach scan mode, and JSON, CSV, or TXT export. The README does not expose internal module boundaries, queue behavior, or storage internals, so the exact concurrency and parsing architecture cannot be confirmed beyond these interfaces.

Installing Darkdump and Running an Ahmia Search

The documented Linux and macOS installation path clones the repository and delegates dependency and launcher setup to `install.sh`. On macOS, the script might require `sudo`. After installation, the README says the process creates both the `darkdump` and `darkdump-cli` commands.

git clone https://github.com/josh0xA/darkdump
cd darkdump
chmod +x install.sh
./install.sh
darkdump

The `darkdump` command starts the primary local web interface. The README directs users to open `http://127.0.0.1:50001` after the server is running. That address is bound to the loopback interface rather than presented as a public service.

For a first CLI search without Tor, use Ahmia. The command requests ten results for `privacy tools`.

darkdump-cli -q "privacy tools" -a 10

A manual installation is also documented. It installs the root and web-interface dependency files but does not show a subsequent launcher-creation step.

git clone https://github.com/josh0xA/darkdump
cd darkdump
pip3 install -r requirements.txt
pip3 install -r darkdump-web/requirements.txt

The root requirements pin packages including `beautifulsoup4==4.12.3`, `requests==2.32.3`, `click==8.1.7`, and `PySocks==1.7.1`. The repository layout also contains `darkdump.py`, `headers/`, `darkdump-web/`, `install.sh`, and `uninstall.sh`.

Configuring Tor Control for Darkdump's Proxy Mode

Darkdump's Tor setup requires more than installing the Tor process. The project documentation calls for enabling a control port and setting a hashed control password in `torrc`. It identifies `/etc/tor/torrc` as the usual Linux location and notes that the file can also exist on macOS.

ControlPort 9051
HashedControlPassword [YourHashedPasswordHere]

The placeholder must be replaced with a password hash generated through Tor. This is a configuration value supplied by the administrator, not a literal secret to copy from the README.

tor --hash-password "my_password"

The documented platform-specific Tor commands are:

sudo apt install tor

For systems using Homebrew, the README gives:

brew install tor

The service commands are also platform-specific:

sudo systemctl start tor.service

brew services start tor

Tor is not required for Ahmia. It becomes relevant when selecting an onion-only engine or using `-p` for proxy routing. The README does not document Windows installation, automatic `torrc` modification, or recovery behavior when the control port is unavailable, so users following this path must verify those setup details in their own environment.

Darkdump's Engine Choice Changes Both Access and Filtering

Darkdump offers a meaningful choice between a strict default and indexes with broader coverage. Ahmia is identified as Tor Project-endorsed, filtered, and available without Tor. It is therefore the sensible first engine for a user who wants the documented safety mechanism rather than maximum index breadth.

Not Evil is the closest documented alternative within Darkdump. It is described as an Ahmia fork with a broader index, partial filtering, and a Tor requirement. A query can be routed through it, deduplicated, and saved as JSON with this command:

darkdump-cli -q "security research" -a 20 -e notevil -p -u -o results.json

The difference is approach rather than output format. Ahmia applies strict filtering and can serve searches without Tor. Not Evil accepts a broader result set through a Tor-routed fork, while still receiving Darkdump's Ahmia blacklist check. That may expose an investigator to more relevant listings, but it also removes the guarantee of Ahmia's stricter filtering.

The other four indexes broaden scope further. OnionLand is described as indexing Tor, I2P, and the clearnet, while Tordex and Tor66 are presented as uncensored or crawled indexes. Excavator is described as a general dark web index. Since these choices can affect what a user sees, selecting an index should be based on the investigation's need for coverage rather than habit. Darkdump requires confirmation before running an unfiltered engine, but confirmation does not change the underlying content risk.

Darkdump's Deep Scraping Boundaries and Failure Cases

Deep scraping increases the work performed per search result. With `-s`, Darkdump attempts to collect metadata, links, emails, and documents from each result. With `-i`, it also collects images. The image option depends on scraping, which prevents a standalone image pass through that documented flag combination. Deduplication can hide repeated title and description pairs with `-u`, while `-o` stores the selected result format.

The documented combined scrape demonstrates the full dependency chain:

darkdump-cli -q "marketplaces" -a 15 -s -p -i

This request uses Tor, retrieves fifteen results, scrapes each one, and adds image collection. That is a substantially broader operation than the Ahmia example. The README does not state how Darkdump handles unreachable sites, malformed HTML, large files, hostile downloads, duplicate content with different metadata, or pages that require JavaScript. The pinned `beautifulsoup4` and `html5lib` dependencies confirm an HTML parsing stack, but they do not establish browser-level rendering.

Darkdump is also the wrong choice when an investigator needs guaranteed source coverage, a documented chain of custody, or a crawler with formal retry and scheduling controls. The project describes result counts and extraction categories, but it does not claim completeness, deterministic ordering, or preservation of original page evidence. Researchers who need those properties would have to establish them separately. The breach options add another boundary: the feature can query a target, and `--breach-delay` changes delay between breach queries, but the README does not identify every supported data provider or explain its accuracy.

Darkdump's Release State, License, and Upgrade Cost

The repository is not archived, but its last push was on 2026-04-08. The release list also identifies `Darkdump-4` on 2026-04-08, following `Darkdump2.0` from 2022-12-28 and `1.0` from 2021-02-11. That release history shows substantial time between the earlier versions and the latest listed release, so upgrading from an older Darkdump may involve more than a small patch. The README does not provide migration notes or a compatibility matrix.

The root `requirements.txt` uses exact pins for several core packages while leaving others unpinned. That creates a mixed dependency policy: important HTTP, parsing, CLI, and proxy packages are fixed to specific versions, but the repository does not explain why the version set should be changed or how upgrades are validated. Web-interface dependencies are maintained separately in `darkdump-web/requirements.txt`, adding another file to inspect before an upgrade.

Darkdump is distributed under the MIT License, with the README identifying copyright holder Josh Schiavone. The MIT license permits broad use, modification, distribution, and private reuse under its stated conditions, while preserving the copyright and license notice. That is a license summary rather than legal advice. Operators should review the actual `LICENSE` file before redistribution or incorporation into a larger product.

Editorial conclusion

Adopt Darkdump for legitimate OSINT work when a local CLI or web interface, Tor-aware search, and optional site extraction fit the investigation. Do not treat it as a guaranteed, complete, or court-ready crawler. Verify engine filtering, Tor control configuration, output handling, and breach-scan authorization before collecting or sharing results.

Frequently asked questions

Official sources

1. Issues
2. josh0xA/darkdump on GitHub
3. License: MIT
4. README
5. Releases