# TorCrawl.py, a command line crawler that routes through Tor

> TorCrawl.py is a Python script that fetches and crawls web pages, either ordinary or onion, over a SOCKS5 proxy that defaults to a local Tor instance. It prints to standard output so the output can be piped, and it adds the things an OSINT workflow needs: yara keyword search, a link graph in SQLite and an HTML visualisation of it.

**MikeMeliz/TorCrawl.py** — Crawl and extract (regular or onion) webpages through TOR network

- Repository: https://github.com/MikeMeliz/TorCrawl.py
- Stars: 543 · Forks: 95
- Language: Python
- License: GPL-3.0
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/mikemeliz-torcrawl-py

## One argument, standard output, and a pipe

TorCrawl.py is described as a Python script designed for anonymous web scraping via the Tor network, and the project puts its emphasis on being simple for someone who lives in a terminal.

The stated argument is that things have to be simple and clear for a terminal user, that passing the output into other tools is necessary, and that accuracy is the key. The design that follows is a single argument for the URL, output to standard output, and nothing that requires a browser or a GUI.

```shell
$ torcrawl -u http://www.github.com/ | grep 'google-analytics'
    <meta-name="google-analytics" content="UA-XXXXXX- "> 
```

The output of the fetch goes to the terminal, so the natural thing to do with it is pipe it into grep, or into a file, or into anything else that reads standard input. That is the design constraint that shapes the whole tool, and it is why the extractor mode exists separately from the crawler mode.

The tool reads a regular URL or an onion one over the same argument, and the routing is the part that is specific to it. The proxy options are the IP address for a SOCKS5 proxy, with a default of 127.0.0.1 for using Tor, and the port, with a default of 9050. Those are the standard local Tor SOCKS ports, so with the defaults the script talks to a Tor daemon on the same machine and nothing else needs configuring.

There is also a flag to turn the Tor routing off entirely, the `-w` flag, which is useful for the cases where you want the extraction behaviour without the network hop, and for testing.

The project is MIT-licensed in the repository metadata, while the setup script declares the GNU General Public License v3.0 as the package licence, and the primary language is Python with the topics naming crawler, extractor, onion, osint, python and tor. The last push to the repository was on 2026-06-29 and the recorded releases are v1.35 from 2026-01-03, v1.34 from 2025-12-28 and v1.33 from 2025-12-25.

## Installation, and the Tor service you have to bring

There are two installation routes, and one of them is still a placeholder.

From PyPI it is one command.

The package installs from PyPI with `pip install torcrawl`.

Homebrew is listed as coming soon, which is a small sign of the project's maturity, since a Tor-adjacent tool with a Homebrew formula is a natural thing for someone to want.

The manual route is three steps: clone the repository, install the dependencies with `pip install -r requirements.txt`, and install and start the Tor service. The Tor part is where the platform differences are, and the documentation gives each one.

On Debian and Ubuntu it is `apt-get install tor` followed by `service tor start`. On Windows you download `tor.exe` and then run `tor.exe --service install` and `tor.exe --service start`. On macOS it is `brew install tor` and `brew services start tor`. For other distributions the documentation points at the Tor setup documentation.

The Windows path is the one worth noting, because installing a Tor service on Windows and running it as a service is not the same operation as starting a daemon, and the two-command form with the service flag makes that explicit.

The dependency list is short and each entry has an obvious purpose.

```text
pysocks
beautifulsoup4>=4.15.0
yara-python
lxml
networkx
pyvis
```

`pysocks` is the SOCKS5 client, which is how the traffic reaches Tor. `beautifulsoup4` and `lxml` are the HTML parsing and the extraction, which is what the extractor mode is built on. `yara-python` is the rule engine behind the keyword search, which is the feature that makes this an OSINT tool rather than a general crawler. `networkx` and `pyvis` are the graph library and the visualisation library, and they are there for the link graph output.

So the tool is a crawler with a built-in evidence workflow: fetch a page through Tor, search the HTML for indicators with yara rules, and map the link structure as a graph. That combination is aimed at someone investigating a site, not at someone archiving one.

## The argument surface, in three groups

The command line is documented as a table with three groups, and reading it is the fastest way to understand the tool's scope.

The general group covers the basics. `-h` prints the help and `-V` prints the version. `-v` turns on more information about the progress, which is what produces the banner output you see in a crawl. `-u` takes the URL of the webpage to crawl or extract, and the documentation marks it as accepting an onion address. `-w` runs without using the Tor network. `-f` names the directory that will contain the generated files.

Two of the general options are about looking less like a crawler. `-rua` enables random user-agent rotation for requests, and `-rpr` enables random proxy rotation from a list at `res/proxies.txt`, one proxy per line in host and port format. That one requires the `-w` flag, since random proxy rotation and Tor routing are alternatives rather than a combination.

The extract group is about getting one page. `-e` extracts the page's code to the terminal or to a file, with the terminal as the default. `-i` takes an input file with URLs, separated by line, so a list of targets can be processed. `-o` outputs a page to a file. And `-y` performs a yara keyword search, with `h` searching the entire HTML object and `t` searching only the text.

The crawl group is about walking a site. `-c` crawls the website, with the default output written to a file called `links.txt` under the website directory. `-d` sets the depth of the crawler's travel, with a default of 1. `-p` is the pause in seconds between requests, with a default of 0. `-j` exports the crawl findings to JSON in addition to the text output, and `-x` does the same for XML. `-DB` exports the findings and the link graph to a SQLite database, and `-vis` generates an HTML visualisation from that database, which requires the database option to have been used. `-l` names a log file that records the visited URLs and their response codes.

The output formats are worth noticing as a set. Text for a person, JSON and XML for a script, SQLite and an HTML visualisation for a graph view, and a log of what was visited and what came back. That is the difference between a scraper and a tool for reconstructing a site's structure.

## A crawl in practice, and what the output looks like

The crawl example in the README shows the whole thing at once, with the verbose flag on and both a depth and a pause set.

```shell
$ torcrawl -v -u http://www.github.com/ -c -d 2 -p 2
# TOR is ready!
# URL: http://www.github.com/
# Your IP: XXX.XXX.XXX.XXX
# Crawler started from http://www.github.com/ with 2 depth crawl and 2 second(s) delay:
# Step 1 completed with: 11 results
# Step 2 completed with: 112 results
# File created on /path/to/project/links.txt
```

Several things are visible there. The banner confirms that the Tor connection is up and prints the exit IP, which is a useful sanity check that the routing is doing what you think. The crawl then proceeds in steps, one per depth level, and reports the number of results at each step, so you can see the site growing as the crawler descends. The links are written to a file whose path is printed at the end.

The two-second pause in that example is worth comparing with the default of zero. A pause between requests is the difference between a crawl that is polite and a crawl that hammers a host, and against an onion service the pause matters more than usual, because the exit node and the origin are both carrying the load and the origin may not know who is asking.

The verbose output is also the place where the log file earns its keep, since a crawl of any depth will visit a mix of responses, and a record of which URLs returned what is what tells you afterwards which parts of the site were reachable.

For the yara search, the two modes are a sensible distinction. Searching the entire HTML object covers the raw markup, which is where indicators often hide, in script tags, in comments, in inline styles, in data attributes. Searching only the text is the version that gives you readable output. A rule set that matches on a script tag is how you find something that a visitor to the page would never see, and which is a different capability from the one a text search gives you.

The rule file, the proxy list and the user agent list are all package data, listed in the setup configuration as `keywords.yar`, `proxies.txt`, `user_agents.txt` and `regex_patterns.txt`, all under the `res` directory. They are shipped with the package, so they are editable, and the user agent list in particular is what the rotation feature draws on.

## The ethics note in the README, and the limits of the anonymity claim

The project states its own position on crawling, and it is worth quoting it because it is more nuanced than the marketing copy suggests.

The tip says that crawling is not illegal, but violating copyright is, that it is always best to double-check a website's terms and conditions before crawling it, and that some websites set up what is called `robots.txt` to tell crawlers not to visit those pages. Then it says that this crawler will allow you to go around this, but that the project always recommends respecting robots.txt.

So the tool can ignore the exclusion rules and the project says you should not. That is a candid framing: it is a capability statement plus a recommendation, and the decision sits with the user. It is also the reason the robots.txt handling is not something to rely on the tool for.

The anonymity claim needs a similar reading. The project describes the tool as enabling secure and untraceable data collection, and the Tor routing does what it says for the transport: your request reaches the origin through a Tor circuit, and the origin sees an exit node's address rather than yours. What it does not do is make the crawl unlinkable. The banner printing your exit IP is a reminder that you are being identified to the origin as a Tor user, which is a much smaller set than your real address but is not nothing.

Three practical limits follow from the architecture rather than from the code. The first is speed. A Tor circuit is three hops by default, and each hop adds latency, and the tool then pauses between requests on top of that. A crawl of the same site through Tor will take far longer than a direct crawl.

The second is that the exit node sees the traffic, and the exit node is not run by you. The connection is encrypted to the origin with the origin's own TLS, so the exit sees the hostname in the clear and the encrypted payload, which is the same position any network observer is in, but the exit is a single machine that specifically chose to be in that position.

The third is that the tool is a script, not a service. There is no scheduler, no politeness policy, no retry budget, no robots.txt compliance, no rate limiting beyond the manual pause flag. The output formats are rich, but the operational discipline is entirely manual. For a one-off investigation that is the right shape, and for anything that runs repeatedly against a site that does not want it, no amount of output formatting makes it appropriate.

## Packaging details, and where the project stands

The setup file is worth reading because it tells you how the tool is meant to be distributed and what the author's view of it is.

The package name on PyPI is `torcrawl`, at version 1.35, with the entry point defined as a console script named `torcrawl` pointing at `torcrawl:main`. So the installed command is the short one, which is why the README examples use `torcrawl` rather than `python torcrawl.py`, even though the repository's own examples in the usage section run the script file directly.

The classifiers describe the audience as developers, system administrators, information technology, education and science and research, and the development status as 5, Production/Stable. The Python requirement is 3.9 or later.

The repository layout is small and readable: `torcrawl.py` at the top level as the entry point, a `modules/` directory for the implementation, a `res/` directory for the packaged data files, an `output/` directory, `requirements.txt`, `setup.py` and a `tox.ini` for the test matrix.

The topic list in the repository metadata is osint, web-scraping, crawler, tor, onion, web-crawler, scraping, anonymous, privacy, security and data-extraction, and the keywords in the package metadata add onion, anon, python-notebook and others.

The alternatives to this tool are worth naming by category. For a general crawler, there are the mainstream scraping frameworks, which are faster, more configurable and considerably better tested, and which have no notion of Tor. For working with onion services specifically, there are the Tor browser and the curl and wget flags that speak to the SOCKS port, which cover the fetch half of the job without any crawling logic. And for the OSINT half, the graph and keyword search capabilities here overlap with a general purpose crawling and analysis stack.

The case for TorCrawl.py is that it puts those pieces together in one small script with a terminal-first interface, and the case against it is that the pieces it puts together are available separately and better elsewhere. If your work involves pulling a handful of pages through Tor and searching them for indicators, it is a good fit for that. If it involves crawling a large site, you want something built for the job, with or without Tor.

## Conclusion

TorCrawl.py suits an OSINT practitioner who needs to fetch a page or walk a site through a Tor exit without switching tools, and who wants the output in the terminal so it can be piped into the rest of a shell pipeline. It does not suit a production crawler, since the project presents itself as a script and its own description of crawl ethics points readers to each site's terms. Verify first that a Tor service is running and reachable on the SOCKS5 port, that the yara rules and proxy list in the res directory are what you intend to use, and that the depth and pause settings you choose will not put undue load on the target, since the default pause is zero seconds.

## FAQ

### What is TorCrawl.py?

It is a Python script for crawling and extracting web pages, regular or onion, through the Tor network. It prints to standard output so the result can be piped into other tools, and it adds yara keyword search, JSON and XML output, a SQLite link graph and an HTML visualisation of that graph.

### How do I install TorCrawl.py and Tor?

The package installs with pip install torcrawl, or by cloning the repository and running pip install -r requirements.txt. You also need a running Tor service: apt-get install tor and service tor start on Debian or Ubuntu, brew install tor and brew services start tor on macOS, or tor.exe --service install followed by tor.exe --service start on Windows.

### What options does TorCrawl.py take for crawling?

The crawl group includes -c to crawl, -d for the depth of the crawl with a default of 1, -p for the pause in seconds between requests with a default of 0, -j and -x to export findings to JSON and XML, -DB to export them and the link graph to SQLite, -vis to generate an HTML visualisation from that database, and -l for a log of visited URLs and their response codes.

### How does the yara keyword search work in TorCrawl.py?

The -y option performs a yara keyword search, with h searching the entire HTML object and t searching only the text. The rules are the packaged res/keywords.yar file, alongside proxies.txt, user_agents.txt and regex_patterns.txt, all shipped as package data and therefore editable.

### Does TorCrawl.py respect robots.txt?

No, and the project says so directly: the crawler will allow you to go around it, but the project always recommends respecting robots.txt. The same note reminds you that crawling is not illegal while violating copyright is, and that you should check a site's terms and conditions first.

## Sources

- [Issues](https://github.com/MikeMeliz/TorCrawl.py/issues)
- [License: GPL-3.0](https://github.com/MikeMeliz/TorCrawl.py/blob/master/LICENSE)
- [MikeMeliz/TorCrawl.py on GitHub](https://github.com/MikeMeliz/TorCrawl.py)
- [README](https://github.com/MikeMeliz/TorCrawl.py/blob/master/README.md)
- [Releases](https://github.com/MikeMeliz/TorCrawl.py/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/mikemeliz-torcrawl-py