ahmia-site: the Django search engine that indexes .onion addresses

Three repositories make up one search service

Ahmia is the search engine for `.onion` domains on the Tor anonymity network, led by Juha Nurmi and based in Finland. The repository in question, ahmia-site, contains the site and the crawlers used by the search engine.

The split across repositories is the first thing to understand. This repository is the web application. The crawling is done by a separate project, ahmia-crawler, and the search index is provided by a third, ahmia-index, which the documentation lists as a prerequisite that should be installed and running. That means the operational picture is three moving parts, not one application you clone and start.

The stack is named in the compatibility section: the newest version of Ahmia is built with Python 3, Django 5 and Elasticsearch 8, and the documentation is blunt that you will need to know these technologies to create a working installation. The requirements file pins the three direct dependencies: Django at 5.2.16, elasticsearch at 8.17.1 and python-decouple at 3.8.

The licence is the 3-clause BSD licence, and the default branch is master with no GitHub releases, so the deployment is from source rather than from a tagged artefact.

Configuration is a file copy and an env file you have to edit

The configuration step is the same for local development and for production, and it starts with copying an example file into place:

cp ahmia/example.env ahmia/.env

The documentation is emphatic that the values in `.env` must be modified to fit your needs, and specifies the minimum: you have to specify at least the postgresql credentials if you are using the production settings. The example file is named `example.env` rather than `.env.example`, which is a small difference from the usual convention but is what the copy command expects.

The presence of python-decouple in the dependency list is what makes this file meaningful. Decouple parses environment variables and settings files into settings objects, which is how the same codebase can read configuration from a file in development and from the process environment in production.

That two-source design is why the documentation describes the copy as a common step for both local and production environments. The development path and the production path differ in the values, not in the mechanism, and the same `.env` file structure serves both.

The database migration and static file steps are standard Django

Once configured, the setup is the standard Django project sequence.

The database is migrated with the usual two commands, scoped to the `ahmia` app:

python manage.py makemigrations ahmia
python manage.py migrate

Static files are collected with:

python manage.py collectstatic

There is then a step that is specific to the Elasticsearch deployment: copying the Elasticsearch CA certificate into place, with the source path given as `/usr/local/share/ca-certificates/http_ca.crt`. That file is there because the site talks to Elasticsearch over a connection whose certificate has to be trusted by the system store, and the certificate has to land in the right directory for that to work.

For development, the setup is short. The development settings use sqlite as a database, and the documentation says the default settings work out of the box, with the server started by:

python manage.py runserver

So the local experience is a single command once the environment file exists, and the production path adds nginx, gunicorn and systemd on top of the same Django project.

Production means nginx, gunicorn and two systemd services

The production deployment is documented in two halves, and it is worth reading as a whole because the halves interact.

Nginx is configured by copying the shipped configuration and the map files into place, then starting the service:

cp *.map /etc/nginx/
cp conf/nginx/nginx.conf /etc/nginx/nginx.conf
cp conf/nginx/django-ahmia /etc/nginx/sites-enabled/django-ahmia
service nginx start

The documentation notes a dependency on the deployment directory being `/usr/local/lib/ahmia-site`, and asks you to replace that path if yours differs, which tells you the configuration is written for that layout rather than generated for your host.

It also asks you to raise `worker_connections` in the nginx configuration, with the example setting it to 2048. That is a throughput setting for the reverse proxy, and it is a specific suggestion rather than a general one, which suggests the default is too low for the connection counts the service is expected to handle.

The application side is gunicorn, and there are two documented ways to run it. The first is the bash scripts, which run as daemons and whose files you edit to change the configuration:

bash ./bin/run-ahmia.sh
bash ./bin/run-ahmia-onion.sh

The second is the systemd route, with unit files copied into place and the services enabled, including the separate onion service and a restart of gunicorn at the end. The two services are the point: one serves the clearnet site and one serves the onion version.

The systemd route is noted as tested with gunicorn==21.2.0, which is the kind of detail that saves time when a newer gunicorn behaves differently.

The onion service has to start after the app, and onions are deleted weekly

Two operational details stand out, and both are specific to running a service that also answers on the Tor network.

The first is the systemd ordering between nginx and the two Ahmia services. The documentation asks you to edit the nginx unit so that the Ahmia services are pulled in:

systemctl edit nginx

[Unit]
After=network-online.target remote-fs.target nss-lookup.target ahmia.service ahmia-onion.service
Requires=ahmia.service ahmia-onion.service

systemctl daemon-reload
systemctl cat nginx

The `After` and `Requires` directives together mean nginx does not start until both the clearnet and onion application services are up. Without that, a reboot can leave the reverse proxy running with nothing behind it.

The second is a scheduled cleanup of onion addresses, and the crontab entry is given in full:

# Every three days
40 5 */3 * * cd /usr/local/lib/ahmia-site && venv/bin/python manage.py deleteonions >> weeklydelete.log 2>&1

The management command is `deleteonions`, the schedule is every three days at 05:40, the virtual environment is invoked explicitly as `venv/bin/python` rather than through an activated shell, and the output is appended to a log file.

That last detail is a scheduled task that exists because onion addresses churn. A service that indexes hidden services has to remove addresses that have gone away, and doing it on a schedule rather than on request keeps the index from filling with dead entries. The schedule being every three days rather than daily is a choice about how stale the index is allowed to get, and the log file is the only trace of what was removed.

Where the search stack stops and the policy begins

Three things set the boundaries of this project, and none of them are in the code.

The first is that the content is not the site's to decide. Ahmia indexes `.onion` addresses, which means it is indexing content it does not host and does not control, and a search engine over that content carries obligations that a normal site does not. The `banned_terms.map` file at the repository root is the visible part of that: a map file of terms to filter, deployed to nginx as part of the configuration step. The filtering happens at the proxy layer rather than in the application, which is a deliberate architectural choice, since it means the terms are removed before a request ever reaches Django or Elasticsearch.

The second is the Tor dependency. The service is a front end for the Tor network, and a meaningful part of its value is that it makes that content reachable without the user running Tor themselves. Running your own instance means running two services and a crawler, and the onion-facing variant needs a working Tor setup that the documentation does not describe in the visible portion.

The third is operational. Elasticsearch 8 with a trusted certificate, PostgreSQL for the application data, a separate index service and a crawler that has to run for the index to be useful. None of this is unusual for a search service, but the total is a multi-service system rather than a single deployable, and the documentation says as much when it tells you to know Python 3, Django 5 and Elasticsearch 8 to get a working installation.

The alternative to name is a plain Elasticsearch deployment with a thin search interface, which gives you the same query capability and none of the Tor-facing policy work. The difference in approach is that ahmia-site is a complete opinionated service for one specific corpus, and reproducing it means accepting that corpus and the filtering rules that go with it.

Editorial conclusion

Adopt ahmia-site if you are running a search service for Tor hidden services and want the same Django and Elasticsearch stack that the Finnish operated instance uses, or want to study a production Django and Elasticsearch deployment with an onion-facing variant. Do not adopt it expecting a turnkey search product, since the crawler, the index service and the content policy work are all separate repositories and there is no packaged distribution. Verify first that you can run the ahmia-index service, that your Elasticsearch instance uses a certificate the site can trust, and how the banned terms filter is applied before you expose an indexing endpoint.

Frequently asked questions

Official sources

1. ahmia/ahmia-site on GitHub
2. Issues
3. License: BSD-3-Clause
4. Project website
5. README