Lookyloo: capturing a page to map the domains that call each other

A capture turns a page into a tree of calling domains

Lookyloo is a web interface that captures a webpage and then displays a tree of the domains that call each other. The project description says the same thing more plainly: a web interface to track the trackers.

That framing tells you what the tool is for. When you open a page in a normal browser, the network activity is a flat list of requests. Lookyloo keeps the structure: who asked for whom, which host is first party and which is a third party, and how a chain of redirects and script loads connects them. The tree is the output, and the capture is the raw material.

The audience is analysts and researchers. The packaging classifiers name Science/Research, Telecommunications Industry and Information Technology as the intended audiences, and the topic list covers capture, dfir, information-security, scraping, web-security and privacy. The name is spelled out in the README from an Urban Dictionary entry about onlookers, which is a light joke but not a bad description of the workflow.

There is a public instance at lookyloo.circl.lu and the project homepage is lookyloo.eu, with documentation hosted on the same domain under a docs path. The README shows a sample capture of github.com as the illustration of a typical tree, and the repository carries a Dockerfile plus a docker-compose file, which is the intended way to run it.

One caveat belongs at the start. Lookyloo renders pages, so it fetches content from wherever you point it. The topics include scraping and the tool is meant for investigating suspicious sites, so run it on infrastructure you are willing to let make outbound requests to untrusted hosts.

The REST API is self documented and reachable on the demo instance

Automation support is a first-class part of the project rather than an afterthought. The API is described as self documented with swagger, and the README points at the demo instance so you can try the endpoint explorer directly at lookyloo.circl.lu/doc/.

That matters for how Lookyloo fits into a workflow. Capturing a URL by hand through a web form is the interactive path, but the interesting use is scripted: feed a list of URLs through the API, collect the trees, store them, and compare across time or across cases. The capture tree is the artifact that gets archived, and the API is how a pipeline produces it.

The repository layout shows the moving parts. There are separate top-level directories for cache/, indexing/, full_index/, known_content/, known_content_user/ and kvrocks_index/, which indicates that the capture data, the derived indexes and the known-content matching are stored separately from the web application code in lookyloo/ and website/. There is also a data/ directory and a user_agents/ directory, the latter carrying a User-Agent list credited in the README to a tech blog post from 2012.

The separation between captured content and indexes is the architectural decision with the most operational consequence, because it is where the storage volume grows and where the documented failure mode lives. The background indexing work is exposed through console entry points, including background_indexer, background_indexer_archives and background_full_indexer, so indexing is a separate process from capture rather than something that happens inline while you wait for a tree to render.

pylookyloo is the recommended way to drive a Lookyloo instance

The README states plainly that pylookyloo is the recommended client to interact with a Lookyloo instance, and that it is available on PyPI.

pip install pylookyloo

That is the whole installation step for the client side, which is the easiest part of working with Lookyloo. The package has its own documentation on readthedocs, an overview page in the Lookyloo docs, and a GitHub repository of its own under the Lookyloo organisation, so the client is maintained separately from the server.

The server side is documented separately. The README does not give an inline installation procedure and instead refers to an install guide on the documentation website, which is where the Python version requirements and the configuration steps are described. The packaging metadata puts the floor at Python 3.10 and the ceiling at 3.15, so the project supports a narrow band of interpreter versions.

For anyone building a pipeline, the division of labour is clear. Put pylookyloo in your analysis code, point it at a Lookyloo instance, and let the instance handle the capture and the tree. The project also exposes a public instance, so a script can be developed against lookyloo.circl.lu before your own deployment exists.

The compose stack runs valkey caches, a web port and a set of poetry entry points

The repository ships a docker-compose file that shows the deployment shape more concretely than any prose would. Three services are defined: a redis-cache service and a redis-indexing service, both using the valcky/valkey image with their own configuration files and their own volumes, plus the lookyloo service itself, which is built from the local Dockerfile.

The lookyloo service maps a single host port, 5100, and mounts named volumes for cache, indexing, scraped, archived_captures, discarded_captures, user_agents, config and two log directories. The two valkey services are linked to it. The container command runs poetry run start and then tails the LICENSE file, which keeps the container alive.

The Dockerfile shows what the image needs. It starts from ubuntu:22.04, installs Python, git, poetry and a list of Chromium-style shared libraries including libnss3, libgbm1, libcups2 and libasound2, then runs poetry install and installs Playwright with its dependencies. The browser stack is not incidental: Lookyloo renders pages, so it needs a real browser engine, and the Playwright setup is how that engine gets into the image.

The application itself is driven through console scripts declared in pyproject.toml. The list is long and descriptive, and among them are start, stop, update, shutdown, run_backend, async_capture, archiver, processing, background_build_captures, background_indexer, background_full_indexer, start_website, scripts_controller and mastobot. That set tells you the shape of the system: a capture path that can run asynchronously, a processing stage that builds captures into trees, a background indexer, an archiver, and separate website and backend processes.

S3FS versioning caps the index file at 10.000 versions and breaks writes

The most useful part of the README is the section on S3FS storage, because it documents a failure that is hard to diagnose from the error message alone.

The problem starts with directory listing. The notes are blunt about it, stating that it is slow. If you keep many captures, say more than 1000 per day, in an s3fs bucket mounted with s3fs-fuse, running a directory listing in bash will most probably lock the I/O for every process trying to access any file in the whole bucket. The same is true for Python filesystem methods such as iterdir and scandir. The suggested workaround is the Python s3fs module, which does not touch the filesystem to list directories, configured through the s3fs key in config/generic.json. The warning is that this will not save you if you run ls on a directory that contains a lot of captures.

The deeper trap is versioning. By default a MinIO bucket, the backend for s3fs, keeps versioning enabled, which means it keeps a copy of every version of every file stored. Because the index files are updated on every change and the maximum number of versions is 10.000, once a directory holds more than 10.000 captures you get I/O errors when the index file is updated. The notes are clear that you do not care about that versioning in Lookyloo, and the fix is to suspend it:

mc version suspend <alias_in_config>/<bucket>

You can check the current state with mc version info against the same alias and bucket. If you already hit the ceiling, the recovery path is to delete the older versions of the affected index file:

mc rm --non-current --versions --recursive --force <alias_in_config>/<bucket>/Year/Month/Day/index

The Python equivalent of the same operation uses the s3fs configuration read from the Lookyloo config:

from lookyloo.default import get_config
import s3fs

s3fs_config = get_config('generic', 's3fs')
s3fs_client = s3fs.S3FileSystem(key=s3fs_config['config']['key'],
                                secret=s3fs_config['config']['secret'],
                                endpoint_url=s3fs_config['config']['endpoint_url'])

s3fs_bucket = s3fs_config['config']['bucket_name']
s3fs_client.rm_file(s3fs_bucket + '/Year/Month/Day/index')

The error that points at this is unhelpful from bash, where a removal fails with an Input/output error, while the Python path reports the real cause as MaxVersionsExceeded, meaning the number of versions created for that object has passed the limit.

Regular releases, an active push, and a licence that reads two different ways

The project is not archived and the last push was on 2026-10-19. The release history is dense and consistent: v1.42.0 on 2026-10-05, v1.41.1 on 2026-09-26 and v1.41.0 on 2026-08-18. The packaging metadata pins the version to 1.42.0, which matches the newest tag, so the two do not drift.

The licence needs a careful reading because the two sources disagree in form. The pyproject metadata declares license as BSD-3-Clause with an MIT-style classifier set alongside a classifier for the tool's purpose as Topic :: Security and Topic :: Internet. The repository licensing information, on the other hand, is listed as a custom licence that GitHub cannot classify, and the README simply says to see the LICENSE file. The practical instruction is to read that file rather than assume the SPDX identifier in the metadata covers everything, especially if you intend to redistribute a deployment.

Other operational details are worth knowing before you commit to it. The project has a Security Policy for reporting vulnerabilities, a code of conduct, and a pre-commit configuration, which points to a repository with real development hygiene rather than a script collection. The lookyloo directory holds the application package, website/ holds the web frontend, and the config/ directory holds the configuration files the S3FS notes refer to.

For a tool that is meant to sit in an analysis pipeline and keep a searchable history of captures, that combination of frequent releases, an active push and a documented storage pitfall is a reasonable basis for a decision. The scale you plan to run at is what decides whether the S3FS section becomes a footnote or your main operational concern.

Editorial conclusion

Lookyloo is worth running when you need to answer what a suspicious page loads and who calls whom, because the tree view compresses a capture into something a human can read, and the REST API makes the result scriptable. It is the wrong tool for anyone who only wants to check a URL in a browser, since the interesting parts are the self-hosted stack and the capture pipeline you have to operate. Before you deploy it, verify three things yourself: that your storage layer does not reproduce the S3FS index versioning trap, where a limit of 10.000 versions on the index file produces I/O errors, that you can reach the container port 5100 in your network plan, and that the licence in the LICENSE file matches what you need, since the packaging metadata declares BSD-3-Clause while the repository licence itself is one GitHub cannot classify.

Frequently asked questions

Official sources

1. Issues
2. Lookyloo/lookyloo on GitHub
3. Project website
4. README
5. Releases