xurlfind3r and passive URL discovery for a domain

Passive means it never talks to the target

xurlfind3r is a command-line utility designed to discover URLs for a given domain in a simple, efficient way, and the mechanism is the passive sourcing. The project description says it works by gathering information from a variety of passive sources, meaning it doesn't interact directly with the target but instead gathers data that is already publicly available.

That single design decision shapes everything about the tool. There is no crawling, no port scanning, no probing of discovered hosts. Every URL it reports was indexed by somebody else at some earlier point, which means the results are a historical record rather than a live view. A path that was removed last month may still be listed, and a path added this morning may be missing.

The stated audience is security researchers, IT professionals, and anyone looking to gain insights into the URLs associated with a domain. The repository's topic tags place it in the OSINT, content discovery, reconnaissance and bug bounty tool categories.

The feature list is short and specific: it fetches URLs from multiple online passive sources to provide extensive results, it supports `stdin` and `stdout` for easy integration in automated workflows, it supports multiple output formats with JSONL, file and stdout named, and it is cross-platform across Windows, Linux and macOS.

The stdin and stdout support is the one that matters most in practice. A tool that reads a list of domains from standard input and writes JSON lines to standard output slots into a pipeline or a shell loop without any file handling of its own, which is what makes it usable across a list of hundreds of domains rather than one at a time.

Source selection is a first class part of the interface

The help output shows that the CLI is organised around a small number of concepts, and the sources section is the one that reveals how the tool actually works.

There is a `--sources` flag that lists the supported sources, and then two filtering flags, `-u, --sources-to-use` and `-e, --sources-to-exclude`, both taking comma separated lists of source names. So the tool can be run with everything, restricted to a chosen set, or run with a known noisy source removed.

That matters because passive sources have very different characteristics. Some are historical archive indexes, some are current, some cover a narrow set of URL patterns, and some are noisy. A researcher who knows that one source returns a flood of irrelevant URLs can exclude it, and a researcher who needs a specific source can demand it.

The input side is similarly flexible. There is `-d, --domain string[]` for the target domain and `-l, --list string` for a file of target domains, and the help text spells out the options: for multiple domains use comma separated values with `--domain`, specify multiple `--domains`, load from file with `--list` or load from stdin. There is also a `--include-subdomains` boolean to match subdomain URLs, which is off by default.

That flag is the difference between a URL list for one host and a URL list for a whole domain, and it changes the result set by orders of magnitude. Turning it on when you did not mean to is a good way to get a list you cannot use.

Configuration is separate from the domain list, and lives under `-c, --configuration string` with a default of `$HOME/.config/xurlfind3r/config.yaml`.

API keys are what stand between you and the full source list

The post installation section is the one that tells you what the tool cannot do on its own.

The documentation says xurlfind3r will work right after installation, however, some sources require API keys to work. These keys can be added to a configuration file at `$HOME/.config/xurlfind3r/config.yaml`, created upon first run, or set as environment variables.

The environment variable form is:

XURLFIND3R_KEYS_BEVIGIL=your_bevigil_key
XURLFIND3R_KEYS_ONTELX=your_intelx_key

The naming pattern is worth reading carefully. It is a prefix for the tool, then `KEYS`, then the source name in upper case. So every keyed source follows the same convention, and the two named in the example, Bevigil and IntelligenceX, are the two the documentation chooses to show. There is no indication of how many more require keys, which is a bit of a gap, since a source that silently returns nothing is indistinguishable from a source that has no data for your domain.

The configuration file is created on first run, which means you run the tool once before you can populate it. That is a small sequencing detail that trips people up: the file path in the help output is the default, and the file does not exist until the first run creates it.

The tool also has a rate limiter and a retrier in its dependency list, and the HTTP layer is a separate module by the same author. That is worth noting for a tool that queries many third party APIs: the sources will rate limit you, and the tool is built to cope with that rather than to hammer them.

The project also publishes an MCP server as a separate repository, so the same discovery can be driven by an agent rather than only from a terminal.

Three install paths, and the release binaries are the easy one

The installation section covers three routes, and the fastest one does not need Go at all.

For release binaries, you visit the releases page and find the archive for your operating system and architecture, then download it with wget or curl. The curl form keeps the original filename, so the archive is retrieved as `xurlfind3r-<version>-linux-amd64.tar.gz`, and it is then extracted with:

tar xf xurlfind3r-<version>-linux-amd64.tar.gz

The documentation notes that the download and extract steps can be combined into one, and shows a piped version that pipes the archive into tar. On Windows you can double-click the zip to extract the executable.

Either way the binary then has to go somewhere in your PATH, and the suggested location on GNU/Linux and macOS is:

sudo mv xurlfind3r /usr/local/bin/

Windows users are pointed at the Microsoft documentation on adding tool locations to the PATH environment variable.

For source installs with Go present, the one-line route is:

go install -v github.com/hueristiq/xurlfind3r/cmd/xurlfind3r@latest

The development build clones the repository, builds from the `cmd/xurlfind3r` directory with `go build .`, and moves the binary as above. The documentation carries a caution on this route: the development version is a good way to take a peek at the latest features before release, but it may have bugs, and officially released versions will generally be more stable.

The Docker route pulls `hueristiq/xurlfind3r:latest` and runs it. The Dockerfile is a two stage Alpine build that ends up running as an unprivileged `runner` user in a `runners` group, with `bind-tools` installed in the runtime stage.

What passive discovery cannot tell you

The limitation is inherent to the method, and it is worth being blunt about it because it is the reason to reach for a different tool as well.

A passive tool cannot confirm that a URL currently exists. Everything it returns was seen by a third party at some point, which means the list contains paths that have since been removed, paths that were never publicly reachable and were only exposed by a mistake, and paths that exist on a completely different infrastructure now. There is no timestamp on the result and no way to distinguish these cases from the tool's own output.

The corollary is that the tool is a starting point rather than an inventory. The usual workflow is to take the passive list, then verify the interesting parts against the live host with something that actually sends requests. Doing that verification with a tool that speaks HTTP is a different category of activity, and the boundary is worth being deliberate about.

There is a second limitation in the source model. Because results come from third party indexes, the completeness of your answer depends on which indexes you have access to. A source behind an API key that you have not configured contributes nothing, and a source that has stopped indexing contributes nothing, and in both cases the tool gives you no signal that it asked and got nothing. For a reconnaissance task where coverage matters, that is a real weakness, and it is why the source include and exclude flags exist at all.

A third, smaller one is the tool's own scope. It is a domain-oriented URL finder, and it does not attempt to do host discovery, port scanning, vulnerability scanning or data extraction. Anyone expecting a broader reconnaissance suite in one binary will not find it here.

Against other discovery tools, the difference is where the lookups happen

The realistic alternative is a different passive discovery tool, and the difference in approach comes down to the source set, the interface and the surrounding engineering.

The sources are the substantive difference. Any two passive tools query overlapping sets of public indexes, and the overlap is large, so the interesting comparison is not which tool finds more URLs but which sources each one can reach. A tool that ships with fewer keyed sources out of the box will give you a smaller list unless you sign up for accounts, and a tool with a configurable source list lets you decide that trade yourself.

The interface is the second difference. xurlfind3r reads from stdin and writes to stdout, and offers JSONL as a first class output. That combination is what makes it scriptable in a way that a tool which insists on its own output file or its own config format is not. JSONL in particular is the format that a downstream script can consume line by line without parsing a whole document, which is the right choice when you are processing thousands of URLs.

The engineering around the edges is the third, and it is where the author's other modules show. The dependency list includes a dedicated HTTP client module, a rate limiter, a retrier, a logger and a URL utility, all under the same `hq-go` namespace. For a tool that fires many requests at many third party APIs, having rate limiting and retrying as first class components rather than as an afterthought is what keeps it from getting itself blocked.

So the honest way to choose is by source coverage and by how you intend to consume the output. If your workflow pipes URLs into another tool and you already have accounts with the sources that matter, this fits well. If you want a single command that also resolves hosts, probes ports and does the verification step, you want a broader reconnaissance tool, and this is the wrong size of tool for the job.

Editorial conclusion

xurlfind3r fits a security researcher doing reconnaissance on an authorised target, or an IT professional mapping a domain's public surface, who wants URL lists gathered from third party indexes rather than by sending requests to the target. It is the wrong tool if you need to confirm whether a path actually exists on the live host, because that is exactly what a passive tool cannot tell you. Before adopting, verify three things: which sources you can get keys for, since some require an account, what the default source set is, because the help output offers both an include list and an exclude list and the behaviour of the default matters, and how the tool handles the keys file at `$HOME/.config/xurlfind3r/config.yaml`, which is created on first run. The code is MIT licensed, the current release is 1.3.0 from 2025-11-17, and the last push was on 2026-10-19.

Frequently asked questions

Official sources

1. hueristiq/xurlfind3r on GitHub
2. License: MIT
3. Project website
4. README
5. Releases