URLFinder: passive URL gathering from public archives instead of live crawling

Passive means it never sends a request to the target domain

URLFinder is described as a high-speed tool for passively gathering URLs, optimized for efficient and comprehensive web asset discovery without active scanning. That word passive is the whole design. Instead of crawling a domain and asking it for pages, the tool queries public sources that have already indexed those pages and reads the URLs back out. The stated audience is penetration testers, security researchers and developers who want to gather URLs without active scanning. The consequence is immediate and worth understanding before you use the output. You get a picture of what has been indexed, which is a historical and third-party view of the asset, not a live view. A URL that exists in an archive may be gone from the site, and a live endpoint that was never indexed will be missing entirely. The tool is therefore a fast way to assemble a candidate list, not a way to establish what a server currently exposes.

The feature list is four items long, and one of them is scope control

The documented features are curated passive sources to maximize comprehensive URL discovery, support for multiple output formats in JSON, file and stdout, optimization for speed and resource efficiency, and stdin and stdout support for easy integration into existing workflows. The stdin and stdout point is the one that changes how you use the tool. Because it reads targets from standard input and writes results to standard output, it composes with other ProjectDiscovery tools and with shell pipelines rather than being an isolated binary you run and then parse. The output formats follow the same idea. You get plain URLs on stdout for a human, a file with -o for a result set, and JSONL with -j or --jsonl when a downstream program needs structure. Each JSONL line carries the URL, the input it came from and the source that produced it, which is what makes it possible to weight results by where they came from.

Scope filters decide whether a result is in, and the default is a registrable domain match

The scope system is the part of the tool that deserves attention, because passive sources return URLs for neighbouring infrastructure whether you want them or not. There are two layers. The -us and -uos flags take in-scope and out-of-scope regular expressions that the tool follows, and the -fs field-scope flag accepts a pre-defined scope field, dn, rdn or fqdn, or a custom regular expression, with the default being rdn, which is the registrable domain name. The -ns flag disables the host based default scope entirely, and -do displays external endpoints that turned up during scoped crawling, which is how you find the third-party links your default scope would otherwise hide. The practical advice is to think about which scope you actually want before running anything, because with a passive tool the default filter decides what you see and the difference between rdn and fqdn is the difference between a company's own sites and every subdomain it uses.

Match and filter take files as readily as comma separated lists

Refinement happens after collection with two flags that mirror each other. The -m and -match option keeps URLs matching a pattern, given as a file or a comma-separated list, and -f and -filter removes them. The documented examples are simple. Matching on shop,model keeps URLs containing those strings, filtering on privacy,terms drops them, and combining -m support with -f faq finds URLs mentioning support but not faq. Because both accept a file as well as a list, the pattern set can live in a file and be reused across many domains, which is how you keep an exclusion list consistent across a whole assessment rather than retyping it per target. Note the order the flags are named in the help text, match first and filter second, and note that both are plain string or regex matching rather than anything aware of path structure.

Installing needs Go, and the module and the README disagree about the version

URLFinder is a Go project and can be installed from source with the standard install command:

go install -v github.com/projectdiscovery/urlfinder/cmd/urlfinder@latest

Pre-compiled binaries are also published on the releases page. There is a version inconsistency worth knowing before you start. The README states that URLFinder requires Go 1.21, while the go.mod in the repository declares go 1.25.0. The module manifest is what the toolchain actually enforces, so on a machine with Go 1.21 the install will fail on the module directive rather than on anything the README warned you about. There is also a container route. The Dockerfile builds the command from the cmd/urlfinder package on a golang alpine builder image and ships it on alpine, adding bind-tools and ca-certificates, with the entrypoint set to urlfinder. If you use the Go module path rather than the binary release, you will need a toolchain new enough to satisfy go.mod.

Rate limits exist on purpose, because the sources are shared infrastructure

The rate limiting flags are not a nicety. The -rl and -rate-limit flag sets a global maximum of HTTP requests per second, and -rls and -rate-limits sets per-provider limits, with the example given being a named provider at a value such as 15 per minute. Separately there is a -timeout flag in seconds with a default of 30 and a -max-time flag in minutes for enumeration with a default of 10. The reasoning is straightforward once you understand the architecture. URLFinder is a client of other people's services, so hammering a source with parallel requests is how you get blocked, and a shared archive serving your request at full speed is a service that other users depend on. The per-provider limit also protects against one slow or failing provider eating your entire enumeration budget, which is exactly what the -max-time cap exists for. A passive tool that hammers its sources is a bad neighbour, and the defaults reflect that.

A library package sits alongside the command, which changes how you can use it

The repository is laid out with cmd, internal and pkg directories, and the README badge points at a Go reference page for the package at pkg/urlfinder inside the module github.com/projectdiscovery/urlfinder. That means URLFinder is not only a binary. The discovery logic is exposed as a Go package you can import and drive yourself, which is how you would build a tool that runs many domains concurrently, feeds URLs into your own pipeline, or embeds the source registry with custom handling. The go.mod shows the dependency set that makes this possible. There is projectdiscovery/utils for shared helpers, projectdiscovery/goflags for the flag parsing that produces the grouped help output, projectdiscovery/gologger for the INF style logging, projectdiscovery/ratelimit for the throttling, projectdiscovery/fdmax for file descriptor limits, and projectdiscovery/dnsx in the dependency list. The repository also ships a Makefile with build, test and tidy targets, and a .goreleaser.yml for the release builds.

The results inherit the blind spots of whatever archives are configured

The limitation to hold onto is that URLFinder can only report what its sources know. The tool exposes that directly through -ls, which lists all available sources, and -stats, which displays source statistics, and the -cs or -collect-sources flag, which includes all sources in the JSON output so you can see the attribution per URL. Any provider you have not enabled, or any source that has not indexed your target, simply contributes nothing, and there is no error to tell you that. The example run in the README shows the shape of a result: a banner, the version line, an enumerating line naming the domain, then a stream of URLs, and a final count with the elapsed time. Whether the count is impressive or thin depends entirely on the sources, which is why -stats and -cs are the flags to reach for before drawing conclusions. The tool also supports updating itself with -up, and disabling that check with -duc, which matters on air-gapped or locked-down machines where an update check is just noise.

Editorial conclusion

URLFinder suits a security researcher or developer who wants a list of URLs for a domain assembled from public archives and then filtered down with regular expressions, particularly as a cheap complement to a live crawler rather than a replacement for one. It does not suit someone who needs to know what is reachable right now, because the tool never touches the target, so the results are only as good and as current as the archives feeding it. Before you rely on it, run it with -ls to see which sources are configured on your machine, compare its output against a live enumeration of the same domain, and read the source statistics with -stats to see whether any single provider is carrying the whole result set.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. projectdiscovery/urlfinder on GitHub
4. README
5. Releases