cariddi: crawl a domain list and scan for endpoints, secrets and API keys

What cariddi does that a generic crawler does not

A crawler that only returns URLs leaves the analysis to you. cariddi folds the analysis into the crawl. You feed it a list of domains, it follows links, and while crawling it matches what it finds against several hunting modes: juicy endpoints, secrets, API keys and tokens, errors, useful information, and file extensions ranked by how interesting they are. The README frames the whole tool in one line: take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more.

The audience is narrow and practical. The repository topics list bugbounty, penetration-testing, recon, redteam and osint, and the usage examples assume you already work on the command line with target lists. If your workflow is a shell pipeline that moves text files between tools, cariddi is shaped for that. If your workflow is a dashboard, it is not.

One design decision worth noting: results go to stdout by default, and the tool offers -plain to print only results, -json for structured output, -ot and -oh to write TXT or HTML files. That is a deliberate choice to stay composable rather than own the reporting layer.

How the crawl and the scanning modes fit together

The mechanism is a pipeline of three stages visible in the repository layout. Input arrives on stdin, either a single URL echoed into the binary or a file of URLs. The crawler, built on github.com/gocolly/colly/v2 per go.mod, walks the pages and collects links and resources. The scanners then test what the crawler collected against the enabled hunting modes.

Depth is bounded by -md, which sets the maximum depth level the crawler will follow from the initial target URL. Breadth is bounded by -intensive, which the README describes as crawling while searching for resources matching the second level domain, the equivalent of *.target.com. Concurrency defaults to 20 per the help output, and -d inserts a delay between one crawled page and the next. Those three knobs are the difference between a scan that finishes in seconds and one that hammers a host.

The scanners are not always on. Secrets need -s, errors need -err, endpoints need -e, information needs -info, and file extensions need -ext with an integer from 1 (juicy) to 7 (not juicy). Both endpoint and secret hunting accept custom pattern files through -ef and -sf, which matters because the built-in patterns are a starting set, not a complete one. The default ignore list is long: png, svg, jpg, jpeg, bmp, jfif, gif, webp, woff, woff2, ttf, tiff, tif, mp4, webm, mkv, avi, mov, flv, wmv, mp3, wav, flac, ogg, m4a, aac, ico, cur, eot and otf are skipped while scanning for secrets, info and errors. That is sensible for noise reduction and also a blind spot if a target hides data in an image or font file.

Installing cariddi and running a first scan

The README lists five package managers plus a source build. Homebrew, Snap, Pacman and NixOS each get a one-liner:

brew install cariddi

If you have Go, the module path installs the latest release directly. The README states you need Go 1.24.0 or newer for a source build, and go.mod declares go 1.25.0, so a current toolchain is the safe choice:

go install -v github.com/edoardottt/cariddi/cmd/cariddi@latest

For a first run, the README gives a single-target example. The output is the crawl and scan result on stdout:

echo https://edoardottt.com/ | cariddi

For more than one target, put one URL per line in a file. The README uses urls.txt as the example name:

cat urls.txt | cariddi -s -e

That combination turns on secret hunting and endpoint hunting. To see what the tool is doing while it crawls, add -debug; to keep only the findings and drop the surrounding formatting, add -plain. On Windows the README notes the executable works only in the cariddi folder and gives `cat urls.txt | cariddi.exe` from PowerShell as the equivalent.

Where cariddi stops being the right tool

The most concrete limitation is that cariddi does not execute JavaScript. The crawler is built on colly, an HTTP-based scraping library, and nothing in the README or the dependency list suggests a headless browser. On a single-page application where the interesting API routes are assembled at runtime, the crawler sees the shell of the page and the bundles it loads, not the endpoints the application actually calls. You will get the JavaScript files and whatever URL strings appear inside them, which is useful, but it is not the same as driving the app.

Second, secret detection is pattern matching over crawled text. The README documents custom secrets through -sf, which is an admission that the built-in set is a starting point. Pattern matching produces false positives on test fixtures and false negatives on any credential format nobody has written a rule for. There is no validation step that confirms a found key is live, and the README does not claim one.

Third, the default ignore list means the scanners skip images, fonts, video and audio. If your target's problem lives in a metadata-bearing image or a font file, the default configuration will not look at it, and you would need to override -ie to change that.

Finally, there is no persistence layer. Results are printed, optionally written to TXT or HTML, or emitted as JSON. There is no database, no diff between runs, and no scheduling. cariddi is a step in a pipeline, not the pipeline.

cariddi against a general-purpose web crawler

The obvious comparison is a general crawler such as the colly library cariddi itself uses, or a scraping framework like Scrapy. Those tools give you a programming model: you write extraction logic, you control the queue, you decide what to store. cariddi gives you a fixed set of hunting modes behind flags. The difference in approach is who writes the matching rules. With a framework, you do. With cariddi, the project does, and you extend them through -ef and -sf files when the built-ins miss.

That trade is the whole point. A framework is more flexible and demands more work to reach a first useful result. cariddi produces findings on the first command you type. It also means you inherit the project's idea of what counts as juicy, and if that idea does not match your target, your only lever is the custom pattern files plus the ignore flags.

The same logic applies against a full attack-surface platform. Those systems store results over time, correlate assets and re-scan on a schedule. cariddi does none of that, and in exchange it has no server, no account and no data leaving the host you run it on. For a single engagement or a quick pass over a new scope, that is the better trade. For continuous monitoring of an estate, it is not.

Maintenance, upgrades and the GPL-3.0 licence

The repository is not archived, and the last push was on 2026-09-21. Releases are infrequent but real: v1.4.6 on 2026-03-29, v1.4.5 on 2026-01-11 and v1.4.4 on 2025-11-13. Between v1.4.4 and v1.4.6 the project shipped two minor releases across roughly four and a half months, so the upgrade cadence is measured in months rather than weeks. If you pin a version, expect to revisit it a few times a year, not every sprint.

Upgrading is cheap because there is no state to migrate. Homebrew, Snap, Pacman and NixOS users update through their package manager; Go users re-run the go install command at the latest tag. The Makefile adds a convenience target, `make update`, which removes the installed binary, pulls the latest modules and git changes, and reinstalls to /usr/bin/cariddi. That target writes to /usr/bin via sudo, which is worth knowing before you run it on a shared host.

The licence is GPL-3.0. For individual pentesters and internal security teams running the binary, that is unremarkable. For anyone embedding cariddi's code inside a closed-source product, the copyleft terms are the thing to read, and that reading is for your legal team rather than this article. The scan patterns and custom endpoint or secret files you write are your own content, separate from the tool's source.

Editorial conclusion

Adopt cariddi if you already pipe target lists between CLI recon tools and want endpoint, secret and error hunting in one binary that installs through Homebrew, Snap, Pacman, NixOS or go install. Do not adopt it if you need a persistent scan database, scheduled re-scans, or a browser-driven crawler that executes JavaScript. Before trusting it on a client engagement, verify three things: that your Go build is at least 1.24.0 as the README states, that the default ignore list does not hide the asset types you care about, and whether -cache writing a .cariddi_cache folder is acceptable on the host you run it from.

Frequently asked questions

Official sources

1. edoardottt/cariddi on GitHub
2. License: GPL-3.0
3. Project website
4. README
5. Releases