# Crawler: an Elixir crawler that separates queueing, fetching, parsing and offline rewriting

> Crawler is a web crawler and scraper written in Elixir, built for archiving a site and its assets with worker pooling, OPQ rate limiting and configurable depth, timeout and retry policy. The architecture is unusually well decomposed, while the documentation leaves installation and configuration defaults partly for the Hex docs.

**fredwu/crawler** — A high performance web crawler / scraper in Elixir.

- Repository: https://github.com/fredwu/crawler
- Stars: 961 · Forks: 88
- Language: Elixir
- License: not declared
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/fredwu-crawler

## What problem Crawler solves, and who reaches for it

Crawler is a web crawler and scraper written in Elixir, built around two ideas the README names directly: worker pooling and rate limiting via OPQ, the rate limiter maintained by the same author. It is aimed at the job of walking a site and pulling down more than HTML. It can crawl assets such as JavaScript, CSS and images, save the result to disk, and expose a hook for scraping content out of the pages it visits.

The audience is engineers who want a site mirrored with a predictable policy rather than a quick scrape. Every knob that matters for a long crawl is present as an option: restrict which domains, paths or content types are crawlable, cap concurrent crawlers, cap the rate, set the maximum crawl depth, set timeouts, set a retry strategy, set the user agent, and pause, resume or stop the crawler manually.

What separates it from a one-off HTTP client is the crawl store and the offline publishing layer. Saved pages keep their assets, and Crawler.Snapper rewrites links and publishes files while leaving unrelated source text untouched. For anyone mirroring a documentation site or an internal dashboard for offline reading, that rewriting step is the point of the project, not the fetching.

## Queue, Store, Fetcher and Parser each own one responsibility

The architecture section of the README is unusually specific about which process does what, and it is worth reading closely because it explains the failure modes later.

Crawler.Queue owns the workers and the rate limiter for each managed queue, Crawler.QueueHandler enqueues work, and Crawler.Worker runs each fetch and parse. Crawler.Store exposes a registry and coordinates updates through an internal server, with modules under lib/crawler/store/ handling page slots, pending work, redirect aliases and file publication. Crawler.Fetcher applies crawl policy and records responses, and Crawler.HTTP checks every redirect before Req follows it, which means redirect handling is a policy decision rather than a library default.

The Parser side discovers links and invokes the scraper. It also gathers more than links: image buttons, audio and video preloads, document and worker preloads, paint and audio worklets, CSS image() values and the older touch icons are saved alongside the page. CSS and JavaScript scanners share decoded link values and original source boundaries with the offline rewriting step, so a link found inside a stylesheet is treated consistently with one found in a script.

Two scoping rules complete the picture. A scope shares seen URLs and the page budget, and a scope generation prevents stale workers from writing back into a scope that has been reset. Queue ownership decides which processes Crawler.stop/1 shuts down, and a caller can supply an external queue instead of letting Crawler manage one.

## Starting a crawl and reading the page back afterwards

A crawl is launched with a URL and an options list. The README gives this as the opening example:

```elixir
url = "https://elixir-lang.org"
{:ok, opts} = Crawler.crawl(url, max_depths: 2, store: Crawler.Store)
```

Note that crawl returns options rather than a result. The work is asynchronous, so the pattern for consuming it is to poll a running check until it reports false while the crawl is not paused, and then read the processed page back:

```elixir
page = Crawler.Store.find_processed({url, opts[:scope]})
```

That lookup returns nil if the page was never processed, so nil is the signal that the page is missing rather than an error to raise.

The store option is the first decision to make. The default of store: nil keeps crawl metadata without retaining response bodies, which is enough to track scope and status but not to read what was fetched. Passing Crawler.Store retains the bodies, and they can then be read through the Store API or through its Crawler.Store.DB registry. Setting :save_to instead writes crawled pages to disk, and a custom scraper or parser module can be supplied when the built-in ones do not fit the site.

The documentation points to the Hex documentation for the full API, and there is no install command printed in the README, so the package is picked up through the standard Hex dependency flow in mix.exs.

## Options that decide how wide and how fast the crawl runs

The configuration table in the README is the most concrete part of the documentation, and several defaults are worth arguing with before you start.

Concurrency defaults to ten workers, and the rate limit option:interval defaults to 0, which the table describes as effectively no rate limit. In practice that means a fresh install will hit a site as fast as the network allows, with ten concurrent connections and no delay between pages. It is a workable default for your own small site and a poor one for anything shared, so:interval is the first value to set on a real crawl.

Depth defaults to three nested levels, and:max_pages defaults to:infinity, so nothing stops an unexpectedly large crawl except the depth cap and your patience. The HTTP receive timeout defaults to 5000 milliseconds and is passed to Req as:receive_timeout. Asset fetching is off by default and is enabled per type with the:assets list, whose options are "css", "js" and "images". A related option,:javascript_goal, selects the source goal for direct JavaScript crawls, where:module is the default and:script covers classic JavaScript.

The project publishes Hex documentation separately for exactly this kind of detail.

## Page identity decides when two URLs count as the same page

A crawler that treats the same page as two different pages will fetch it twice, waste the page budget, and produce inconsistent output. Crawler handles this with a URL module that validates browser host spellings and defines page identity, and the rules are specific enough to be worth knowing.

Page identity keeps one trailing DNS dot and normalizes other browser-equivalent domain and IPv4 spellings. A path pipe, a bare percent sign, and the query characters backtick, brace, pipe, backslash and caret stay literal, and they do not share a page with the percent-encoded form of the same character. Other bytes that require escaping do share an identity with their encoded form, while escaped separators remain distinct. HTTP requests encode Unicode and the remaining unsafe path or query bytes into ASCII, while preserving valid escapes and an explicit empty query, so a URL ending in a bare ? stays meaningful.

Saved links keep their fragments after URL sanitization, and delimiter bytes in those fragments are escaped so that a fragment cannot break out of its context in HTML, CSS or JavaScript source. That last detail matters more than it sounds. Fragments drive client-side routing in single page applications, and rewriting them carelessly is a common way for an offline copy to stop behaving like the original.

## Offline rewriting, asset saving and source map handling

The publishing layer is where Crawler does more than fetch. Crawler.Linker builds offline paths, and Crawler.Snapper rewrites links and publishes files while preserving unrelated source text. Assets are saved with the page rather than treated as a separate pass, which covers image buttons, audio and video preloads, document and worker preloads, paint and audio worklets, CSS image() values and the older touch icons.

Source maps get explicit treatment. A source map named by a crawled script or stylesheet is saved with the sources that map names. The same applies to a map identified by its SourceMap or X-SourceMap header. In practice this means the offline copy keeps its original sources reachable, which is the difference between a mirror you can debug and one that only renders.

The design trade-off is visible in the dependency on Req. The HTTP layer checks each redirect before Req follows it, and the timeout is handed to Req as :receive_timeout. That keeps the HTTP surface small and familiar, but it also means the behaviour you inherit from Req is part of the behaviour you inherit here, and the README does not document how to substitute a different client.

## Limits in scope, in time and in maintenance

The scope constraints are visible in the option list. There is no built-in sitemap handling described, no robots.txt policy described, and no per-host politeness delay beyond the single global:interval value, so a crawl of several hosts is governed by one rate setting for all of them. The default of no rate limit and unbounded page count are the two settings most likely to produce an unhappy server-side reaction on a first run.

The timeout default of 5000 milliseconds is also a real constraint. It applies as a receive timeout, so a slow endpoint that streams headers late is treated as a failure.

On maintenance, the repository is not archived and the last push was on 2026-10-20, but the published releases stop earlier: v1.5.0 on 2023-10-10, v1.4.0 on 2023-10-07 and v1.3.0 on 2023-09-30. So the working tree is being changed while the version history has not advanced since October 2023. Anyone pinning to a release is pinning to code from 2023, while anyone using master is taking unreleased changes. The repository links a LICENSE.md file from its README badge, but no licence identifier is recorded, so redistribution terms cannot be read off what is published here.

## Conclusion

Crawler fits teams that already work in Elixir and want to archive a site with its assets intact, especially when offline link rewriting and saved source maps matter. It does not fit a request-response scraping job, and it is the wrong tool when you need a polite crawl without tuning:interval yourself. Before adopting it, check the mix.exs dependency list for the Elixir and Req versions you are pinned to, confirm how:interval is set against the target site, and read the Hex documentation for the full option list.

## FAQ

### How do I install Crawler in an Elixir project?

The README does not print an install command and instead points to the Hex package page and the Hex documentation for the API. The package is published on Hex as crawler, and the dependency is declared through the standard mix.exs dependency flow for Hex packages.

### How do I limit how many concurrent crawls Crawler runs?

Set the :workers option, which is an integer with a default of 10, giving the maximum number of concurrent workers for crawling. The separate :interval option controls the rate limit as milliseconds between crawling more pages, and it defaults to 0, which the README describes as effectively no rate limit.

### Can Crawler fetch CSS, JavaScript and images along with the page?

Yes, through the :assets option, a list whose available options are "css", "js" and "images", and which defaults to an empty list. Assets are saved with the page, including image buttons, audio and video preloads, document and worker preloads, paint and audio worklets and CSS image() values.

### How do I read the content of a crawled page?

Pass store: Crawler.Store when you start the crawl, since the default of store: nil keeps crawl metadata without retaining response bodies. Once the crawl has stopped running, read the page with Crawler.Store.find_processed({url, opts[:scope]}), which returns nil when the page was not processed.

### Does Crawler save source maps for scripts and stylesheets?

Yes. A source map named by a crawled script or stylesheet is saved with the sources that map names, and the same handling applies to a map referenced by its SourceMap or X-SourceMap header.

### What happens if I want to stop a running Crawler job?

The crawler can be paused, resumed and stopped manually. Which processes Crawler.stop/1 shuts down is decided by queue ownership, so a caller that supplies its own external queue keeps control of what gets torn down.

## Sources

- [fredwu/crawler on GitHub](https://github.com/fredwu/crawler)
- [Issues](https://github.com/fredwu/crawler/issues)
- [README](https://github.com/fredwu/crawler/blob/master/README.md)
- [Releases](https://github.com/fredwu/crawler/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/fredwu-crawler