vessel: a Ruby web crawling framework with two very different drivers

One crawler API over Chrome and over plain HTTP

vessel is a high-level web crawling framework for Ruby, and its central idea is that the fetching strategy should be a detail of the crawler class rather than the shape of the code you write. The project markets itself as fast as Chrome, and the mechanism behind that claim is the driver layer. Pages are fetched either by Ferrum, which drives a real Chrome and is the default, or by Mechanize, which does plain HTTP with no browser at all.

The stated use cases are broad: data mining, monitoring and historical archival. For automated testing, the project recommends Cuprite. That recommendation tells you something about the design, since a test suite usually wants to drive the browser directly rather than go through a crawler's scheduling and extraction layer.

The practical consequence is that a single crawler class can be pointed at a heavy single page application or a plain content site without rewriting the extraction logic. The trade-off is that the two drivers do not behave identically, which is the subject of a later section here.

Crawlers are classes, and settings travel by inheritance

A crawler in vessel is a subclass of Vessel::Cargo, and Vessel::Crawler is an alias, so both names work. Configuration lives in the class body rather than in a separate settings file, which is what makes the inheritance story interesting. Settings are inherited by subclasses and deep-copied into them, and that copying behaviour is the reason a shared ApplicationCrawler base class is useful: you define the common settings once and each concrete crawler receives its own copy rather than sharing one mutable object.

The settings table is wide enough to shape real behaviour. Domain and start_urls define the crawl. Driver and its options pick how pages are fetched. Delay, headers, cookies and allow_cookies shape the request. Threads sets the min and max size of the thread pools. Middleware is the pipeline that yielded items travel through, proxy is a rotation class, and blacklist and whitelist restrict which resources are fetched or allowed.

Every one of these can also be passed at runtime to the run method, which merges the runtime values on top of the class level ones.

From start_urls to a scheduled request

The request loop is short enough to read in one pass. Vessel uses the driver to visit the urls given in start_urls, waits for the page and its resources to load, and then invokes the handler registered for that url, parse by default. Inside the handler you select elements with a CSS selector, yield a Hash of what you extracted, and, if there is another page, yield a request with the absolute url and the handler to use for it.

The step that matters for scale is that yielding more than one request sends those requests to the driver, which loads the pages in parallel. All requests are scheduled and handled concurrently through a thread pool, with one page per core by default, and you can set a limit per class with the threads max option.

Start urls may be plain strings, an array of strings, or a hash mapping a url to a handler, so different entry points can have different parse methods. If start urls is empty the crawler still runs once with a stub response, which is a deliberate convenience for crawlers that compute their first url inside parse.

Two Gemfile lines and a browser on PATH

Installation is a Gemfile entry, nothing more. The default Ferrum driver also needs Chrome or Chromium installed and available in PATH:

gem "vessel"

The Mechanize driver is optional and ships as a sibling dependency, so it has to be requested explicitly:

gem "mechanize"
gem "vessel"

A first crawler looks like this, taken from the project documentation:

require "json"
require "vessel"
class QuotesToScrapeCom < Vessel::Cargo
  domain "quotes.toscrape.com"
  start_urls "https://quotes.toscrape.com/tag/humor/"
  def parse
    css("div.quote").each do |quote|
      yield({
        author: quote.at_xpath("span/small").text,
        text: quote.at_css("span.text").text
      })
    end
    next_page = at_xpath("//li[@class='next']/a[@href]")
    return unless next_page
    yield request(url: absolute_url(next_page[:href]), handler: :parse)
  end
end
quotes = []
QuotesToScrapeCom.run { |q| quotes << q }
puts JSON.generate(quotes)

Save that as quotes.rb and run it with bundle exec ruby quotes.rb, redirecting the output into quotes.json. One thing to expect on the first run is log noise, because Vessel logs to $stdout at the debug level by default. Either redirect that stream somewhere else or write your collected items to a file instead of standard output.

Mechanize raises where Ferrum hands you a status

The two drivers differ in ways that will change your code, not just your speed. Ferrum drives a real Chrome, so JavaScript runs and a page can be rendered before you parse it. Vessel passes its defaults into Ferrum::Browser.new: timeout 60, js_errors false, process_timeout 30, pending_connection_errors false, and certificate errors are ignored. Headless mode is the default and can be turned off at runtime:

MyScraper.run(driver_options: { headless: false })

A driver can be configured in the class body as well:

class MyScraper < Vessel::Cargo
  driver :ferrum, headless: true, timeout: 30
end

Mechanize is described as much faster and lighter because it makes plain HTTP requests with no browser and no JavaScript. The cost is a behavioural difference that catches people out: an error status is raised rather than parsed, so a 404 or a 500 is retried and ends up in the on_error callback instead of reaching your handler with response.status set. Blacklist and whitelist are not supported by this driver at all.

Delay only means something on a single thread

Polite crawling and parallelism pull in opposite directions here, and vessel resolves it by making delay conditional. The delay setting takes a number of seconds or a range, and a range is sampled randomly per request. It is applied only when the crawler is single threaded, because with a pool of more than one thread a shared delay no longer describes the actual spacing between requests to a given host.

So a polite configuration pairs the delay with a thread limit:

class MyScraper < Vessel::Cargo
  delay 4..6
  threads max: 1
end

The same values can be overridden at run time, and the runtime values are merged on top of the class level ones:

QuotesToScrapeCom.run(delay: 2, max_threads: 1)

Notice that the two forms use different names. The class body setting is threads max, while the runtime call in the documentation passes max_threads. That inconsistency is worth checking against your own code before you assume a delay is doing anything, because a delay silently ignored is indistinguishable from a site that stopped rate limiting you.

Custom drivers plug in through a named interface

The driver is an extension point rather than a fixed choice. Custom drivers are supported by subclassing Vessel::Driver, implementing start, stop and create_page along with a Vessel::Driver::Page subclass, and registering the result with driver_name and the chosen name. That is a small surface, which means the driver layer is thin enough that a different browser backend or an HTTP stack of your own can be added without touching the extraction code.

Around the driver sits the rest of the request shaping. Headers are set as a hash, with the following form:

class MyScraper < Vessel::Cargo
  headers "Content-Type" => "text/plain",
          "Referer" => "https://example.com"
end

Cookies are handled by cookie for a single value and cookies for an array, with the domain defaulting to the crawler domain. The allow_cookies setting decides whether cookies received from a page are kept for the next requests, and the domain setting doubles as the crawler name used by the CLI and for the loader registration.

Middleware, proxy rotation and resource filters

Three settings sit at the edges of the pipeline and are easy to overlook. Middleware defines the pipeline that yielded items go through, which is the place to add your own processing, filtering or validation without rewriting parse methods. Proxy takes a rotation class, so proxy handling is delegated rather than built into the settings table. Blacklist and whitelist hold patterns of the resources to block or to allow, which is how you stop a crawler from fetching analytics beacons or images that cost time and add nothing.

None of these are documented in depth in the project README, and the full reference lives under docs in the repository and at the documentation site the README links to. That is worth saying plainly before you build on them: the settings table names them, but the details are elsewhere, and a blacklist pattern that does not match the url form you expect will silently change your crawl coverage.

The repository also ships an example file, examples/quotes.toscrape.com.rb, and the top level includes a CHANGELOG.md, a Rakefile and a spec directory. The last push was on 2026-10-15, and the project publishes no GitHub releases, so version tracking runs through the CHANGELOG and the gemspec rather than through release notes. The license is MIT.

Editorial conclusion

Use vessel when you want to write one crawler class that can switch between a browser backed fetch and a plain HTTP fetch, and when a thread pool with per class settings fits your site. Do not pick it for sites that need a status code handled in your parser under Mechanize, since that driver raises on error statuses, and do not expect its blacklist and whitelist settings to work there. Before writing a parser, confirm that Chrome or Chromium is on your PATH for the default driver, decide which driver class you actually need, and check the CHANGELOG.md entry for the version your Gemfile resolves to, since the repository publishes no GitHub releases.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. rubycdp/vessel on GitHub