Wombat: structured Ruby web scraping through a declarative DSL

Wombat's extraction model for Ruby page data

Wombat addresses a specific problem: turning selected parts of an HTML page into a Ruby-friendly structure without making every extraction read like low-level document traversal code. It is intended for Ruby developers who know the page they want to scrape and can identify stable CSS selectors or XPath expressions for the fields they need. The project describes itself as a lightweight web crawler and scraper, but the documented usage centers on structured extraction from individual pages rather than broad site discovery.

The DSL separates crawler settings from extracted fields. `base_url` and `path` identify the page, while statements such as `headline xpath: "//h1"` map an expression to a named value. CSS selectors are supported as well, including `css: "p.alt-lead"`. Wombat therefore works for readers who prefer naming fields over assembling every result value manually. That makes the abstraction most useful for small, stable scrapers whose structure can be expressed directly in the source.

The repository is not archived. Its last push was on 2026-04-07, and release `v3.3.0` was published on 2026-04-07. Release `v3.2.0` appeared on 2025-11-27. These dates provide repository activity context, although the README does not describe a support policy, compatibility matrix, or long-term maintenance commitment.

How Wombat turns selectors into nested Ruby results

The documented flow starts with `Wombat.crawl` and a block that defines the URL, path, and requested fields. A selector-backed statement becomes a key in the returned hash. In the README example, `headline xpath: "//h1"` produces `"headline"`, while `css: "p.alt-lead"` supplies the subheading. Nested blocks are also part of the data model: a `links` block creates a nested hash containing `explore`, `features`, and `business`.

List extraction is explicit rather than implicit. The statement `what_is({ css: ".one-fourth h4" }, :list)` asks for all matching heading values as a list. This distinction matters for adoption. A scalar field should use a single selector, while repeated elements need `:list`; Wombat does not infer every repeated match as a collection. The output example shows that choice producing an array under `"what_is"`.

Blocks can also receive individual matches as a block argument. The `explore` example uses an XPath selector and calls `e.gsub(/Explore/, "Love")`, showing that an extracted value can be transformed before it enters the output. The repository also contains `examples/hashes.rb`, `examples/iterator.rb`, `examples/list.rb`, `examples/no_class.rb`, and `examples/xml.rb`, which indicate additional extraction shapes and an XML input path. The README does not explain each example's complete behavior, so the exact iterator and XML semantics should be checked in those files or the linked API documentation.

Installing Wombat and defining a first crawl

The README gives RubyGems as the installation route. Install the `wombat` gem in the environment that will run the scraper:

gem install wombat

A first script can require the library and define one crawl. The following reproduces the README's page setup and selects a heading, a lead paragraph, and a repeated list without depending on its deeper navigation example:

require 'wombat'

Wombat.crawl do
  base_url "https://www.github.com"
  path "/"

  headline xpath: "//h1"
  subheading css: "p.alt-lead"
  what_is({ css: ".one-fourth h4" }, :list)
end

The block requests GitHub's root page and builds a hash with string keys. Based on the README's example, `headline` is expected to contain the page heading, `subheading` the selected lead text, and `what_is` an array of matching heading strings. The exact current text belongs to the target page and may differ from the README example.

The gem should normally be managed through the application's `Gemfile` in a deployed Ruby project. The repository contains `Gemfile`, `Gemfile.lock`, `wombat.gemspec`, and `.ruby-version`, but the README does not show an application-level dependency declaration or a supported Ruby version. Before production use, compare the locally resolved gem version against the application's Ruby environment.

Lists, nested blocks, and value transformation

Wombat's clearest features are the ones visible in the README: scalar extraction, lists, nested output, and a small transformation hook. These can cover a catalog, a set of article summaries, or a page header without requiring a separate class for every result field. The `links` block demonstrates that a crawler definition can preserve grouping rather than flattening every selector into one undifferentiated hash.

Transformation remains tied to the extraction DSL. In the README, a block receives `e`, then changes "Explore" to "Love" with:

explore xpath: '/html/body/header/div/div/nav[1]/a[4]' do |e|
  e.gsub(/Explore/, "Love")
end

That example is deliberately tied to a long GitHub navigation path. It is evidence that transformations are possible, not evidence that this exact XPath is stable. A change to the site's header could break it immediately. CSS selectors such as `.nav-item-opensource` and `.nav-item-business` communicate intent more clearly in this example, although any selector can still fail when markup changes.

The README does not document request headers, cookies, authentication, rate limits, proxy configuration, response encoding, pagination, or concurrency. It also gives no error policy for a missing match or malformed expression. For these reasons, Wombat is more credible as an extraction library than as a complete production crawling platform based on its published documentation.

Where Wombat is the wrong crawling tool

The main failure mode is a mismatch between static HTML extraction and a page whose useful content is produced dynamically. The README requires selectors such as `//h1`, `p.alt-lead`, and `.one-fourth h4`, but it does not document JavaScript execution, a headless browser, waiting conditions, or rendering hooks. Wombat's documentation therefore does not establish that it can load a client-rendered application before evaluating selectors. Sites that require those capabilities fall outside the documented mechanism unless another component obtains and supplies the rendered HTML.

Operational crawler requirements are also absent. There is no documented scheduler, crawl frontier, URL normalization, duplicate detection, retry policy, backoff strategy, proxy pool, or distributed execution model. A one-page script calling `Wombat.crawl` is not equivalent to a system designed to crawl thousands of pages on a schedule. Developers needing those controls would have to supply and own that surrounding infrastructure.

Another limitation is selector fragility. XPath such as `/html/body/header/div/div/nav[1]/a[4]` depends on page structure and position. If Wombat maps no match to an empty value, raises, or omits the field, the README does not say. That uncertainty should be resolved against the API documentation or source before relying on the behavior. Silent partial output and a crawler that stops are both possible engineering concerns, but the project documentation does not establish which one Wombat implements.

Wombat compared with direct Ruby scraping code

The practical alternative is to write direct Ruby code that fetches a page, parses it, selects nodes, performs value transformations, and assembles the expected hash explicitly. That approach has less DSL surface, but the application owns selector traversal, missing-value behavior, type conversion, and result construction. It can also be preferable when an extraction has conditional branches that do not map cleanly to declarative statements.

Wombat's difference is the scraping definition itself. Field names, selector types, list semantics, nesting, and transformations are expressed in one crawler block. For a compact, regular extraction, that can make the desired output easier to review than a longer procedural equivalent. When the project needs nonstandard request handling or elaborate branching, direct code may be clearer because those responsibilities are not hidden behind a page-oriented DSL.

This comparison is about implementation approach, not a claim about speed. No benchmark data is supplied, and the repository documentation gives no compatibility or performance comparison. The decision should therefore follow the shape of the extraction and the need for crawler operations rather than an unsupported claim that one approach is faster.

Wombat's MIT license and upgrade path

Wombat is distributed under the MIT license, with copyright attributed to Felipe Lima. That gives adopters a permissive licensing model, although obligations such as attribution and preservation of the license notice should be checked against the full `LICENSE.txt` for the project's intended distribution. The README does not discuss warranties, support, or commercial service commitments beyond the license information.

Upgrade information is limited but not absent. The repository includes `CHANGELOG.md`, `VERSION`, `wombat.gemspec`, `Gemfile.lock`, and `.ruby-version`. Recent releases include `v3.3.0` and `v3.2.0`, but the README does not describe their changes or breaking behavior. Reviewing `CHANGELOG.md` is therefore part of assessing upgrade cost, especially because selector changes, DSL behavior, or Ruby compatibility could affect extraction code.

The `Gemfile.lock` reflects the repository's development dependency resolution, not necessarily an application's production graph. An adopter should add Wombat to its own dependency definition and inspect the resolved versions rather than copying the repository lock file blindly. The repository's last push was on 2026-04-07, but that fact alone does not establish support for every Ruby version or future change.

Editorial conclusion

Wombat is suitable for developers who can define a small, repeatable extraction from server-rendered pages and value a concise Ruby DSL. It is a poor fit for sites that require browser execution, scheduled crawling, retry policy, or distributed collection, because the README documents none of those capabilities. Before adoption, verify that the target pages expose stable CSS selectors or XPath expressions in their initial HTML and compare one extraction against the expected output shape.

Frequently asked questions

Official sources

1. felipecsl/wombat on GitHub
2. License: MIT
3. Project website
4. README
5. Releases