scala-scraper: a CSS selector DSL for pulling data out of HTML in Scala

Two operators do most of the work

scala-scraper describes itself as a library providing a DSL for loading and extracting content from HTML pages, and the DSL is small enough to hold in your head.

The first operator is >>, which applies a CSS query and extracts from the result. Text of an element by id, a list of elements matching a selector, the text inside a nested element, the value of an attribute, all of it is written as an expression:

// Extract the text inside the element with id "header"
doc >> text("#header")
// res0: String = "Test page h1"

The second is >?>, which does the same thing but wraps the answer in an Option, for the case where the element may or may not be on the page:

// Extract the element with id "footer" if it exists, return `None` if it
// doesn't:
doc >?> element("#footer")

What makes the two operators compose is that an extracted element is itself an Element, so the same syntax works on it. The README shows a list of spans pulled out of a menu, and then the text of the anchors inside each one, which is the ordinary shape of a scraper expressed without an intermediate variable for every step.

The DSL methods come from three imports, and the split between them is worth noting because it tells you what the library considers separate concerns:

import net.ruippeixotog.scalascraper.dsl.DSL._
import net.ruippeixotog.scalascraper.dsl.DSL.Extract._
import net.ruippeixotog.scalascraper.dsl.DSL.Parse._

The general DSL brings the operators and the document model, Extract brings the extraction functions such as text and elementList, and Parse brings the parsing step that turns extracted content into typed values.

The Browser is the pluggable part, and it decides what you can see

Documents are obtained through implementations of the Browser trait, and the README lists the methods they implement as get, post, parseFile and parseString, for retrieving documents from different sources.

The trait is where the design decision sits. The same extraction code has to work whether the HTML came from a file, a string, or the network, and whether it was rendered by a real browser engine or parsed as raw source. The README notes that depending on the browser used, Document and Element instances may have different semantics, mainly on their immutability guarantees, which is a fair warning that the same DSL is not the same object model underneath.

Two implementations ship with the library. JsoupBrowser is backed by jsoup, a Java HTML parser library, and the README says it provides powerful and efficient document querying but does not run JavaScript in the pages. That last point is the one that causes real bugs in practice: it is limited to working strictly with the HTML sent in the page source, so a site that renders its content with a script will look empty to JsoupBrowser and populated to a browser. Any team using this library against a modern single-page site needs to know that before they start.

HtmlUnitBrowser is the other built-in, and the visible portion of the README cuts off mid-sentence, so the detail of what it adds is not described in the text available here. The reference to a JavaScript rendering module in the repository layout, under modules/, is the more useful signal: the library splits the pure extraction core from the browser-dependent parts, which is what allows JsoupBrowser to stay fast.

The core model is Document and Element, described as simple interfaces containing methods for retrieving information and navigating through the DOM, and the source files are in the model directory of the core module.

Getting a document takes one import and one call

The quick start is a build dependency and a browser instance.

For an existing SBT project with Scala 2.13 or newer, the dependency is:

libraryDependencies += "net.ruippeixotog" %% "scala-scraper" % "3.3.0"

The binary artifact is suffixed for the Scala version, which is what the %% operator does, and the group and artifact names follow the pattern of the project.

Then a browser is created and used to get a document, either from a local file or over HTTP:

import net.ruippeixotog.scalascraper.browser.JsoupBrowser

val browser = JsoupBrowser()
val doc = browser.parseFile("core/src/test/resources/example.html")
val doc2 = browser.get("http://example.com")

The README points at that example.html file as the fixture to follow through the whole document, which is the right way to learn a scraping DSL, since the expected output is printed next to each expression.

A short for-comprehension shows the shape of a real scraper. Fetch a page, optionally find a headline link, follow it, and pull the description out of a lead paragraph:

for {
  headline <- browser.get("http://observador.pt") >?> element("h1 a")
  headlineDesc = browser.get(headline.attr("href")) >> text(".lead")
} println("== " + headline.text + " ==\n" + headlineDesc)

The use of >?> in the for-comprehension is the idiom to copy, because the Option composes with the rest of the block and a missing headline produces a skipped body rather than a crash.

The README also points to the Examples.scala file in the test sources and to the unit specs under the scalascraper test directory as usage examples, so the library ships its documentation as runnable code.

Content validation and custom extractors are the parts the DSL hints at

The README's table of contents lists a section on content validation and one on other DSL features, and the closing paragraph of the quick start describes what the full DSL covers: parsing content after extracting, validating the contents of a page, and defining custom extractors or validators.

That framing matters for how the library is meant to be used. Extraction on its own produces a String, and a String tells you nothing about whether the page was the page you expected. A validator is the piece that turns a scraper from a script that breaks silently into one that fails loudly when a site changes, and the presence of validators in the DSL rather than leaving them to the caller says the author thought about that failure mode.

The three import groups map onto this. Extract is the extraction functions, Parse is the step that turns extracted content into typed values, and the main DSL namespace is where custom extractors and validators would be plugged in. Parsing and validating are separated from extraction, so a pipeline can extract a raw value, parse it into a case class, and validate it, with each stage independently replaceable.

The README does not document the custom extractor mechanism in the visible text, so the specifics of how a user registers one are not something this material states. What it does state is that these capabilities exist in the DSL, which is the reason to look at the library rather than at a plain jsoup wrapper.

The remaining documented areas are the browser-specific feature section, which covers using capabilities beyond the common abstraction, the proxy section for working behind an HTTP or HTTPS proxy, and the Typesafe Config integration for pulling browser settings such as proxy and user agent from an application configuration.

The release tags and the documented version disagree

The maintenance picture here needs two separate facts, and the difference between them matters for anyone pinning a dependency.

The repository is not archived and the last push was on 2026-10-15, so the code is being worked on. The documented dependency in the quick start is version 3.3.0.

The release list, however, stops at v2.0.0 on 2017-07-21, before that v1.2.1 in May 2017 and v1.2.0 in January 2017. Those are the GitHub releases, and they are nine years behind the version the README tells you to depend on.

So the tags are not the version history. There is a CHANGELOG.md at the top of the repository, which is where the actual version history lives, and MAINTAINING.md sits next to it, so the release process is documented for maintainers even though the tags were not kept in step. The build file is version.sbt, which is where a single version string is kept for the whole multi-module build, and the project/ directory holds the usual SBT build definition.

The library is MIT licensed, with the terms in the LICENSE file, and the README carries badges for the CI build, coverage on Coveralls, the Maven Central artifact for the 2.13 cross-build and a Gitter room. The CI workflow is referenced on the master branch badge.

The practical advice for a new user is to read the CHANGELOG rather than the tag list, since the tag list reflects a period when the project published releases and the current workflow does not.

Editorial conclusion

scala-scraper is worth reaching for if your team writes Scala and is tired of chaining selector calls by hand, because the operator syntax plus the Extract, Parse and Validate extensions turn a scraper into something that reads like a pipeline, and the Browser abstraction means the same extraction code can run against static HTML or a rendered page. It is the wrong tool if you are on Scala 2.12, since the documented dependency is for 2.13 or newer, and it is the wrong tool if you need the library itself to be current, since the GitHub release list stops at v2.0.0 in 2017 even though the README documents version 3.3.0. Before you add it, verify three things yourself: that the version you resolve from Maven Central matches the API the README describes, since the release tags and the documented version disagree, that you have chosen the right Browser for the page, because JsoupBrowser does not run JavaScript and will silently see a different document than a browser does, and that the licensing suits you, since the project is MIT licensed.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. ruippeixotog/scala-scraper on GitHub