skrape.it: a Kotlin DSL for scraping that wants to be a test library

The library's own pitch is testing first, scraping second

skrape{it} describes itself as a Kotlin-based HTML and XML testing and web scraping library that can be used in Spring-Boot, Ktor, Android or other Kotlin-JVM projects, and it says outright that it aims to be a testing tool, not tied to a particular test runner, while also being usable to scrape websites in a convenient fashion.

That framing explains most of the design. If this were only a scraper, the API would be a set of selectors and a result object. Instead the README lists built-in matchers and assertions based on infix functions, and an idiomatic, type-safe DSL whose behaviour is described as a fluent API for data extraction. The assertion layer is not an afterthought bolted onto parsing, it is half of the product.

The audience is therefore a Kotlin team that writes tests against live or staging web applications, and a secondary audience of people who need to pull structured data out of HTML without writing a pile of selector calls. The licence is MIT, the required JDK is 17, and the project is published to Maven Central under the coordinates it.skrape:skrapeit. The documentation lives at docs.skrape.it, and there is a Slack channel on the Kotlin workspace, plus an entry in the Awesome Kotlin list.

The topics are broad, and they describe the range honestly: crawler, dom, html-parser, parse, scraper, integration-testing, system-testing, test-automation, kotlin-dsl, jsoup and hacktoberfest.

The module layout matches the feature list one to one

The repository is a Gradle multi-project build, and the top-level directories line up with the four feature groups in the README.

dsl/ is the DSL itself, assertions/ holds the matchers, fetcher/ is the HTTP client, and html-parser/ is the layer that turns markup into data classes. test-utils/ supports the test-oriented use, and integrationtests/ holds the project's own tests, including a DslTest.kt file that the README points at as the place to find further examples. There is a buildSrc/ directory for build logic, a config/ directory, and a qodana.yaml at the root for static analysis.

Two extension modules are listed separately, and they are the clearest statement of the audience. The mock-mvc-extension integrates with Spring's MockMvc, and the ktor-extension integrates with Ktor. Both exist so that the same skrape{it} DSL can be used inside the framework the test is already written in, rather than forcing a second request layer on top of the application's own.

The examples directory has three entries, and each one shows a different deployment context. examples/android/ is a Jetpack Compose application using AGP 9.4 and Kotlin 2.4, where the CI tests locally published artifacts in an R8-optimised release on Android, which is a statement that the library survives a minified release build on a mobile target. examples/scraping/ is the plain scraping use case, and examples/use-pre-release-version/ shows how to depend on a build that is not yet released.

The build files are Kotlin DSL, with build.gradle.kts and settings.gradle.kts, and there is a gradlew wrapper alongside.

A stable coordinate and a documented snapshot rule

The install is the standard JVM dependency, and the README gives it for both build tools. In Gradle:

dependencies {
    implementation("it.skrape:skrapeit:1.2.2")
}

and in Maven:

<dependency>
    <groupId>it.skrape</groupId>
    <artifactId>skrapeit</artifactId>
    <version>1.2.2</version>
</dependency>

All official releases go to Maven Central, which is what makes this a safe dependency to pin in a build.

The interesting part is what the README calls the bleeding edge route. Every successful CI build on master publishes a snapshot to the Central Portal snapshot repository, and the version follows a rule: it is the next minor version after the latest tag plus -SNAPSHOT. The example given is that a 1.3.0-alpha.2 release produces 1.4.0-SNAPSHOT, and the README is careful to say that 1.4.0-SNAPSHOT is an example, not a permanent version alias. You have to read the actual version from the snapshot job summary of the latest successful CI run.

Consuming one takes a repository block and a changing module:

repositories {
    maven { url = uri("https://central.sonatype.com/repository/maven-snapshots/") }
}
dependencies {
    implementation("it.skrape:skrapeit:1.4.0-SNAPSHOT") { isChanging = true }
}

The isChanging flag and the optional resolutionStrategy block that caches changing modules are there because a snapshot can be replaced in place, so a build that caches it forever will keep serving you an old jar. The README's warning is direct: these are non-official releases, they may be unstable, and breaking changes can occur at any time.

One housekeeping note from the README: the examples in the file you are reading track the current master, while the release notes and examples for official releases are on the documentation site. So the README and the docs can describe different versions at the same time.

Client-side rendering is a fetcher concern, not a parsing one

The feature that distinguishes this library from a plain parser is in the HTTP client section. The README states that the client can handle client-side rendered web pages, and that JavaScript execution results can optionally be considered in the response body. The other half of that section is a request API without verbosity or ceremony, with headers, cookies and other request options in a fluent style, and the ability to pre-configure the client regarding auth and other request settings.

The design decision here is that rendering happens before parsing, in the fetcher, and the parsed result is the same either way. From the DSL's point of view there is no difference between a page whose content was in the initial HTML and one whose content only exists after a script runs. That is what makes a single set of selectors and a single set of data classes work against both, and it is the reason the README claims the ability to analyze client-side rendered DOM trees is what makes the library unique.

Two compatibility promises support that. The library is open to implementing your own fetcher, so the HTTP layer is replaceable, and it supports non-blocking fetching with coroutine support, so a crawler built on it does not need a thread per request. The first promise matters if you need a custom transport, a proxy or a recording client, and the second matters if the scraping runs at any volume at all.

A caveat follows from the same design. Optional means optional: if the JavaScript results are not enabled for a given request, the parser will simply find nothing where the content would have been, and a test that asserted on that content would fail with a null rather than with a clear message. The distinction is worth making deliberately per request rather than by accident.

Infix matchers instead of a test runner

The idiomatic section of the README is where the project's identity shows. The DSL is described as type-safe and built to ensure a high level of readability, and the matchers are described as built-in assertions based on infix functions, with the DSL behaving like a fluent API.

The compatibility section makes the intent clear: the library is not bound to a specific test runner, framework or whatever, and it is open to using any other assertion library of your choice. This is a deliberate refusal to become a test framework. A library that forced its own runner would have to define lifecycle, reporting and discovery, and the author has chosen to stay out of that territory.

The consequence for a user is that the assertions work the same whether they run inside JUnit, Kotest, Spek or a plain main function, and the same DSL can be used in a production scrape where no test framework exists at all. That duality is the point of the project, and it is why the README says the library is first and foremost a testing tool but can also be used to scrape websites.

Two extension modules exist for the same reason. If a team has a Spring application under test, the mock-mvc-extension lets the skrape{it} DSL run against the application's own MockMvc layer, so the test exercises the server's real response without a network request. The Ktor extension does the equivalent for Ktor applications. These are the cases where binding to a framework would have been wrong, and the library offers them as add-ons instead of as a requirement.

What the README leaves to the documentation site

The README is a good summary and a poor reference, and the project says so. It states that the documentation site always holds the latest documentation, release notes and examples for official releases, and that the file itself provides examples relating to the latest master, and it points to a documentation by example section for a rough overview.

The pointer for further examples is the integration test file DslTest.kt in the project's own test sources, which is a sensible place to look because those are the examples the library is tested against.

Two quality signals in the repository deserve a mention. The build workflow has a status badge and the project uses Codecov for coverage reporting, so the test suite is treated as the source of truth for behaviour rather than as an afterthought. The qodana.yaml file at the root configures static analysis, and the Awesome Kotlin badge places the project in the curated list of Kotlin libraries.

The version picture is worth keeping straight. The repository has no GitHub releases of its own, and versions are tracked on Maven Central instead, with the coordinates it.skrape:skrapeit. The last push was on 2026-10-19, so the work is current, and the repository also carries a contributions welcome badge and a funding link, which is the usual sign of a project that expects outside patches rather than one that only releases from a single maintainer's machine.

Editorial conclusion

Adopt skrape{it} when you want one Kotlin library to both scrape a page and assert on what came back, and you prefer a typed DSL over a soup of selectors, since the same dependency can serve an integration test and a collector. Do not adopt it when your target pages are rendered by a browser you cannot drive, or when a scraping tool would blur the line between a test and a production crawler, because the library is deliberately not tied to a test runner and will not give you reporting you did not build. Before you pin a version, check the group and artifact coordinates it.skrape:skrapeit against the version listed in the documentation, and if you take a snapshot build, remember the README's own rule that the snapshot version is the next minor version after the latest tag, ending in -SNAPSHOT, and that such builds are unofficial and can break at any time.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. skrapeit/skrape.it on GitHub