Frameless: a typed Dataset API for Spark

Three modules, and only one of them pulls in cats

The library is split into three modules and the split is deliberate, because each one has a different dependency footprint. frameless-dataset is for a more strongly typed Dataset and DataFrame API, frameless-ml is a more strongly typed Spark ML API based on frameless-dataset, and frameless-cats is for using Spark's RDD API with cats. The dependency story is stated precisely. The only dependency of the frameless-dataset module is on shapeless 2.3.2, so depending on frameless-dataset has minimal overhead on your Spark application jar, and only the frameless-cats module depends on cats and cats-effect, which means a project that works only with Datasets and not with RDDs can choose not to depend on frameless-cats at all. That is a well-judged design for a library that is meant to be adopted incrementally into an existing Spark codebase, where every transitive dependency is a negotiation with whoever owns the build. The version history also shows the module set is stable in shape rather than growing new pieces.

The artifact naming tells you which Spark you are compiling against

Getting the right artifact is the first practical hurdle, and the naming convention is worth internalising. Since 0.11 the project has published Spark cross-published artifacts. By default the frameless artifacts depend on the most recent Spark version, and a suffix of the form for major and minor version is added to artifacts released for the previous Spark versions. The examples given are the default frameless-dataset, which carries the default Spark 3.5.x dependency, frameless-dataset-spark40, which is the Spark 4.0.x dependency and is Scala 2.13 and JDK 17 only, and frameless-dataset-spark34 for Spark 3.4.x. The footnotes explain why the newer ones are restricted, since Spark 4 dropped Scala 2.12 and JDK 8 and 11, so the Spark 4.0, 4.1 and 4.2 artifacts are published for Scala 2.13 only and require JDK 17 or newer. The default unsuffixed artifacts still target Spark 3.5, which is the detail that catches people out, since adding the library with no suffix gives you Spark 3.5 behaviour even on a Spark 4 cluster.

No compile dependency on Spark at all

There is a decision in this project that is unusual and pays off over years. Frameless intentionally does not have a compile dependency on Spark. The stated effect is that this essentially allows you to use any version of Frameless with any version of Spark, and that the compatibility table simply provides the versions of Spark the project officially compiles and tests against, with other versions probably working as well. That is a large reduction in the usual matrix problem, where a library pins a Spark version and a project either matches it or fights it. The cost is that you lose compile-time checking of the Spark API itself, since the library is written against types it does not depend on, so a method signature mismatch surfaces in a different way than it would with a direct dependency. The table itself is long and precise, covering everything from 0.17.0 against Spark 4.2.0, 4.1.3, 4.0.4, 3.5.8 and 3.4.4 down to 0.4.0 against Spark 2.2.0, with the cats, cats-effect and Scala columns tracked alongside, and there is a note that 0.11.0 has broken Spark 3.1.2 and 3.0.1 artifacts published, which is the kind of specific warning that saves a debugging afternoon.

TypedDataset is the new API, and the compiler is the point

Frameless introduces a new Spark API called TypedDataset, and the benefits over the standard Dataset API are listed as four concrete properties. Columns are referenced in a typesafe way, so there are no more runtime errors when accessing a non-existing column. Encoders are customizable and typesafe, so if a type does not have an encoder it should not compile. Built-in functions have an enhanced type signature, so applying an arithmetic operation to a non-numeric column is a compilation error. And casting and projections are typesafe. Each of those is a runtime class of bug moved to compile time, and the arithmetic example is the clearest one, because applying addition to a string column in Spark compiles fine and fails when the data contains something unexpected, which in a scheduled job is a failure discovered at three in the morning. The comparison page on the Typelevel site is linked for a detailed side-by-side, and the feature overview, typed encoder documentation and typed Spark ML pages cover the individual pieces. One breaking change in 0.9 is worth knowing, since Spark 3 changed the ExpressionEncoder approach, so the schema for single value DataFrames became value rather than _1.

A Typelevel project, with the conventions that come with it

The project sits inside the Typelevel ecosystem and the repository shows the conventions that implies. There is a .scalafmt.conf for formatting, a build.sbt and a project directory for the sbt build, a github.sbt for release automation, and a scripts directory, with a .git-blame-ignore-revs file for the commits that reformat the whole codebase. The source tree is organised by module, with cats/, core/, dataset/, ml/, refined/ and docs/ directories, which matches the module list in the readme and adds a core and a refined module. The CI badge and the codecov badge are both present, so the project is tested and its coverage tracked. The readme also states that the Frameless project and contributors support the Typelevel Code of Conduct and want the associated channels, GitHub and Discord among them, to be a safe and friendly environment for contributing and learning. That is a small signal about the culture, and for a library where the API is explicitly still settling, it is the kind of thing that determines whether you can actually contribute the fix you need.

The warning about breaking changes, and the release cadence

The project is direct about its own stability, and you should read that sentence before committing a team to it. The note says that while Frameless is still getting off the ground, it is very possible that breaking changes will be made for at least the next few versions. That is an unusual thing to find in a library's own readme, and it is a fair warning. The release history is consistent with it. The current release is v0.17.0 from 2026-07-30, the one before it is v0.16.0 from 2023-12-24, and v0.15.0 from 2023-10-03, so there is a long gap between 0.16 and 0.17 and then a quiet few months since. The last push was on 2026-10-17, so the branch is still being worked on. Two other signals sit alongside this. The library is published to Maven Central under the org.typelevel group with a snapshot repository for the development builds, and the artefacts are cross-published per Scala version, with the readme showing the core artifact under a Scala 2.12 coordinate and separate ones for 2.13. The practical reading is that this is a project with a real user base and a serious maintainer, whose API surface is not frozen. That is fine for an experimental team and wrong for a platform team that cannot absorb a migration.

Editorial conclusion

Frameless is a good fit for a Spark team where a wrong column name or an arithmetic operation on a string column is discovered at runtime rather than at compile time, because that is the failure class it is built to remove. It is the wrong tool for a pipeline that never leaves the DataFrame API, and it is the wrong tool for a team that has standardised on Spark 4 on Scala 2.12, because the Spark 4 artifacts are published for Scala 2.13 only and require JDK 17 or newer. It is also the wrong tool if you cannot accept an API that is still moving, since the project itself warns that breaking changes are likely for at least the next few versions. Before adopting it, check the compatibility table for the exact Spark and cats versions you run, and pick the artifact suffix deliberately, since the default artifact still targets Spark 3.5 while the suffixed ones track older and newer releases. The project is Apache-2.0 licensed, the last push was on 2026-10-17, and the current release is v0.17.0 from 2026-07-30.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Releases
5. typelevel/frameless on GitHub