# Spline: data lineage tracking for Apache Spark pipelines

> Spline is an open source data lineage solution for data processing frameworks such as Apache Spark, built in Scala on an ArangoDB persistence layer and published with a documented public API and versioning policy. It suits platform teams who need to know where a dataset came from, and it depends on instrumenting the jobs that produce the data.

**AbsaOSS/spline** — Data Lineage Tracking And Visualization Solution

- Repository: https://github.com/AbsaOSS/spline
- Website: https://absaoss.github.io/spline/
- Stars: 668 · Forks: 154
- Language: Scala
- License: Apache-2.0
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/absaoss-spline

## Lineage as a service layer, not a feature of the query engine

Spline is described as an open source data lineage tracking solution for data processing frameworks like Apache Spark and others. The problem it addresses is the one every data platform team eventually hits: nobody can say, for a given table, which job produced it, which upstream tables it read, and what the job was doing at the time.

The approach is to capture lineage at the point where the data moves rather than infer it afterwards. Instead of parsing query logs or asking the engine to explain itself, the framework is instrumented, the metadata is extracted, and the result is written to a store that can be queried. That is why the project is not a Spark library you drop into a job and forget, but a set of producer, consumer and persistence components that have to be deployed.

The topics list gives the domain vocabulary: bigdata, hadoop, lineage, scala, spark, tracking and visualization. The homepage is a GitHub Pages documentation site, and the README links a separate getting started repository, which is where the step by step setup lives, along with a page about building the Docker containers. The running instructions point at the step-by-step section of the documentation site rather than into the README, which tells you that the operational instructions are substantial and maintained elsewhere.

The audience is a data platform team rather than an individual analyst. Somebody has to own the metadata store, keep the instrumentation versioned with the jobs, and answer questions about a dataset's history. Spline gives that team a system to run; it does not give a data consumer a dashboard on its own.

The primary language is Scala, the licence is Apache-2.0 with a LICENSE.md file at the root, and the default branch is develop, which is the integration branch rather than a release branch.

## Producer, consumer, persistence and the ArangoDB layer

The module list in the repository is the best description of the architecture, and it is worth reading as a data flow.

On the producing side there are producer-model, producer-model-mapper, producer-rest-core and producer-services, which together form the part of the system that extracts lineage from a processing job and exposes it. On the consuming side there are consumer-rest-core and consumer-services, which is the read path, the side a user or a tool queries to answer where a dataset came from. The persistence/ directory holds the storage layer, and it is built on ArangoDB, which is visible in the arangodb-foxx-api and arangodb-foxx-services modules, the ArangoDB Foxx extension mechanism.

Between them sit two gateways. The rest-gateway/ module is the HTTP surface, and kafka-gateway/ is the Kafka surface. That pairing matches the versioning policy in the README, which defines the public API as the combination of the producer API over both REST and Kafka, the consumer REST API, and the set of command line interfaces such as the Admin CLI.

So the shape is: a job produces lineage, it is published either over REST or to Kafka, it is persisted in ArangoDB through the Foxx services, and consumers read it back through the consumer REST API. The Admin CLI is a first class part of the public contract, which tells you that operating and maintaining the deployment is expected work, not an afterthought.

The rest of the tree supports that core. There is a commons/ module, an admin/ module, a build/ directory, an integration-tests/ directory, a test-data-generator/ for producing test inputs, and a producer-model-mapper that maps between the captured model and the persisted representation.

One structural point deserves attention from anyone running this. The ArangoDB dependency is not incidental to the design, it is the persistence layer, which means the operational footprint of Spline includes a document database that has to be run, upgraded and backed up.

## Building from source with Maven and Java 21

Spline is a Maven project, and the README gives both a one line build and the full prerequisites.

The short form is:

```shell
mvn install
```

The requirements are stated explicitly: install Java 21 and Maven 3.9 or above, then run the Maven build. The repository also carries an .sdkmanrc file, which is the usual way a JVM project pins its SDK for contributors.

The comment in the README explains what the build produces. A standard Maven build creates Java artefacts, meaning JAR and WAR files. If you also want container images, the build is told to do that instead:

```shell
# or, if you also want Docker images use this command
mvn install -Ddocker -Ddockerfile.repositoryUrl=my
```

The dockerfile.repositoryUrl property is how you point the image build at your own registry, which is a detail that matters for anyone who does not want to publish images to a public registry, and a reminder that the Docker build is part of the standard Maven lifecycle rather than a separate script.

There is a second documented build path for people who prefer containers, the building Docker page in the getting started repository, and the published images are on Docker Hub under absaoss/spline-rest-server, which the README links through a pulls badge.

For a project of this size, the TeamCity build configuration in the README is worth a look too, since the badge points at an automatic build with tests for Spark 2.4 on the develop branch. That tells you which combination the project considers the reference build, and it also tells you that the continuous integration runs against a specific Spark version rather than the latest.

Coverage is measured with a separate Maven invocation:

```shell
./mvn verify -Dcode-coverage
```

which generates a JaCoCo report under the target site directory of each module that contains measurable data.

## Why a major version bump here breaks your integration

The versioning section of the README is unusually precise, and it is the part most integrators should read before pinning a version.

The server follows Semantic Versioning. The catch is in the definition of the public API, which is stated as a combination of the APIs of all Spline modules: the Producer API in both its REST and Kafka form, the Consumer REST API, and the set of all command line interfaces, with the Admin CLI given as the example. Any incompatible change introduced in any of those APIs or CLIs is accompanied by an increment of the major version component.

The consequence is that the blast radius of a major bump is wide. A change that is internal to one module, or that only affects an endpoint you do not use, still counts as a breaking change for the version number if it touches any of those surfaces. That is a deliberate choice, and it is the safe one for a metadata service where clients are usually scripts and jobs rather than interactive applications: a version number that changes only when something you depend on has changed is easier to act on than one that changes on every internal refactor.

The trade-off is that a major version bump implies more migration work than the number of surfaces you personally use suggests, and the public API spans three transports plus a CLI. For a team that wants to pin a version and move on, the version policy is a promise of clarity rather than of effort, and it is worth checking the release notes for the module you care about rather than assuming the bump is small.

It is also worth noting that the policy is written down at all. For a platform component with a REST API, a Kafka contract and a schema in a database, having the compatibility contract stated in the README is the difference between planning an upgrade and reacting to one.

## A database schema version that deliberately ignores semver

Next to the application version, the README describes a second version number, and it is the one that surprises people.

The database schema version does not follow Semantic Versioning, it does not directly correlate with the application version, and it can only be compared to itself. The only relationship to the application version is stated like this: the schema version indicates in which application version that given schema was introduced.

That is a deliberate design, and it is the correct one for a metadata store that a client writes into. A schema version answers the question of what a particular piece of data looks like, and comparing it across releases tells you nothing about whether an application can read an existing store. What matters operationally is whether the schema your data has is the schema the running version expects, and that is a compatibility question, not a version ordering question.

The practical consequence for an operator is that you cannot use the schema version as an upgrade gate in the way you might use the application version. There is no rule of the form higher schema means newer, and there is no automatic relationship between an application version and the schemas it accepts. If you plan to move between Spline versions, the question to answer is which schema your ArangoDB holds and which one the target version expects, and the README does not tell you how to migrate between them.

That is the real gap here. The policy is clear about what the number means, and honest that it does not map onto the application version, but the documentation excerpt does not describe an upgrade or migration procedure for the persistence layer. For a system whose value is historical metadata, an upgrade path that can rewrite or preserve that history is a question to ask before you deploy, not after.

## Where Spline fits, and where it does not

The first boundary is the instrumentation requirement. Lineage is captured by the processing job, so the framework has to be integrated into the code that runs Spark jobs. A team that wants lineage for historical runs of code that has already been retired will not get it, and the README does not describe any backfill or log-parsing path that would reconstruct lineage for the past.

The second boundary is the ArangoDB dependency. The persistence layer, including the Foxx API and services, is ArangoDB, so a deployment means running and operating a document database in addition to the Spline services. The arangodb-foxx-api and arangodb-foxx-services modules are part of the build rather than an optional extra, which confirms that this is not a pluggable store.

The third is coverage of frameworks. The description says data processing frameworks like Apache Spark and others, and the build configuration visible in the README badge is for Spark 2.4 on the develop branch. The topic list mentions hadoop and bigdata, but the documented integrations in the repository are Spark oriented, with the producer and consumer modules handling the metadata side. If your pipelines are not Spark based, the question of what the producer side actually collects for your engine is not answered by the documentation excerpt.

The fourth is the thinness of the README itself. It gives the build, the versioning policy, a link to the getting started repository, a link to the documentation site, and a link to the Docker build page. There is no architecture overview, no API tour and no troubleshooting section. The operational knowledge lives in the separate spline-getting-started repository and the GitHub Pages site, which is a reasonable arrangement, but it means the repository is a starting point rather than a reference.

## Licence, release candidates and the shape of the 1.0 line

The licence is Apache 2.0, with the copyright held by ABSA Group Limited and the standard licence text in the README and the LICENSE.md file at the root of the repository. This article does not interpret it, but for a component that sits in the data path of an organisation, a permissive licence with an explicit patent grant is a straightforward choice.

The release history shows a project approaching its first stable version. The tags are release/1.0.0-RC3 from 2026-06-02, release/1.0.0-RC2 from 2026-05-01, and release/0.8.4 from 2025-09-25. The naming with a release/ prefix and the release candidate suffix means the project is still calling its pre-1.0 line, and the gap between RC2 and RC3 in May and June shows the iteration is moving quickly.

The repository is not archived and the last push was on 2026-10-08, which is after the newest tag, so work is landing on the develop branch that has not yet produced a release. The default branch being develop rather than main tells you the same thing: the published artefacts come from release branches, and the trunk is where the next version is assembled.

Two quality signals in the README are worth mentioning without overreading them. The project publishes SonarCloud badges for maintainability, reliability and security status, and a Codacy grade, which indicates that static analysis runs in the pipeline rather than being an occasional manual review. The TeamCity configuration builds with tests on the develop branch against a named Spark version.

For an adopter the practical summary is this. The code is permissively licensed, the project is moving towards a 1.0 release, and the deployment is a multi-component system with a database behind it. What you should check before committing is the documentation site, because the README deliberately does not contain the operational detail you will need on day one.

## Conclusion

Adopt Spline if you run Spark jobs and need a queryable record of how datasets were produced, and you can accept adding instrumentation to those jobs plus operating an ArangoDB instance. Do not adopt it if you need lineage for a platform it does not cover, if you cannot run ArangoDB, or if you want a system whose database migrations are covered by semantic versioning, because the schema version is tracked separately from the application version. Verify first that the artefact you depend on is the one you want, since the release tags are release candidates such as 1.0.0-RC3 dated 2026-06-02 and 0.8.4 from 2025-09-25, that the public API definition covers the modules you consume, and that the licence terms suit you, since the project is Apache-2.0 with the last push on 2026-10-08.

## FAQ

### What is Spline and what does it track?

Spline is an open source data lineage tracking solution for data processing frameworks like Apache Spark and others, published by ABSA Group. It captures lineage as jobs run and stores it so consumers can query where a dataset came from.

### How do I build Spline from source?

The README asks for Java 21 and Maven 3.9 or above, then mvn install, which produces JAR and WAR files. Adding -Ddocker with -Ddockerfile.repositoryUrl also builds the container images, and ./mvn verify -Dcode-coverage generates coverage reports.

### What breaks when Spline moves to a new major version?

The public API is defined as the combination of the producer API over REST and Kafka, the consumer REST API, and all command line interfaces such as the Admin CLI. Any incompatible change in any of those surfaces increments the major version.

### Does the Spline database schema version follow semantic versioning?

No. The README states the database schema version does not follow Semantic Versioning, does not directly correlate with the application version, and can only be compared to itself. It indicates in which application version a given schema was introduced.

## Sources

- [AbsaOSS/spline on GitHub](https://github.com/AbsaOSS/spline)
- [License: Apache-2.0](https://github.com/AbsaOSS/spline/blob/develop/LICENSE)
- [Project website](https://absaoss.github.io/spline/)
- [README](https://github.com/AbsaOSS/spline/blob/develop/README.md)
- [Releases](https://github.com/AbsaOSS/spline/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/absaoss-spline