# OpenLineage: an open standard for job and dataset lineage metadata

> A specification plus a set of integrations that emit run, job and dataset events as pipelines execute. How the facet model works, and what the release history says about where the effort goes.

**OpenLineage/OpenLineage** — An Open Standard for lineage metadata collection

- Repository: https://github.com/OpenLineage/OpenLineage
- Website: http://openlineage.io
- Stars: 2,754 · Forks: 563
- Language: Java
- License: Apache-2.0
- Published: 2026-10-07 · Updated: 2026-10-07 · Language: en
- Canonical page: https://hysenlabs.com/projects/openlineage-openlineage

## A specification first, integrations second

OpenLineage describes itself as an open standard for metadata and lineage collection designed to instrument jobs as they are running. That ordering is deliberate. The project defines a generic model of run, job and dataset entities identified with consistent naming strategies, and the core model is extensible by defining specific facets that enrich those entities. Everything else in the repository follows from that.

The specification lives in the repository as an OpenAPI definition in spec/OpenLineage.md and allows extension through custom facets. The tree shows how much surrounds it: client, integration, proxy, generator, website, doc, dev and proposals directories, plus a Taskfile.yml, a ruff.toml, a pyproject.toml and a release.sh. Governance is written down too, with GOVERNANCE.md, TECHNICAL_STEERING_COMMITTEE.md, CODEOWNERS and COMMITTERS.md alongside CODE_OF_CONDUCT.md and a security policy.

The licence is Apache-2.0, the project is an LF AI & Data Foundation graduate project, and the status badge in the README says active. The most recent push was on 2026-09-23, with 2,673 stars, 535 forks and 373 open issues.

## The problem the standard is trying to solve

The README states the problem in two panels. Before OpenLineage: duplication of effort, since each project has to instrument all jobs, and integrations that are external and can break with new versions. With OpenLineage: the effort of integration is shared, and an integration can be pushed inside each project so there is no need to play catch up.

That second point is the structural argument and the one worth sitting with. If lineage instrumentation lives in a separate tool that has to keep up with a changing job engine, it will always be behind. Moving the integration into the job project itself, emitting a standard event as a side effect of normal execution, removes the version skew problem rather than managing it.

The scope section narrows the claim honestly: OpenLineage defines the metadata for running jobs and the corresponding events, and a configurable backend lets the user choose what protocol to send the events to. That second half matters more than it first appears. The standard does not mandate where events go, so the same producer can feed a Kafka topic, an HTTP endpoint or something else, and the choice belongs to whoever operates the pipeline rather than to whoever wrote the integration.

## Runs, jobs, datasets and the facet escape hatch

Three entity types and one extension mechanism carry the whole model. Run is a single execution, job is the logical unit of work, and dataset is the data it reads or writes. Each is identified with a consistent naming strategy, which is what lets a consumer join events from different producers into one graph rather than a collection of private vocabularies.

A facet is described in the README as an atomic piece of metadata attached to one of the core entities. That word atomic is the right one: facets are small, self-contained and independently versioned, so a producer can add information without renegotiating the schema. The practical result is that dbt model metadata, Spark catalog details and Flink checkpoint state all live alongside the core entities instead of replacing them.

The recent release notes show that mechanism being used rather than theorised. Version 1.51.0 added a dbt_model dataset facet carrying per-model config values such as materialization, access, owner and group along with the user-defined meta map from the dbt manifest, attached in both the legacy/local and structured-logs dbt processors. The same release extended the job type facet so that streaming producers emitting a single event covering a time period, rather than START, RUNNING and COMPLETE events, can document that lifecycle. Both changes are facet work, not model changes, which is the design working as intended.

## What the integrations actually are

The README names Spark, Airflow, dbt and Flink among supported integrations, then immediately redirects to the official integrations documentation for the up-to-date and complete list, their capabilities and matrices. That redirect is the most important sentence in the section for anyone evaluating this, because capability parity is not uniform and no single page can honestly summarise it.

What the release notes reveal about the internals is more informative than the list. Version 1.53.0, published 2026-09-01, added configurable retries to the GCP Lineage transport and to the GCS transport in the Java client, exposing retry policy through configuration rather than always using defaults. The same release added Oracle TNS connect descriptor support, parsing Oracle RAC and failover descriptors into a stable host and port form, and switched code generation to produce singleton-enum discriminated unions so explicit lineage entries construct and deserialize correctly.

That last item is a good indicator of project maturity. Generating typed union interfaces rather than hand-writing polymorphic parsing is the kind of internal cleanup that only matters once many payload shapes have accumulated. Version 1.52.0 also shows the standard simplifying itself: a generic OpenLineage context configuration replaced six separate config keys for propagating parent-run information in dbt and Spark, and dbt exposures such as dashboards and notebooks became a dataset facet so lineage covers table-to-exposure links.

## Marquez, Egeria and the surrounding ecosystem

Two related projects are named in the README, and they occupy different positions. Marquez is an LF AI & Data project for collecting, aggregating and visualising a data ecosystem's metadata, and it is the reference implementation of the OpenLineage API. That is the distinction worth holding: the standard is the contract, Marquez is one server that implements it, and the README even links to the OpenLineage collection implementation inside Marquez so you can read the consuming side.

Egeria is the other, offering open metadata and governance for enterprises, automatically capturing, managing and exchanging metadata between tools and platforms regardless of vendor. Where Marquez is a focused backend for this event model, Egeria is a broader metadata management layer that OpenLineage can feed.

The proxy directory in the tree suggests a third thing: a way to sit between producers and a backend. With 373 open issues on the repository, the volume of work is visible, and the pattern in the notes suggests it goes into transports, catalog handlers and facet coverage rather than into redesigning the core model. Release 1.51.0 adding a catalog handler for the open source Unity Catalog's UCSingleCatalog in Spark 4.0, with the note that it cannot reuse the existing DeltaHandler implementation, is a good example of the recurring cost: every new catalog needs its own handler.

## Who runs this and how to start

The governance structure is documented in the repository rather than implied. A technical steering committee, a code owners file, a COMMITTERS file and a release process with its own RELEASING.md and release.sh mean this is run as a foundation project with a maintainer model, not as a single company's codebase. The community channels are extensive: a Slack workspace, a mailing list at the foundation, a wiki, a LinkedIn group, a YouTube channel and a Mastodon account.

For a reader deciding whether to adopt it, the sequence that follows from the design is clear. Pick one producer, the dbt or Spark integration being the usual entry points, and get events flowing to a backend. Marquez is the reference implementation and the easiest thing to point at, since its own collection code is linked from the README. Only then start looking at consumers, because the value of the standard shows up when more than one producer feeds the same graph.

The documentation site at openlineage.io carries the integrations matrix and is where capability questions get answered. The repository is the place to read the specification itself, and spec/OpenLineage.md is the artefact to skim before committing to anything, since it is the thing every producer and consumer has to agree on.

## Conclusion

OpenLineage's contribution is not the code but the agreement. Once a producer emits run, job and dataset events with the naming strategies the specification defines, any consumer can draw the graph, which is why the README argues that integration effort becomes shared rather than repeated per project. The facet mechanism is what keeps that agreement from calcifying, letting dbt, Spark and Flink add domain metadata without changing the core entities. Two caveats for an adopter: the integrations matrix lives in separate documentation rather than the repository, so capability checks mean leaving the README, and with 373 open issues the project carries real bug volume. Start with one producer such as the dbt or Spark integration, send events to Marquez as the reference backend, and evaluate from there rather than from the specification alone.

## FAQ

### What is OpenLineage Marquez?

Marquez is a separate LF AI & Data project for collecting, aggregating and visualising a data ecosystem's metadata, and it is the reference implementation of the OpenLineage API. The OpenLineage repository links to Marquez's OpenLineage collection code so you can read how the consuming side handles the events the standard defines.

### What is open lineage?

Lineage metadata is the record of where data came from and what was done to it, so you can answer what a table depends on and what breaks if it stops updating. OpenLineage is an open standard for collecting that metadata by instrumenting jobs as they run, using a shared model of run, job and dataset entities plus facets that add domain specific detail.

### How do you add metadata that the core model does not cover?

With a facet, which the README describes as an atomic piece of metadata attached to one of the core entities. Facets are how dbt model config, Spark catalog details and job type information live alongside run, job and dataset without changing the core model. The specification in spec/OpenLineage.md is defined using OpenAPI and allows extension through custom facets.

### Where do OpenLineage events get sent?

Wherever you configure them to go. The README scopes the project to defining the metadata for running jobs and the corresponding events, and says a configurable backend lets the user choose what protocol to send them to. Marquez is the reference implementation of the API, and a proxy exists in the repository tree for sitting between producers and a backend.

## Sources

- [License: Apache-2.0](https://github.com/OpenLineage/OpenLineage/blob/main/LICENSE)
- [OpenLineage/OpenLineage on GitHub](https://github.com/OpenLineage/OpenLineage)
- [Project website](http://openlineage.io)
- [README](https://github.com/OpenLineage/OpenLineage/blob/main/README.md)
- [Releases](https://github.com/OpenLineage/OpenLineage/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/openlineage-openlineage