Marquez: metadata service for data lineage and job provenance with an OpenLineage API

Marquez tracks how datasets are produced and consumed, not just what exists

Marquez is a metadata service with a specific scope: recording what data came from where, which jobs produced it, and how often each dataset is accessed. Its model is event-driven, built around OpenLineage, an open specification for capturing lineage as events during job execution. A team that needs to know whether a dataset is fresh, which upstream job produced it, or which downstream consumers would break if it changed will find this model useful.

Collecting metadata this way differs from a data catalog. A catalog stores descriptions of what assets exist and provides ways to tag or classify them. Marquez answers provenance questions: who produced this dataset, when, and what did that job consume to build it. For teams that need metadata enrichment or business glossaries rather than lineage tracking, Marquez covers a narrower problem.

Marquez was originally released and open sourced by WeWork and is now a Graduated project under the LF AI and Data Foundation. Adopters listed in the repository include Astronomer, Datakin, Northwestern Mutual, Ilum, and CsvPath.

PostgreSQL 14 stores all metadata, and the Java API processes incoming OpenLineage events

The running stack consists of three services: the Java API, a PostgreSQL 14 database, and the web UI. The API service runs in a container from the marquezproject/marquez image and waits for the database via a wait-for-it.sh check inside the Docker entrypoint.

PostgreSQL holds the metadata state. The docker-compose.yml configures the database with POSTGRES_USER=postgres, a database named marquez, and a separate marquez user with marquez as the password. Those credentials are development defaults and must be replaced before production use. Configuration passes between containers through environment variables: MARQUEZ_PORT, MARQUEZ_ADMIN_PORT, and POSTGRES_HOST in the API container's environment block.

The Dockerfile builds from eclipse-temurin:17, runs the Gradle shadowJar task to produce a single deployable JAR, and exposes ports 5000 and 5001. Building from source therefore requires JDK 17 and the Gradle wrapper at the repository root.

docker/up.sh starts the full stack, with a port 5000 conflict on macOS to work around

Getting a working instance starts with cloning the repository and running the startup script. On macOS and Linux:

$ ./docker/up.sh

On Windows, configure Git for Unix-style line endings with `git config --global core.autocrlf false` before cloning, verify Bash and PostgreSQL are on the PATH, then run:

$ sh ./docker/up.sh

On macOS, port 5000 is reserved by the operating system for AirPlay Receiver. The README instructs macOS users to pass an alternative port: `./docker/up.sh --api-port 9000`. Any documentation URLs referencing port 5000 must then use 9000 instead. Passing `--seed` starts the stack with sample lineage metadata; `--build` builds images from source.

The .env.example file sets TAG=0.51.1 as the Docker image version, but the most recent publicly tagged release is 0.50.0, so that default pulls a Docker image for a build that predates any formal changelog entry in the repository.

Port 5000 serves the REST API without any authentication by default

Once running, the HTTP API listens on port 5000 for all calls and port 5001 for the admin interface. A note in the README states directly that by default the HTTP API does not require any form of authentication or authorization. Anyone who can reach port 5000 can read and write lineage metadata.

Port 5001 exposes /healthcheck and /metrics endpoints for monitoring. Those are also unauthenticated in the default configuration.

To begin collecting lineage metadata, send OpenLineage events to the LineageAPI endpoint documented in the OpenAPI spec. The web UI at http://localhost:3000 shows the lineage graph, job run history, and dataset relationships. A GraphQL endpoint at http://localhost:5000/api/v1-beta/graphql provides programmatic access to metadata; the README marks that endpoint as currently in beta.

OpenLineage spec versioning is independent and must be checked before upgrading Marquez

Marquez and OpenLineage are versioned independently, and the README maintains a compatibility table showing which OpenLineage spec version each Marquez build consumes, labeled CURRENT, RECOMMENDED, or MAINTENANCE.

The README advises understanding how the OpenLineage specification is versioned and published before upgrading. Backward compatibility is maintained by accepting events written against an older OpenLineage spec in a newer Marquez instance. What happens when an older Marquez instance receives an event written against a newer spec version is not described in the README.

For teams using Airflow, the repository includes an examples/airflow/ directory. Additional OpenLineage integrations for other tools are documented at openlineage.io rather than in this repository.

For data discovery and classification rather than lineage, Marquez covers the wrong scope

Marquez answers provenance questions: how did this dataset get here, which job produced it, what did that job consume. That model is event-driven, with each job run generating an OpenLineage event that records inputs, outputs, and run metadata.

Teams whose primary need is to find datasets by searching descriptive metadata, assigning business owners, or applying classification tags are working a different problem. No classification or tagging layer is documented in the repository. A data catalog that indexes asset descriptions and supports metadata policies serves that use case, while Marquez tracks the dependency graph between running jobs.

OpenLineage events from Marquez can feed a catalog tool to provide lineage context, but the two tools address distinct problems and do not substitute for each other.

Helm chart, multiple docker-compose variants, and a Gradle build for source deployments

The repository layout includes separate directories for api/, clients/java, web/, chart/, and examples/. A Helm chart in chart/ is available for Kubernetes deployments. Multiple docker-compose files cover different configurations: docker-compose.search.yml enables search functionality on port 9200, docker-compose.dev.yml sets up a development environment, and docker-compose.seed.yml starts the stack with sample data.

For source builds, the Gradle wrapper is at the repository root. The Dockerfile's build stage runs `./gradlew --no-daemon clean :api:shadowJar` to produce the fat JAR under api/build/libs/marquez-*.jar, which the final container image copies alongside postgresql-client and bash.

The last push to the repository was on 2026-09-27.

Editorial conclusion

Marquez suits data engineering teams that need a central store for job and dataset provenance, visualized as a lineage graph, and are already using or planning OpenLineage instrumentation. Avoid it for pure data discovery or asset classification, because Marquez models events and runs rather than static asset catalogs. Before deploying, add an authentication layer in front of port 5000, since the API exposes all lineage data without credentials by default. On macOS, port 5000 is reserved; run `./docker/up.sh --api-port 9000` and update all documentation URLs to match.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. MarquezProject/marquez on GitHub
3. Project website
4. README
5. Releases