Apache Hudi: A table format for mutable data on cloud storage

What problem Hudi solves

Hudi addresses a gap in the data lake ecosystem: cloud storage like S3 is immutable and append-only, but many analytical workloads need updates, deletes, and time-based views. The README positions Hudi as an open data lakehouse platform built on a high-performance open table format. It targets teams that want relational-style mutations on top of Parquet files without moving to a proprietary warehouse. The intended users are Spark and Flink developers who need to ingest change data, handle late-arriving records, and serve consistent snapshots. Hudi's value is not in storing data, it is in managing the metadata and file layout that make mutations possible at scale.

How Hudi works: timeline, indexing, and table services

Hudi's core mechanism is a timeline metadata structure that tracks the history of changes to a table. Every commit, rollback, or savepoint is recorded, which enables time-travel and incremental queries. Writes are atomic with rollback and restore support, and the system provides snapshot isolation between writers and queries. Indexing is a separate subsystem that speeds up snapshot queries by maintaining file listings, column-level and partition-level statistics, and record-level indexes built on row-oriented formats and bloom filters. The README also mentions logical partitioning, which uses expression indexes to decouple logical layout from physical storage. Table services run automatically inside Spark or Flink writers, or independently, and handle cleaning old versions, clustering data layout, and asynchronous compaction of row-oriented data into columnar formats. This is a substantial architecture, not a simple file format wrapper.

Query types: snapshot, incremental, CDC, time-travel, read-optimized

Hudi supports five query modes on a single table, which is a differentiator. Snapshot Query gives the latest committed state, accelerated by indexes. Incremental Query returns the latest values of records inserted or updated since a point in time, useful for diffing table states. Change-Data-Capture Query provides a change stream with before and after images for inserts, updates, and deletes. Time-Travel Query gives a view as of a given point in time. Read Optimized Query uses purely columnar storage like Parquet for excellent snapshot performance, but it depends on a compaction policy to provide a transaction boundary. The README does not detail the exact SQL syntax or API calls for each query type, so you will need to check the documentation for that. The range of query modes is the main reason to consider Hudi over a plain file layout.

Getting started: build and run with Spark

Building Hudi from source requires a Unix-like system, Java 11 or 17, Git, and Maven 3.6.0 or newer. The README gives a concrete build command: `mvn clean package -DskipTests -Dspark3.5 -Dflink2.2`. After building, you start Spark with the bundle jar and several configuration flags. The example uses `spark-3.5.0-bin-hadoop3/bin/spark-shell` with `--jars` pointing to the Hudi bundle jar, and sets `spark.serializer=org.apache.spark.serializer.KryoSerializer`, `spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension`, `spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog`, and `spark.kryo.registrator=org.apache.spark.Hoo...` (the README truncates the last one). The build profile determines which Spark version you target. The default is Spark 3.5.3 with Scala 2.12, but you can pass `-Dspark3.3`, `-Dspark3.4`, or `-Dspark4.0` to get different bundles. Scala 2.13 is supported for Spark 3.5 and above. For integration tests, add `-Dintegration-tests` to the Maven command.

Concurrency and consistency controls

Hudi offers two concurrency models, which is a notable design choice. Optimistic concurrency control implements a relational data model with read-modify-write style consistent writes. This is suitable for workloads where conflicts are rare and you want strong consistency. Non-blocking concurrency control is designed for streaming data models, supporting out-of-order and late data handling. The README does not explain how these are configured or what the trade-offs are in practice. That is a gap. You will need to read the documentation to understand when to use each mode. The existence of two models suggests that Hudi tries to serve both batch and streaming use cases, but the operational complexity is higher than a single concurrency mechanism.

Table management and catalog integration

Hudi includes automatic table services that run hands-free, either integrated into Spark or Flink writers or operated independently. These services handle cleaning older versions, time-to-live expiration to reclaim storage, clustering with space-filling curve algorithms to optimize layout, and asynchronous compaction. The scheduling strategies are configurable with built-in failure handling. Catalog sync is supported with Apache Hive Metastore, AWS Glue, Google BigQuery, and Apache XTable. This means Hudi can fit into existing metadata ecosystems, which is important for teams that already use a Hive Metastore or Glue. The README does not specify how to configure these services or what the default scheduling policies are, so expect a learning curve.

Limitations and when Hudi is the wrong tool

Hudi is not a small dependency. It requires a Java build with Maven, and the bundle jar is large. The build matrix shows multiple Spark and Scala combinations, which means you must match your runtime exactly. The README does not mention any performance numbers or benchmarks, so you cannot assume it is faster than a plain Parquet table. For purely append-only workloads with no updates or deletes, Hudi adds overhead in metadata tracking and table services without clear benefit. The concurrency controls are not trivial to operate; optimistic concurrency can cause write failures under contention, and non-blocking concurrency requires careful handling of late data. If you only need simple snapshot queries and do not need incremental or CDC views, a simpler format like Delta Lake or Iceberg may be easier to adopt. Hudi's strength is its combination of features, but that combination comes with complexity.

Alternatives and how they differ

The main alternatives are Apache Iceberg and Delta Lake. Iceberg also provides an open table format with snapshot isolation and time travel, but it does not have the same built-in indexing subsystem or the same range of query types. Iceberg focuses on table metadata and atomic commits, while Hudi emphasizes record-level indexing and incremental processing. Delta Lake is tightly integrated with Spark and offers ACID transactions and time travel, but it is not as cloud-agnostic as Hudi in terms of catalog sync, and its indexing is less mature. The README does not compare Hudi to these projects, but the feature list suggests Hudi is more oriented toward streaming ingestion and change data capture. If you need CDC queries with before and after images, Hudi is designed for that. If you just need ACID on a data lake, Iceberg or Delta Lake may be simpler.

Editorial conclusion

Adopt Apache Hudi if you need mutable data on cloud storage with snapshot, incremental, and change-data-capture queries, and you already run Spark or Flink. Do not adopt it for simple append-only analytics where a plain Parquet layout suffices. Before committing, verify which Spark and Scala versions your cluster uses, confirm the table services (cleaning, compaction, clustering) match your write patterns, and test the concurrency controls under your actual workload, since optimistic and non-blocking concurrency have different failure modes.

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes