# Apache ORC and the case for type aware columnar files on Hadoop

> ORC is a self describing, type aware columnar file format with independent Java and C++ reader and writer libraries, built for large streaming reads over Hadoop workloads. The interesting engineering is in the indexes the writer builds as it goes, and the costs are in the build system, the SIMD flags and the number of released versions you now have to choose between.

**apache/orc** — Apache ORC - the smallest, fastest columnar storage for Hadoop workloads

- Repository: https://github.com/apache/orc
- Website: https://orc.apache.org/
- Stars: 770 · Forks: 515
- Language: Java
- License: Apache-2.0
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/apache-orc

## Columnar layout plus a self describing type system

ORC is described as a self describing, type aware columnar file format designed for Hadoop workloads. Two properties come bundled in that one sentence and they pull in different directions. The columnar layout is what lets a reader decode, decompress and process only the values a query actually needs, which is the whole point of storing data this way. The type awareness is what lets the writer pick an encoding suited to each type, and it is also why the format can describe its own contents rather than relying on an external schema.

The type system is complete relative to Hive. ORC supports the full set of Hive types including the complex ones: structs, lists, maps and unions. That last part matters for adoption, because a columnar format that cannot represent a map or a union cannot hold a Hive table without flattening it, and flattening is lossy in ways that surface later in query logic.

The format is optimised for large streaming reads, but the documentation is explicit that this is not a read-everything format. It carries integrated support for finding required rows quickly. For a data engineer choosing a storage format, that is the sentence to weigh: the question is whether your queries are selective, because that is where the format earns its cost.

## Stripes and row indexes are what make predicate pushdown pay

Here is the mechanism that distinguishes ORC from a plain columnar dump. Because the files are type aware, the writer chooses the most appropriate encoding for the type and builds an internal index as the file is written. Those indexes are then used by predicate pushdown: the reader determines which stripes in a file need to be read for a particular query, and the row indexes narrow the search further to a particular set of 10,000 rows.

The important detail is that indexing is a side effect of writing, not a separate build step. A file that was written by an ORC writer already carries the structures needed to skip work later. There is no index rebuild phase to schedule and no metadata store to keep in sync with the data files.

The unit of skipping is the stripe, a large group of rows. A reader that only needs rows matching a predicate that excludes entire stripes reads none of their data. When the predicate is selective enough to fall inside a stripe, the row index narrows the candidate set to a group of 10,000 rows. The example files in the repository show this machinery being exercised directly, with fixtures named for column projection, predicate pushdown, seek, and stripe level statistics.

The practical consequence is that ORC pays off when a query touches a small fraction of the data, and it pays off most when the file was written with a full set of statistics. A file written without an index, and a file written without compression blocks, both exist as named test fixtures, which is a reminder that the format's features are writer-side choices.

## Two independent libraries that read every version of the file

The repository ships a Java library and a C++ library for reading and writing the ORC format, and the documentation is careful about their relationship: the two are completely independent of each other, and each will read all versions of ORC files.

That independence is the strategic point. A Java-only columnar format locks the ecosystem into JVM tooling, and a C++-only format locks out the JVM data stack that Hadoop workloads mostly live in. Shipping both means a native or non-JVM service can read the same files without a bridge, and a Java service can read files produced by native tools. The compatibility claim is stronger than just same-branch parity: each library reads all versions of ORC files.

The repository layout reflects the split. The `c++/` directory holds the C++ reader and writer, `java/` the Java side, `cmake_modules/` the CMake modules, `examples/` a set of ORC example files used to test compatibility, `site/` the website and documentation, `tools/` the C++ tools for reading and inspecting ORC files, and `docker/` scripts for building and testing on various Linuxes. The example files are the interesting entry for anyone worried about format compatibility, since they are versioned artefacts used as a shared test corpus rather than sample data for a demo.

There is also a `conan/` directory and a `meson.build` with `meson.options` at the root, which tells you the project ships more than one way to consume it as a library. The C++ tools directory in particular is worth noting: inspecting ORC files is a first class operation, not something you write yourself against the format specification.

## Building the C++ side needs Java 17, Maven and CMake at pinned minimums

The documented build prerequisites are specific: Java 17 or higher, Maven 3.9.9 or higher, CMake 3.25.0 or higher, and Meson 1.3.0 or higher as an optional item. You will need most of them even if you only intend to build one language, because the CMake project drives the Java build as well.

The default path is a release build with debug information, created out of tree:

```shell
% mkdir build
% cd build
% cmake ..
% make package
% make test-out
```

The `make package` target produces the packages, and `make test-out` runs the test suite. The other two documented variants change only the CMake configuration. A debug build passes `-DCMAKE_BUILD_TYPE=DEBUG`, and a release build without debug information passes `-DCMAKE_BUILD_TYPE=RELEASE`:

```shell
% mkdir build
% cd build
% cmake .. -DCMAKE_BUILD_TYPE=RELEASE
% make package
% make test-out
```

You can also build a single language, which is the fastest path when you only need one library. The Java-only build is a Maven invocation from the `java` directory:

```shell
% cd java
% ./mvnw package
```

The C++-only build turns the Java half off at configure time:

```shell
% mkdir build
% cd build
% cmake .. -DBUILD_JAVA=OFF
% make package
% make test-out
```

The `mvnw` wrapper means a Maven install is not strictly required for that path, though Maven 3.9.9 or higher is still listed as a prerequisite. Expect the full build to take a while, and expect the test suite to be the slow part of it, since the example files include deliberately awkward cases such as union and timestamp combinations and memory management variants.

## AVX512 is a compile time decision that a runtime variable cannot undo

The SIMD configuration is where this build system has a sharp edge worth understanding before you start. There are two separate switches that look like they do the same thing and do not.

The CMake option `BUILD_ENABLE_AVX512` is set to "ON" or, by default, "OFF" at compile time, and it defines the SIMD level (AVX512) that gets compiled into the binaries. The environment variable `ORC_USER_SIMD_LEVEL` is set at run time to "AVX512" or, by default, "NONE", and it decides which code path the dispatch logic takes so it can apply SIMD optimisation.

The gotcha is stated plainly in the documentation: if `ORC_USER_SIMD_LEVEL` is set to "NONE" at run time, AVX512 will not take effect even if `BUILD_ENABLE_AVX512` was set to "ON" at compile time. A binary that can do AVX512 but is dispatched to NONE runs the scalar path, with no warning. The combination that works is a compile-time enable plus a runtime value of AVX512:

```shell
export ORC_USER_SIMD_LEVEL=AVX512
% mkdir build
% cd build
% cmake .. -DBUILD_JAVA=OFF -DBUILD_ENABLE_AVX512=ON
% make package
% make test-out
```

There is a portability consequence here. A binary compiled with AVX512 built into it will fail to run on a CPU without that instruction set, so enabling it turns a build option into a hardware requirement for every machine that loads the library. For a cluster with mixed hardware, that is a decision to make per node image, not per build.

## Meson is offered as an unofficial path with a different default

CMake is the official build system for ORC. Meson is described as unofficial support for building select parts of the project, and the documentation is honest about the difference in defaults. The Meson build produces unoptimised libraries with debug symbols by default, whereas the CMake build generates release libraries by default. If you switch build systems mid project, you are not producing the same artefact.

A debug build with Meson is three commands from the project root:

```shell
meson setup build
meson compile -C build
meson test -C build
```

Because the default buildtype differs, producing release libraries the way CMake does means passing the buildtype option, and the documentation warns that you must either remove the existing build directory or pass the `--reconfigure` flag:

```shell
meson setup build -Dbuildtype=release --reconfigure
meson compile -C build
meson test -C build
```

Two conveniences come with Meson that CMake users will notice. The test suite can run under valgrind out of the box with a wrap argument, which is a cheap way to catch memory errors in a C++ format parser:

```shell
meson test -C build --wrap=valgrind
```

And sanitizers can be enabled through the `-Db_sanitize=` option. The documentation gives address and undefined behaviour sanitizers as an example, combining them with a debug build type and a reconfigure:

```shell
meson setup build -Dbuildtype=debug -Db_sanitize=address,undefined --reconfigure
meson compile -C build
meson test
```

Dependency handling differs too. Meson detects dependencies on the system and downloads missing ones through its Wrap system, with the project's dependencies stored in the `subprojects` directory as individual wrap files, generated by running `meson wrap install <depencency_name>` from the project root. The documentation also notes that if you add a new dependency in future you should check WrapDB for a pre-configured entry first.

## Two version lines released on the same day

The release history shows something that matters for anyone pinning a dependency. The three most recent releases are v2.3.1 on 2026-07-16, v1.9.9 on 2026-07-16, and v2.3.0 on 2026-03-02. Two of them landed on the same day, which reads as a maintenance release for the older line running alongside a patch on the current line.

The branch list in the build table confirms the parallel structure: main, branch-2.3, branch-2.2, branch-2.1, branch-2.0 and branch-1.9, each with its own build status badge. Six maintained branches for a file format is a lot, and it is what you would expect from a format that has to stay readable across a long tail of deployed systems. Each of those branches is still built, so a fix in a patch release can land in a line that most of the world will never adopt.

For a consumer, this cuts two ways. The good side is compatibility: a long tail of maintained branches means the format does not strand older data, and the promise that each library reads all versions of ORC files backs that up. The bad side is that "which version should I use" is a real question with no default answer, because the 1.9 line and the 2.3 line are both current as of 2026-07-16. In a Hadoop installation the answer is usually decided by what the surrounding tooling supports, not by the format itself.

The last push to the main branch was on 2026-10-15, so work on the current line is continuing. The repository also carries `SECURITY.md`, `NOTICE` and `LICENSE` files, and a `dev/` directory, which is the standard Apache project layout for handling patches and releases through the wider Apache process.

## The trade off against Parquet is about the ecosystem, not the format

Any discussion of ORC has to acknowledge the other columnar format in the Hadoop world, Parquet. The difference in approach is worth stating without pretending the documentation settles the question, because the README here does not compare the two and it would be inventing facts to fill that in with performance numbers.

What the documentation does support is a structural comparison. ORC presents itself as self describing and type aware, with the writer choosing encodings per type and building indexes as it writes. The self describing part is a genuine difference in design philosophy, and it has a direct consequence: a reader can work out the structure of a file from the file itself, without a separate schema registry or an external metadata service.

The other half of the comparison is ecosystem, and here the two formats are not symmetrical. ORC is the format Hive writes by default, which is a fact about the Hadoop stack rather than about the format spec, and it means that for a Hive-centric shop the integration path is the shorter one. Parquet has the broader footprint outside that stack, in engines written in other languages and in analytics tools that are not Hadoop components.

The practical decision for a specific workload is narrower than a general claim about which format is better. If your tables live in Hive and your queries read a small column subset from a large immutable file, ORC's stripe and row index machinery is aimed exactly at that and the integration is already in place. If your data is read by engines outside the Hadoop ecosystem, the wider adoption of the alternative format may matter more than index granularity.

One limitation to keep in view regardless of format: this is a file format for data that is written once and read many times. The organisation is at file and stripe level, which is efficient for scanning and skipping, and there is no row level update story in the documentation.

## Conclusion

ORC is worth adopting for analytics workloads on Hadoop where queries read a small share of the columns and the data stays in a stable file format for years, and it is a poor fit for a small dataset or a workload that needs random row level updates, since the format is built around immutable file level organisation. Before choosing a version, confirm which of v2.3.1 or v1.9.9 your Hive and Spark stack expects, since both were released on 2026-07-16 and the repository carries parallel branch lines. If you build the C++ library yourself, decide the SIMD question at compile time rather than at runtime, because ORC_USER_SIMD_LEVEL set to NONE overrides a binary compiled with BUILD_ENABLE_AVX512=ON. The Apache-2.0 licence permits commercial use and modification without extra obligations.

## FAQ

### What is Apache ORC and who is it for?

ORC is a self describing, type aware columnar file format designed for Hadoop workloads, optimised for large streaming reads with support for finding required rows quickly. It is aimed at data platforms that store data in files and query a small share of it at a time.

### How do I build the Apache ORC C++ library from source?

Install Java 17 or higher, Maven 3.9.9 or higher and CMake 3.25.0 or higher, then configure and build out of tree with `cmake ..`, `make package` and `make test-out`. Pass `-DBUILD_JAVA=OFF` to build only the C++ library, and `-DCMAKE_BUILD_TYPE=DEBUG` for a debug build.

### How do I enable AVX512 in Apache ORC?

Set the CMake option `BUILD_ENABLE_AVX512` to "ON" at compile time and the environment variable `ORC_USER_SIMD_LEVEL` to "AVX512" at run time. If `ORC_USER_SIMD_LEVEL` is "NONE", AVX512 will not take effect even when the binary was compiled with it enabled.

### Can I read Apache ORC files from both Java and C++?

The project ships a Java library and a C++ library that are completely independent of each other, and each will read all versions of ORC files. The C++ side also ships tools for reading and inspecting ORC files.

### What complex data types does Apache ORC support?

ORC supports the complete set of types in Hive, including the complex types: structs, lists, maps and unions. Because the format is type aware, the writer picks the encoding suited to each type and builds an internal index as the file is written.

## Sources

- [apache/orc on GitHub](https://github.com/apache/orc)
- [License: Apache-2.0](https://github.com/apache/orc/blob/main/LICENSE)
- [Project website](https://orc.apache.org/)
- [README](https://github.com/apache/orc/blob/main/README.md)
- [Releases](https://github.com/apache/orc/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/apache-orc