Velox: A C++ Execution Engine Library for Building Analytical Systems

What Velox Solves and Who It Is For

Velox addresses a specific problem: the repeated effort of building execution engines for analytical workloads. Instead of starting from scratch, developers of data management systems can reuse Velox's components for vectorized expression evaluation, relational operators, and I/O. The README is explicit that Velox is not meant for direct end-user interaction. It lacks a SQL parser, a dataframe layer, and a query optimizer. The intended audience is developers integrating and optimizing compute engines. If you are building a Presto or Spark compatible engine, or a custom analytical system, Velox offers a foundation. If you need a turnkey database, this is the wrong tool.

Architecture: A Composable Set of Components

The library is structured around several high-level components. The Type system handles scalar, complex, and nested types like structs, maps, and arrays. The Vector module provides an Arrow-compatible columnar memory layout with encodings such as Flat, Dictionary, Constant, and Sequence/RLE, plus lazy materialization and out-of-order writes. Expression Eval is a fully vectorized expression evaluation engine that runs on Vector data. Functions include scalar, aggregate, and window implementations following Presto and Spark semantics. Operators cover scans, writes, projections, filtering, grouping, ordering, shuffle, and joins, including hash, merge, and nested loop. I/O offers a connector interface for data sources and sinks, supporting ORC/DWRF, Parquet, and Nimble formats, with storage adapters for S3, HDFS, GCS, ABFS, and local files. Network serializers handle wire protocols like PrestoPage and Spark's UnsafeRow. Resource management provides memory arenas, buffer management, tasks, drivers, thread pools, spilling, and caching.

Extensibility: Eight Extension Points

Velox is designed to be extended. The README lists eight areas where developers can define engine-specific specializations: custom types, simple and vectorized functions, aggregate functions, window functions, operators, file formats, storage adapters, and network serializers. This extensibility is central to the project's value proposition. For example, if you need a new file format, you can implement a connector without modifying the core. The examples directory in the repository provides integration examples. This modular approach means Velox can adapt to different engines, but it also means you must understand the interfaces and write C++ code to customize behavior.

Getting Started: Build and Dependencies

To start, clone the repository with `git clone https://github.com/facebookincubator/velox.git` and `cd velox`. The first step is installing dependencies. Velox provides setup scripts that use the `DEPENDENCY_DIR` environment variable, defaulting to `deps-download` in the current directory. The `INSTALL_PREFIX` variable sets the install directory, defaulting to `deps-install` on macOS and `/usr/local` on Linux. The README advises against using `/usr/local` on macOS due to Homebrew conflicts. You can control build parallelism with `BUILD_THREADS`. The supported compiler matrix requires at least gcc 11 or clang 15 on Linux, and clang 15 on macOS. Recommended versions include gcc 12 on CentOS 9/RHEL 9 and gcc 11 on Ubuntu 22.04. The build process is non-trivial, so be prepared for a lengthy setup.

Limitations and When It Is the Wrong Tool

Velox is not a complete query engine. It expects a fully optimized query plan as input. That means you must have a planner and optimizer that produces such a plan. The README states this clearly. If you are building a simple data processing tool or a lightweight application, Velox is overkill. The C++ nature and the need to compile and link against the library add complexity. The dependency management is also involved, with many packages to build. Additionally, while Velox supports several file formats and storage adapters, you must verify that your specific needs are covered. For instance, if you rely on a niche format or a custom storage system, you may need to implement a connector yourself. The project is under active development, and the README lists recent blog posts, but the last push date is unknown, so check the repository for the latest status.

Alternative Approaches and Comparison

A common alternative is to use a full-fledged query engine like Apache Spark or Presto, which include SQL parsing, optimization, and execution. These systems are end-user ready but offer less control over the execution layer. For developers who want to build a custom engine, another option is to use a lower-level library like Apache Arrow for columnar memory and then write your own execution logic. Arrow provides data structures and compute kernels but not a complete execution engine. Velox differs by providing a higher-level execution framework with operators and expression evaluation, while still requiring your own planner. Compared to Arrow, Velox offers more out-of-the-box for execution but is more opinionated about the processing model. The choice depends on whether you need a full execution engine or just data primitives.

Maintenance, Upgrade Cost, and License

Velox is licensed under Apache-2.0, which allows commercial use with attribution. The project is developed by Meta in partnership with several companies, indicating a broad community. The README mentions a technical governance document and a list of maintainers, suggesting a structured process. However, upgrade cost can be significant. The library is large, and integrating new versions may require adapting to changes in APIs. The dependency setup scripts and the need to rebuild dependencies when versions change add to maintenance overhead. The blog posts, such as 'War of the Allocators' and 'Hash Table Caching', show ongoing performance improvements, but these come with code changes. Before adopting, review the changelog and migration guides, if any, to estimate the effort for future upgrades.

Editorial conclusion

Velox is for developers building or optimizing analytical engines who need a high-performance, extensible execution layer and are willing to integrate it with their own query planner and SQL parser. It is not for end users or teams seeking a complete database. Before adopting, verify that your required file formats, storage adapters, and network serializers are supported, and assess the build complexity and dependency management overhead. The project's Apache-2.0 license and active community support make it a viable foundation, but the lack of a built-in optimizer means you must provide your own.

Official sources

1. Official documentation
2. Official README
3. Project repository