Vaex: memory-mapped DataFrames that skip the copy

What Vaex is for, and who it is not for

Vaex targets a specific shape of problem: a tabular dataset too large to hold in RAM, stored in a file format that supports memory mapping, and analysed through statistics and plots rather than row-by-row transformation. The README describes it as a library for lazy out-of-core DataFrames, similar to Pandas, for visualising and exploring big tabular datasets. The stated scale is more than a billion rows per second for statistics such as mean, sum, count and standard deviation computed on an N-dimensional grid.

The intended user is someone doing exploratory data analysis on a laptop or a single machine who would otherwise reach for a cluster. The README frames this directly: filtering and evaluating expressions will not waste memory by making copies, and the data is streamed only when needed, which delays the point at which a cluster becomes necessary. That is the pitch. It is not a general replacement for pandas in a small-data script, and it is not a distributed compute engine.

Memory mapping is the whole design, not an optimisation

The core mechanism is that Vaex opens HDF5 and Apache Arrow files by mapping them into the process address space instead of reading them into arrays. The README calls this instant opening of huge data files and pairs it with a zero memory copy policy. Because the operating system page cache handles the actual I/O, opening a multi-gigabyte file is cheap and the resident memory footprint stays proportional to what you touch, not to the file size.

This is why the format support matters so much. HDF5 and Arrow are both laid out so that columns can be addressed without reading the whole file. A CSV cannot be memory mapped in the same way, and the README points to a separate documentation page on efficiently converting data from CSV files, Pandas DataFrames, or other sources. Conversion is a real step in the workflow, not an afterthought. The README also mentions lazy streaming from S3 in combination with memory mapping, which extends the same model to object storage, though the details of that path are not spelled out in the README itself.

The expression system and why filtering does not copy

Expressions are the second half of the design. The README's phrasing is that you do not waste memory or time with feature engineering because the transformations are applied lazily when needed. A derived column in Vaex is a recipe, not an array. When you filter a DataFrame, the result is a selection over the original data rather than a new materialised table, which is what allows the README to claim that filtering and evaluating expressions will not waste memory by making copies.

The practical consequence is that a chain of filters and derived columns costs almost nothing until something forces evaluation. Aggregations force it. So do plots. This is a different mental model from pandas, where each intermediate result is a real object occupying real memory, and it is the reason Vaex can claim interactive exploration of datasets that would otherwise need chunked processing. The trade-off is that errors in an expression may not surface until evaluation, and debugging a lazy pipeline is less direct than inspecting a pandas DataFrame step by step.

Groupby, joins, and where the billion-rows-per-second claim comes from

The README states that Vaex implements parallelised groupby operations, especially when using categories, at more than a billion rows per second. It also states that joins do not copy or materialise the right table, saving gigabytes of memory, with subsecond joining on a billion rows. Both claims are attributed to the project's own documentation and marketing material; they are not independently verified here, and the conditions under which they hold (categorical columns, particular hardware, particular data layouts) are not specified in the README.

What can be said from the material is the mechanism behind the join claim: not materialising the right table means the join is evaluated against the mapped representation rather than a copied one. That is consistent with the memory-mapping design and is the part worth taking seriously. Treat the throughput numbers as the project's own framing and measure on your own data before you plan capacity around them.

Installing and opening a file

Installation is a single command, either from PyPI with pip install vaex or from conda-forge with conda install -c conda-forge vaex. The README links to a documentation page on installing for more detail, and the PyPI badge in the README points at the vaex-core package, which suggests the distribution is split into multiple packages with vaex as a metapackage. That split is worth knowing about if you are pinning dependencies, because the version you care about may be vaex-core rather than vaex.

Beyond installation, the README does not give a worked code example. It shows screenshots of opening files and of the expression system, and links to tutorials and an example_io documentation page for converting data. If you need to know the exact function names for opening an HDF5 file or exporting to Arrow, the README will not tell you; you have to go to docs.vaex.io. That is a gap in the repository's front page, and it means the README alone is not enough to get from install to first result.

The format constraint is the real limitation

The biggest constraint is not performance, it is input format. The memory-mapping model depends on HDF5 and Arrow. If your data arrives as Parquet, which is the default columnar format in most modern data stacks, the README does not claim support for it. You would be converting first, and that conversion is an extra pipeline stage with its own cost and its own failure modes.

The second limitation is API coverage. Vaex is described as similar to Pandas, not compatible with it. Code that relies on pandas idioms, on the index, or on in-place mutation will need rewriting. The README's own framing of machine learning without explicit pipelines suggests the intended ML workflow is scikit-learn style estimators applied to Vaex columns, not a drop-in replacement for a pandas-based feature engineering script. If your work is mostly reshaping small tables, Vaex adds a lazy evaluation layer you will not benefit from.

How it differs from Dask and Polars

Dask takes the partitioning approach: it splits a dataset into chunks and coordinates computation across them, which lets it scale out to a cluster but means each worker holds its chunk in memory. Vaex takes the opposite route, keeping data on disk and mapping it, so the ceiling is your disk and address space rather than your RAM, and the scaling story is a single machine rather than a cluster. The README explicitly positions this as delaying the time before you need a cluster, which is an honest description of the trade: you get further on one box, and you do not get distributed execution.

Polars is the closer comparison on the single-machine axis. It is built around Arrow and a query optimiser, and it targets Parquet as a first-class format. Vaex's distinguishing choice is memory mapping with lazy expressions over HDF5 and Arrow, which is aimed at files that already exist on disk and are too large to load. If your data is in Parquet, Polars is the more natural fit. If it is in HDF5 and you want to open it without reading it, Vaex is doing something the others are not.

Licence, maintenance, and what the release list tells you

Vaex is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are included. That is a permissive licence with no copyleft obligation on your own code. It says nothing about the licence of your data or of HDF5 itself, and none of this is legal advice; check with your own counsel if the distinction matters to you.

On maintenance, the supplied material is ambiguous and worth flagging. The repository is not archived and the last push is dated 2026-04-01, but the recent releases list only three entries: vaexpaper_v1 from 2018, 1.0.0-beta.1 from 2016, and 0.3.10 from 2015. That list does not reflect the pip and conda install commands in the README, which implies releases are tracked elsewhere, most likely on PyPI under vaex and vaex-core. The gap between the repository's release list and its install instructions is a real thing to check before you pin a version. The README also points to a Slack channel and offers enterprise support and training through vaex.io, which suggests a commercial entity behind the project; how that affects long-term maintenance is not something the repository states.

Editorial conclusion

Adopt Vaex if your data already lives in HDF5 or Apache Arrow files that you can memory map, and your workload is filtering, aggregation and histogram-based exploration rather than row-wise mutation. Do not adopt it if your data is CSV-only, if you need the full pandas API surface, or if your pipeline is built around Parquet and columnar pushdown engines. Before committing, verify three things yourself: that your files are in HDF5 or Arrow rather than Parquet, that the operations you actually need are implemented in the expression system, and that the version you install from pip or conda-forge matches the API in the documentation you are reading, since the release list on the repository is sparse and the newest entry dates from 2018.

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. vaexio/vaex on GitHub