# datatable wants your 100GB frame to stay on disk

> datatable is H2O.ai's C++ backed Python frame built to mimic R's data.table, with on-disk and in-memory identical storage so operations memory-map rather than load. It is close in spirit to pandas, and it is still a C extension built with a custom build backend, which is the main thing to know before adopting it.

**h2oai/datatable** — A Python package for manipulating 2-dimensional tabular data structures

- Repository: https://github.com/h2oai/datatable
- Website: https://datatable.readthedocs.io
- Stars: 1,878 · Forks: 164
- Language: C++
- License: MPL-2.0
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/h2oai-datatable

## The goal list explains the design better than the README does

datatable started in 2017 as a toolkit for big data operations on a single-node machine, up to 100GB, at the maximum speed possible. The stated reason is machine-learning applications that need to process large volumes of data and generate many features. The first user was Driverless.ai, which is worth knowing because it explains the emphasis on string handling and feature generation over the conveniences a general analysis library would prioritise.

The project goals list is the real specification, and reading it tells you what you get and what you do not.

The storage decisions are the most consequential. Column-oriented data storage. Native-C implementation for all datatypes including strings, with the specific claim that pandas and numpy do this for numeric columns but not for strings. Date-time and categorical types supported, with object type also supported but promotion into object discouraged. All types support null values with as little overhead as possible.

Then the two goals that define the library's character: data is stored on disk in the same format as in memory, which allows memory-mapping and transparent operation on out-of-memory datasets, and work is done on memory-mapped datasets to avoid loading more than each particular operation needs. That single idea is central to datatable's stated design.

The rest are engineering commitments: fast CSV reading, multi-threaded processing that should attempt to utilise all cores, efficient sorting, grouping and joining, a query syntax similar to data.table, minimal data copying with copy-on-write semantics for shared data, rowindex views in filtering, sorting, grouping and joining to avoid unnecessary copying, and interoperability with pandas, numpy, pyarrow and pure Python so you can convert to another framework with ease.

## Installation is one command on three platforms and a toolchain everywhere else

On macOS, Linux and Windows:

```sh
pip install datatable
```

On all other platforms a source distribution is needed, with build instructions in the documentation. That split is the first practical filter, and the stated requirements are Python 3.6+ on 64 bit with pip 20.3+.

Building from source is not a plain setup.py build, and the repository shows why. The `pyproject.toml` is six lines:

```toml
[build-system]
requires = []
build-backend = "ext"
backend-path = ["ci"]
```

An empty `requires` list, a backend named `ext`, and `backend-path` pointing into `ci/`. That is an in-tree custom build backend rather than setuptools, which means the C++ compilation is driven by code inside this repository. The `Makefile` confirms the shape, with a `clean` target that deletes `*.so`, `src/datatable/lib/_datatable*.so`, `src/datatable/lib/_datatable*.pyd` and `src/datatable/_build_info.py`, all of which are build products rather than checked-in sources.

There are also separate requirement files: `requirements_extra.txt`, `requirements_tests.txt` and `requirements_docs.txt`, each installed by its own Makefile target. A build from source therefore assumes a C++ toolchain, a 64 bit Python, and either an extension module naming convention for your platform (`.so` or `.pyd`).

That matters more than it usually would. A package with a custom in-tree build backend is not going to install cleanly on a Python version its maintainers have not tested against, and Python version support is where a stale native extension fails first.

## The difference from pandas is memory, not syntax

The README positions datatable as close in spirit to pandas or SFrame, with specific emphasis on speed and big data support, and closely related to R's data.table with an attempt to mimic its core algorithms and API.

That last clause is the practical one. If you already write data.table expressions, the query syntax will not be the barrier. If you write pandas, the mental model may differ in how data is copied. Here the goals explicitly call for rowindex views in filtering, sorting, grouping and joining to avoid unnecessary data copying, with copy-on-write semantics for shared data.

That sounds like an implementation detail until you have a chained expression over a memory-mapped frame. Then it is the difference between touching the disk once and touching it several times.

The interop goal is the safety valve and should be planned for rather than discovered. You should have the ability to convert to pandas, numpy, pyarrow or pure Python with ease. A realistic adoption pattern is datatable for the heavy lifting over data that does not fit in memory, and a conversion boundary where the data does fit.

One thing the README does not do is show code. There is no usage example in the text, so the query syntax has to be learned from the documentation site rather than from a first paragraph.

## Release history and the project status question

The version history is the thing to weigh. Releases listed are v1.1.0 from 2023-12-01, v1.0.0 from 2021-07-02 and v0.11.1 from 2020-12-10. The last push was on 2026-09-01.

So the repository is receiving commits nearly two years after the last tagged release. `VERSION.txt` at the repository root holds the version, and `CHANGELOG.md` is not in the top-level entries at all, so there is no in-repository changelog to see what the untagged work covers. That combination, commits without releases and no changelog, means you cannot tell from the repository alone whether the untagged state is a maintenance branch or work in progress.

The signals in the repository are mixed in a way worth naming. The documentation is built with readthedocs, there is a `docs/` directory, a `tests/` directory and a separate `tests_random/` for randomised testing, and CI configuration lives in `ci/` alongside the build backend. Randomised testing at that scale suggests the authors cared about correctness of the C++ core. But the build badge in the README points at Travis CI while the repository has `.github/` for workflows, which suggests the CI configuration has partly moved and the README badge has not.

Licensing needs care here. The repository metadata says MPL-2.0, but the file headers and the README carry an MIT-style copyright and permission notice from H2O.ai. MPL-2.0 is a file-level copyleft licence, so distribution or modification of covered files can create source-distribution obligations; using the package as an installed dependency does not by itself create those obligations. That is a description of the licence mechanics rather than advice, and any conclusion about your specific use belongs with whoever handles licensing for you.

The honest summary is that datatable remains a capable and unusually designed library, and that adopting it is a bet on a project whose tagged releases have been quiet since December 2023.

## Conclusion

Adopt datatable if your pipeline genuinely cannot fit the data in memory and you want data.table syntax with C++ speed, since memory-mapping and on-disk identical storage is a distinctive capability described in its goals. Do not adopt it for a team that wants a maintained dependency: the newest listed release is v1.1.0 from 2023-12-01, the last push was on 2026-09-01, and the packaging uses an in-repo build backend, so Python version support will lag. Verify first that your platform has a wheel, because anything outside macOS, Linux and Windows needs a source distribution built with a C++ toolchain.

## FAQ

### How do I install datatable?

Run `pip install datatable` on macOS, Linux and Windows. On all other platforms you need a source distribution, following the build instructions in the documentation, which require a C++ toolchain.

### What are datatable's system requirements?

Python 3.6 or newer on 64 bit, and pip 20.3 or newer. The package builds a native extension, with shared objects for Unix and .pyd for Windows.

### How does datatable differ from pandas?

The README places it close in spirit to pandas with specific emphasis on speed and big data support, and closely related to R's data.table. The distinctive design is storing data on disk in the same format as in memory so datasets can be memory-mapped and worked on transparently.

### What is the latest datatable release?

The listed releases are v1.1.0 from 2023-12-01, v1.0.0 from 2021-07-02 and v0.11.1 from 2020-12-10. The last push was on 2026-09-01, and the repository has no CHANGELOG.md.

### Can datatable handle data larger than available memory?

Yes, that is a stated project goal. Data is stored on disk in the same format as in memory to allow memory-mapping and transparent work on out-of-memory datasets, with rowindex views to avoid unnecessary copying. The original target was operations up to 100GB on a single machine.

## Sources

- [h2oai/datatable on GitHub](https://github.com/h2oai/datatable)
- [License: MPL-2.0](https://github.com/h2oai/datatable/blob/main/LICENSE)
- [Project website](https://datatable.readthedocs.io)
- [README](https://github.com/h2oai/datatable/blob/main/README.md)
- [Releases](https://github.com/h2oai/datatable/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/h2oai-datatable