PyStore puts a Parquet-backed, Dask-friendly store behind Pandas dataframes

The object model is store, collection, item

PyStore is described as a simple yet powerful datastore for Pandas dataframes, and the key qualifier is that while it can store any Pandas object, it was designed with storing timeseries data in mind.

That qualifier explains most of the design. A timeseries workload appends rows over time and reads them back by symbol and by date range, and the object model mirrors that.

There is the store, which is the top level. There is the collection, which holds items. There is the item, which holds one dataframe plus its metadata. The quickstart shows the whole sequence of calls, and it is worth reading because the API is created-on-access throughout.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import pystore
import yfinance as yf

# Set storage path (optional)
# Defaults to `~/pystore` or `PYSTORE_PATH` environment variable (if set)
pystore.set_path("~/pystore")

# List stores
pystore.list_stores()

# Connect to datastore (create it if not exist)
store = pystore.store("mydatastore")

# List existing collections
store.list_collections()

# Access a collection (create it if not exist)
collection = store.collection("NASDAQ")

# List items in collection
collection.list_items()

# Load some data from yfinance
aapl = yf.download("AAPL", multi_level_index=False)

# Store the first 100 rows of the data in the collection under "AAPL"
collection.write("AAPL", aapl[:100], metadata={"source": "yfinance"})

So set_path sets the storage location, defaulting to a home directory path or the PYSTORE_PATH environment variable if set. list_stores enumerates what is there. store connects to a datastore, creating it if it does not exist. collection does the same for a collection within it. And then write puts a dataframe in under a name, with an optional metadata dictionary.

Reading gives you three things from the same item. item.data is a Dask dataframe, item.metadata is the metadata dictionary, and item.to_pandas() loads the whole thing into a Pandas dataframe.

That three-way split is the whole point of the library. The Dask dataframe is the lazy representation, so a query over it does not read the whole file, and to_pandas() is the explicit step where you decide to pay for the data. A library that only returned Pandas frames would force the load; a library that only returned Dask frames would force you to learn Dask for every read.

Appending to an existing item is collection.append, and the README's example appends the remaining rows of the same ticker after writing the first hundred.

Snapshots and metadata queries are the two features that make it more than a folder of Parquet files

Two capabilities in the quickstart turn a Parquet folder into something you can rely on, and both are about referring to data over time.

The first is metadata querying. collection.list_items accepts key and value pairs, so you can list the items whose metadata matches, and the example queries available symbols based on metadata. Since every write carries a metadata dictionary, you can ask the store which symbols it holds rather than keeping that list yourself, and the example writes a source key of yfinance so you can tell where a frame came from.

The second is snapshots. A snapshot is described as a point-in-time named reference for all current symbols in a collection, created with create_snapshot, listed with list_snapshots, and then used to retrieve a specific version of a symbol by passing the snapshot name when you fetch the item. There is also delete_snapshot for removing one.

So collection.item("AAPL", snapshot="snapshot_name") returns the item as it was at that snapshot rather than as it is now. For a store that is being appended to continuously, which is exactly the timeseries case, that is the difference between a query and a guess.

The alternative would be to partition the files yourself by date and manage the naming. PyStore does it as a first-class concept, which means the snapshot is a consistent cut across every item in the collection rather than a per-item thing.

The rest of the quickstart covers the destructive operations. delete_item removes an item from the current version, and store.delete_collection removes a collection outright.

The 1.0 release is a list of things a store needs and did not have

The release notes in the README are the clearest statement of what changed, and they read as a list of things a datastore is expected to have.

The 2025 release added MultiIndex support for storing and retrieving dataframes with a Pandas MultiIndex. It added full support for complex data types, meaning Timedelta, Period, Interval and Categorical dtypes. It added timezone-aware operations with proper handling of timezone data and UTC storage. It added async and await support for non-blocking I/O. It added a data validation framework with extensible validation rules for data integrity. It added schema evolution for handling schema changes over time with flexible strategies. It added transaction support for atomic operations with rollback capabilities. And it added performance optimisations covering streaming operations and memory management.

Then the phase 3 performance work is listed separately, and the numbers are the interesting part. Streaming operations give memory-efficient append for datasets larger than RAM. Batch processing is 5 to 10 times faster for parallel read and write operations. Intelligent partitioning is automatic time-based and size-based partitioning. Memory management gives a 70 to 90 percent reduction with monitoring and optimisation. Metadata caching is 100 times faster metadata access with a TTL cache. And query optimisation is column selection and predicate pushdown at the storage level.

The headline figures below that list are the ones to hold on to. Append of one million rows is 3.75 times faster with 90 percent less memory. Batch operations are 6 times faster for multiple items. Column selection is 4 times faster when reading a subset of columns. And filtered reads are 8 times faster with predicate pushdown.

The last one is the most architecturally interesting, because predicate pushdown means the filter is applied at the storage level rather than after loading, and that is the difference between reading one partition and reading the file. Combined with the automatic partitioning, that is what makes the store usable at a scale where a Pandas frame would not fit in memory.

Validation, schema evolution and transactions are the three that change how you write to it

Three of the 1.0 features are worth looking at through their API, because they change how the write path behaves rather than how fast it is.

Validation is a rule set attached to a collection. You build one with create_validator from a list of rules, and the two examples are ColumnExistsRule taking a list of column names, and RangeRule taking a field and a minimum value. Then you apply it with collection.set_validator.

So a collection can refuse a write that lacks an Open, High, Low or Close column, or one whose Close is negative. For a store whose items are financial frames, that is exactly the check you would otherwise write at every call site.

Schema evolution is opt-in per item. You call enable_schema_evolution with the item name and a strategy, and the example uses EvolutionStrategy.FLEXIBLE. After that, schema changes are handled automatically during append, so appending a frame with extra columns works without the new columns being declared in advance.

Transactions are the third. The release notes describe them as atomic operations with rollback capabilities, which for a file-backed store is a non-trivial thing to promise, since a transaction has to cover a set of file operations rather than a single row.

Together these three are what separate this from a convenience wrapper around a Parquet directory. A convenience wrapper lets you write anything anywhere. A store with validation, a declared evolution policy and transactions has a contract, and the contract is the reason you can point a pipeline at it.

Async access and complex dtypes are the two rough edges to test

Two features are less settled than the rest, and both are worth probing before you depend on them.

The async interface is a separate namespace rather than a mode of the same one. You import async_pystore rather than pystore, and the store and collection are used as async context managers, with the awaits on the operations themselves:

import asyncio
from pystore import async_pystore

async def async_example():
    async with async_pystore.store("mydatastore") as store:
        async with store.collection("NASDAQ") as collection:
            # Async write
            await collection.write("AAPL", df)
            # Async read
            df = await collection.item("AAPL").to_pandas()

asyncio.run(async_example())

The shape is clean, but it is a parallel API surface. Anything you already have written against the synchronous store does not work against the async one, which means an application that wants both has to carry both, or pick one and use a thread.

The complex dtypes are the other thing to test. The README shows a frame with a Period range, an IntervalIndex built from tuples, a Categorical column, and a nested column holding a mix of a dictionary, a list and None. Writing that is supported as of the 1.0 release. The question is what happens on read, and since the storage is Parquet through pyarrow, the round trip depends on how pyarrow represents each of those types and on the partition and index layout the store chooses for a frame whose columns are not all the same shape.

The nested column in particular is the kind of thing that Parquet handles well but that a time-partitioned, index-based store may not.

So the practical test is a round trip of your own awkward frame, written and read back, before you build on it.

For the rest of the workflow, the tutorial content is unusually good. There is a Jupyter notebook tutorial in the examples directory, and a blog post that covers the reasoning and philosophy behind PyStore along with a detailed tutorial with code examples.

Where this sits against a real database and against plain Parquet files

The comparison is with two alternatives, and PyStore's position between them is exactly what it is designed to occupy.

Against a real database, the trade is serverless against concurrency. A database gives you concurrent writers, transactions across tables, a query language, and indexes you did not have to design. PyStore gives you none of those and gives you files instead, which means no daemon to run, no connection string, and a store you can copy with rsync.

For a single-writer timeseries workload, which is what appending market data or sensor readings is, that trade is usually good. The writer appends, the readers read, and the readers do not need to block the writer. What you give up is a second writer, and the moment you need one, you need a real database.

Against plain Parquet files, the trade is convenience against lock-in. You can absolutely write Parquet directly with pyarrow and read it with pandas.read_parquet, and for a one-off script that is often less code. PyStore earns its place when you want appends to be cheap, when you want to query without loading, when you want metadata attached to each item and queryable, and when you want snapshots so that a point-in-time read is a first-class operation rather than a naming convention you invented.

The Dask layer is the quiet part of the value. Because item.data is a Dask dataframe, everything Dask knows about reading Parquet lazily applies, including column pruning and predicate pushdown. The reported figures for column selection and filtered reads are really Dask's Parquet reader doing its job, and PyStore's contribution is to make that the default rather than something you configure.

The dependency stack explains the rest. Pandas and Numpy for the data model, Dask for the lazy reads, Parquet via pyarrow for the storage format, and Snappy for compression, which the installation note flags as something you may have to install first. The package itself is Apache-2.0 licensed by Ran Aroussi, with the version at 1.0.1 and the last push on 2026-04-28.

Editorial conclusion

Adopt PyStore if you keep timeseries dataframes on disk and want to append to them and read them back without a database server, since the store is files plus Parquet with Dask underneath and the snapshot mechanism gives you point-in-time references. Skip it if you need multi-writer concurrent access or a relational query language, because this is a file-backed store with a dataframe-shaped API, and if you need a stable long history, since the 1.0 line is recent and the changes are large. Verify first that you have Snappy installed, since the documentation says the compression library has to be present, which version of PyStore you are on, because the metadata and validation APIs changed substantially in 1.0, and whether your data fits the partition and index model the store uses rather than being a wide ragged table.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. ranaroussi/pystore on GitHub
4. README
5. Releases