PX-KVStore shards an in-memory cache around deterministic LLM keys

A key-value engine scoped to AI caching, not general storage

PX-KVStore describes itself as a lightweight, high-performance key-value engine for deterministic LLM caching and fast local inference experimentation. That framing is narrower than a general purpose cache and it explains the design choices. The store is sharded and in memory, built for low-latency workloads, and the architecture is described as layered, observable and extensible, positioned between quick scripts and heavy production databases. The project states its own tradeoffs in a table, which is unusually direct for a young repository. Lazy TTL is chosen for simplicity and performance, avoiding the overhead of a global timer wheel, at the cost of expired memory only being reclaimed on access or by a background expirer thread. Snapshot plus WAL is chosen for both fast recovery and zero data loss, at the cost of incremental disk I/O on every write and larger snapshots for big datasets. In-memory first is the design premise, and the stated limit is that dataset size is bounded by available RAM.

Consistent hashing with virtual nodes routes every key

The request flow has three layers. Requests arrive over a standard HTTP interface, and every request is assigned a unique X-Request-Id for tracing. The routing layer maps keys onto shards with consistent hashing and virtual nodes, defaulting to 100 vnodes per shard, which the documentation says keeps distribution uniform and minimises re-mapping when the shard count changes. The alternative it avoids is the simple hash modulo N approach and the thundering herd redistribution problem that comes with it. Each shard is then a self-contained unit with four parts: a hash map for O(1) lookups, a pluggable eviction policy that can be LRU or LFU, a sorted string-key index for lexicographic scans that are merged across shards with a K-way merge, and fine-grained per-shard locking. There is no global lock, so shards operate independently and throughput is expected to scale with the number of CPU cores. The observable surface comes with it: JSON metrics at /admin/metrics, native Prometheus exposition at /admin/metrics?format=prometheus, and per-route latency histograms and counters.

WAL, atomic snapshots and TTL that survives a restart

Persistence is handled in two durability tiers. Every write is appended to a synchronous write-ahead log, and on a crash the engine replays that log to recover the exact state. On top of that, periodic full-state snapshots are taken, and the implementation uses os.replace for the swap so that the database never starts from a corrupted file. The detail that matters for a cache is TTL survival. The documentation points out that, unlike many key-value stores, PX-KVStore snapshots store the remaining TTL rather than the absolute expiry time, and on restore the absolute expiry is recalculated as now plus remaining_ttl. Without that, a cache restored from a snapshot would either expire everything immediately or keep entries alive far longer than intended. Replication is separate from durability and follows an asynchronous leader-follower model: a follower pulls a full snapshot on startup, the leader then pushes changes through a background queue, and replication never blocks the leader so writes stay available.

From prompt to deterministic key, and finding the hot keys

The cache key pipeline is where the AI focus becomes concrete. A prompt is normalized by lowercasing it and collapsing whitespace, which improves hit rates. LLM parameters are sorted and turned into a canonical JSON string. A SHA-256 hash over the tuple of normalized prompt, model, params and version forms the deterministic cache key, so the same request with a different model or a different parameter ordering lands on a different entry. Responses can also be compressed transparently with Gzip plus Base64, which saves memory when outputs are large. Alongside this there is a bounded-memory hot-key tracker for high-cardinality read workloads, enabled through a set of environment variables, with a Count-Min Sketch recording successful GET and MGET reads and only the top-K candidate keys kept in memory. The tracker is inspectable at /admin/heavy-hitters?limit=20, queryable for a single key, resettable with a POST, and exported as pxkv_heavy_hitters_* Prometheus metrics.

Install, run and talk to it with redis-cli

Installation is an editable install from a checkout:

pip install -e .

The server is started as an ordinary Python script, with the durability layer switched on through environment variables that point at the WAL file, the snapshot file and the snapshot interval in seconds:

# Enable WAL and Snapshots
export PXKV_WAL_FILE=./data/wal.log
export PXKV_SNAPSHOT_FILE=./data/snap.json
export PXKV_SNAPSHOT_INTERVAL=60
python server.py

Because the server speaks the Redis protocol, existing clients work. The documentation shows the redis-cli form, using port 6379:

redis-cli -p 6379 SET mykey "Hello Redis"
redis-cli -p 6379 GET mykey

The supported command set is spelled out and includes the string commands SET with EX and PX, GET, DEL, EXISTS, the counters INCR, INCRBY, DECR and DECRBY, the expiry commands EXPIRE, TTL, PTTL and PERSIST, the introspection commands PING, INFO, DBSIZE and FLUSHALL, and the stream commands XADD, XRANGE, XREAD, XGROUP CREATE, XREADGROUP, XACK and XPENDING. That last group is the surprising one for a cache, and it is the clearest sign that the engine is meant to sit in front of an AI workload rather than only behind a prompt cache. Namespaces are also supported, with per-tenant hot-key configuration through PXKV_NAMESPACE_CONFIGS and reports at /admin/hotkeys?namespace=tenant-a.

Where the packaging details do not add up

The repository ships a Dockerfile, and it is worth reading closely because it does not quite match the tree. The image is built on python:3.12-slim, copies kvstore.py and server.py into /app, sets SHARD_COUNT as the default shard count with a note that it can be overridden at runtime, exposes port 8000 and starts python server.py. The top-level file listing does not contain kvstore.py. It contains a pxkv directory, plus server.py, tests, benchmarks, pyproject.toml, ROADMAP.md, SECURITY.md and a px_kvstore.egg-info directory, so the container build expects a file layout the current tree does not show, and the copy step is the part that would break first. The HTTP API port in the image is 8000 while the Redis protocol examples use 6379, which suggests the two front ends are exposed on different ports and the documentation does not say how the Redis listener is configured. Two smaller gaps sit alongside. The package metadata declares version 0.1.0 while the project has a v1.0.0 tag dated 2026-07-10, and no licence field appears in the pyproject, so the licensing terms are not stated by the package itself. The last push was on 2026-07-26.

Editorial conclusion

Adopt PX-KVStore if you are running local inference experiments or a deterministic LLM cache where latency matters more than durability guarantees, and you are comfortable running a single Python process. Do not adopt it as a system of record, because the dataset is bounded by available RAM and there is no cross-shard atomicity for mset or mget. Before you commit, resolve the packaging gaps yourself: the pyproject still says version 0.1.0 while a v1.0.0 tag exists, no licence is declared in the metadata, and the Dockerfile copies kvstore.py from the root although the tree lists a pxkv package directory.

Frequently asked questions

Official sources

1. cchenax/px-kvstore on GitHub
2. Issues
3. README
4. Releases