verneuil: a SQLite VFS that replicates snapshots to S3 without slowing your writes much

A SQLite VFS that copies snapshots to object storage without touching your writes

Verneuil is a VFS, the OS abstraction layer SQLite uses for file access, and it behaves like the default unix VFS while asynchronously replicating snapshots to S3-compatible blob stores. The reason the project exists is stated plainly: it was written to improve the scalability and availability of pre-existing services for which SQLite is a good fit, at least for single-node deployments. Backtrace relies on it to back up and replicate thousands of SQLite databases ranging from 100KB to a few gigabytes, some of which see updates every second, for less than $40 a day in S3 costs. The mechanism is unusual and worth understanding before you install anything. Instead of intercepting the write path and shipping changes, Verneuil watches the database file, generates snapshots into a local spooling directory, and uploads that spooled data to a remote blob store from separate worker threads.

Local disk stays the source of truth, and the spool is bounded

The design goal section is the part to read twice. The primary design goal of Verneuil is to add asynchronous read replication to working single-node systems without introducing new catastrophic failure modes, and avoiding new failure modes takes precedence over all other considerations, including replication lag. There is no attempt to bound or minimise the staleness of read replicas, and the README is clear that Verneuil read replicas should only be used when stale data is acceptable. In keeping with that approach, the local database file on disk remains the source of truth, and the VFS is fully compatible with SQLite's default unix VFS, even for concurrent accesses with file locking. All state that must persist across SQLite transactions is stored on disk, so multiple processes can still access and replicate the same database with Verneuil. This is the opposite of a synchronous replication system, and the trade is deliberate.

The failure story is where the conservative design pays off. Verneuil paces all API calls, with a currently hardcoded limit of 30 calls per second per process, to avoid surprising cloud bills. It decouples the SQLite VFS from the replication worker threads that upload data to a remote blob store, using a crash-safe buffer directory that bounds its worst-case disk footprint to roughly four times the size of the source database file. The README's conclusion is the sentence to remember: it is always safe to disable access to the blob store, because buffered replication data may grow over time, but always within bounds. A node that loses its S3 credentials keeps working and keeps writing to its local database, and the spool grows to a known multiple of the database size and no further. For a service whose primary datastore is a local file, that is the property that matters most.

Writes get slower, and the README says by how much

The cost is stated with the same candour. Replacing the default unix VFS with Verneuil impacts local SQLite operations, and writes must be slower, in order to queue updates for replication. The README puts a bound on that: the slowdown is usually proportional to the time it took to perform the write itself, and is often dominated by the two fsyncs incurred by SQLite transaction commits in rollback mode. There is a second detail about why the added latency lands where it does. The additional replication logic runs with the write lock downgraded to a read lock, so subsequent transactions only block on the new replication step once they are ready to commit. In other words, a transaction that would have blocked anyway is not made to wait longer than the commit it was already waiting for, which is the difference between a slowdown proportional to the write and a slowdown that serialises the whole database.

This is the opposite of litestream, and only one journal mode works

The README draws the comparison with litestream explicitly and calls the two efforts incomparable. Verneuil is meant for asynchronous read replication, with streaming backups as a nice side effect, and the replication approach is completely different. The concrete consequence is a hard incompatibility: litestream only works with SQLite databases in WAL mode, while Verneuil only supports rollback journaling. The README points to `doc/DESIGN.md` for details. If you are running SQLite in WAL mode today, which is the default advice for most deployments, Verneuil is not a drop-in and the decision is made for you. If you are on rollback journaling, you are in the case Verneuil was written for.

Four pieces ship in the repository, and you build one of two ways

The README describes what is in the repo, and it is more than one VFS. There is a Linux VFS in `c/linuxvfs.c` that implements everything SQLite needs for a non-WAL database without the backward compatibility cruft in SQLite's Unix VFS, and the new VFS's behaviour is fully compatible with upstream's Unix VFS, so the README calls it a simpler starting point for new Linux-only SQLite VFSes. There is a Rust crate with a C interface, in `include/verneuil.h`, providing the `verneuil` VFS that hooks into the Linux VFS to track changes, generate snapshots in spooling directories and asynchronously upload spooled data, and the `verneuil_snapshot` VFS that lets SQLite access snapshots stored in S3-compatible blob stores. There is a runtime-loadable SQLite extension, `libverneuil_vfs`, so SQLite can open a database with either VFS without relinking. And there is the `verneuilctl` command-line tool to restore snapshots, forcibly upload spooled data, synchronise a database file to remote storage and perform other administrative tasks.

For quick hacks and test drives the README says the easiest way to use Verneuil is to build it as a runtime loadable extension for SQLite:

`cargo build --release --examples --features='dynamic_vfs'`

The `verneuilctl` tool is built the same way, with a different feature:

`cargo build --release --examples --features='vendor_sqlite'`

Those two features conflict with each other, so you pick one per build. The crate manifest shows the shape of the package. The version is 0.6.4, the licence is MIT, and the crate type is both a library and a static library. The three examples are declared separately, with `verneuil_vfs` built as a cdylib requiring the `dynamic_vfs` feature, `verneuilctl` as a binary requiring `vendor_sqlite`, and `rusqlite_integration` as a plain binary with no feature requirement. That last one is the reference for direct linking: the `rusqlite_integration` example shows how a Rust crate links against the verneuil crate instead of loading it as an extension, with the detailed setup in `doc/VFS.md` and `doc/SNAPSHOT_VFS.md`.

Configuration is JSON in an environment variable

Verneuil needs to know where to spool replication data and where to upload or fetch data from remote storage. That configuration data must be encoded in JSON and deserialised into a `verneuil::Options` struct in `src/lib.rs`. A minimal configuration looks like this:

{
  // "make_default": true, to use the replicating VFS by default
  // "tempdir": "/my/tmp/", to override the location of temporary files
  "replication_spooling_dir": "/tmp/verneuil/",
  "replication_targets": [
    {
      "s3": {
        "region": "us-east-1",
        // "endpoint": "http://127.0.0.1:9000", //for non-standard regions
        "chunk_bucket": "verneuil_chunks",
        "manifest_bucket": "verneuil_manifests",
        "domain_addressing": true  // or false for the legacy bucket-as-path interface
        // "create_buckets_on_demand": true // to create private buckets as needed
      }
    }
  ]
}

Since that is a mouthful to pass as query string parameters to `sqlite3_open_v2`, Verneuil looks for the configuration string in the `VERNEUIL_CONFIG` environment variable. If the value of that variable starts with an at sign, like `@/path/to/config.json`, Verneuil reads the configuration JSON from that file instead. The comments in the example are the parts worth noting. `make_default` switches to the replicating VFS by default, `tempdir` overrides where temporary files go, `endpoint` handles non-standard regions, `domain_addressing` chooses between virtual-hosted-style addressing and the legacy bucket-as-path interface, and `create_buckets_on_demand` creates private buckets as needed. The configuration file does not include any credential.

Where Verneuil is the wrong tool

Three limits are concrete. The first is the journal mode. Verneuil only supports rollback journaling, while litestream only works with WAL mode, so between the two popular options for replicating SQLite there is no configuration that satisfies both, and a team on WAL has to change journaling mode or change tool. The second is the read path. Verneuil is explicitly a mechanism for asynchronous read replication where staleness is acceptable, with backups as a side effect, and it says plainly that there is no attempt to bound or minimise replica staleness. If you need a replica that is never more than a moment behind, or you want to fail over to it automatically, that is not what this is. The third is platform and scale. It has been tested on linux/amd64, linux/aarch64 little endian and darwin/aarch64, and the SQLite file format and the Verneuil replication data are all platform agnostic, but the Linux VFS implementation and the fork-based design mean it is not a Windows story. It is also built for many small databases rather than one large one, since the spool bound scales with the size of each source file.

The alternative is litestream, and the difference is WAL versus rollback journaling

The realistic alternative is litestream, and the difference in approach is fundamental rather than a matter of features. Litestream requires WAL mode, while Verneuil snapshots the database file and uploads chunks and manifests to object storage, so it produces point-in-time images rather than a change stream, and it requires rollback journaling. Verneuil works by snapshotting the database file and uploading chunks and manifests to object storage, so it produces point-in-time images rather than a change stream, it requires rollback journaling, and it is built for the case where you want to restore a database as it was at some past moment rather than replay changes. That difference in the unit of replication is what everything else follows from, including the staleness guarantee and the fact that Verneuil can offer read access to a stored snapshot through its separate `verneuil_snapshot` VFS. If your requirement is point-in-time restore from object storage, and you can accept rollback journaling, Verneuil fits. If your requirement is continuous log shipping to your own storage, litestream fits. There is also the option of running neither and using SQLite's own backup API on a schedule, which is simpler and loses the remote copy.

Editorial conclusion

verneuil fits an operator with many small SQLite databases on Linux that wants off-site copies in S3 and can tolerate stale read replicas and rollback journaling. It is the wrong choice for a WAL-mode deployment, for a workload that needs a replica with bounded lag, and for Windows, since the tested platforms are linux/amd64, linux/aarch64 and darwin/aarch64. Before you adopt it, confirm your databases are in rollback journal mode, size the `replication_spooling_dir` for roughly four times the source database file, and read `doc/DESIGN.md`, since the replication approach differs from litestream in kind rather than in degree.

Frequently asked questions

Official sources

1. backtrace-labs/verneuil on GitHub
2. Issues
3. License: MIT
4. README