sql.js-httpvfs: querying a read-only SQLite database over HTTP range requests

The problem: a database too large to download, on a host with no server

Static file hosting is cheap and simple. It serves bytes and nothing else: no query language, no server process, no connection pool. GitHub Pages is the example the project names, and the same constraint applies to any CDN-backed bucket. If your data is a SQLite file, you cannot ask that hoster a question. You can only download the whole file.

sql.js-httpvfs closes that gap. The README describes it as "a read-only HTTP-Range-request based virtual file system for SQLite", a fork and wrapper around sql.js, which the README calls a light wrapper around SQLite compiled with Emscripten for the browser. The intended audience is narrow and worth stating plainly: someone with a read-only dataset, a static hoster, and a query pattern that touches a small fraction of the file. A public dataset browser, a lookup tool, a reference table. Not an application that writes.

The README is candid about scope. The library was, in its author's words, mainly written for small personal projects and as a demonstration, and requests have arrived for applications outside that scope.

How the virtual file system turns SQLite page reads into HTTP requests

SQLite does not read a file; it asks a virtual file system for pages. sql.js-httpvfs supplies that layer inside Emscripten's filesystem. When SQLite wants a page, the VFS issues an HTTP range request for the corresponding byte range instead of reading from a local file. The README states that the virtual file system is an Emscripten filesystem with logic to accelerate fetching using virtual read heads that speed up when sequential data is fetched. The code lives in src/lazyFile.ts, and the README notes it might also be useful as a native SQLite VFS, which would let SQLite be compiled with WASI SDK without Emscripten's OS emulation.

The consequence is that query cost becomes network cost. A full table scan is a download. An index lookup is a handful of small requests. The README says this directly: the whole thing only works well if the database and indexes are structured well. The debugging section gives the vocabulary. In explain query plan, SCAN TABLE t1 means the table will be downloaded pretty much fully. SCAN TABLE t1 USING INDEX i1 (a=?) means a direct index lookup followed by table lookups by rowid. SCAN TABLE t1 USING COVERING INDEX i1 (a) is the fastest, because no table lookup follows.

The README recommends an index containing exactly the filtered columns and the selected columns, so SQLite reads it as a covering index sequentially with no random access. It also suggests page_size = 1024 as a trade-off between the number of requests and their overhead. There is a second, separate feature: a proof-of-concept DOM virtual table that lets queries read and write the browser DOM directly. The README calls it proof-of-concept level, and it should be treated that way.

Installing sql.js-httpvfs from npm and running a first query

The package is published on npm as sql.js-httpvfs, version 0.8.12 in the repository's package.json, under Apache-2.0. The README says to install it from npm and use it from TypeScript or JS. The entry point is createDbWorker. The README warns that there is no good way to package workers and wasm directly, so the two URLs have to come from your bundler; it shows the webpack 5 form using new URL with import.meta.url.

import { createDbWorker } from "sql.js-httpvfs"

const workerUrl = new URL(
  "sql.js-httpvfs/dist/sqlite.worker.js",
  import.meta.url,
);
const wasmUrl = new URL(
  "sql.js-httpvfs/dist/sql-wasm.wasm",
  import.meta.url,
);

What you should see is that both URLs resolve to emitted asset files in your bundle. If they do not, the worker or the wasm module will fail to load, and that failure is the most common first-run problem with this kind of package.

Before querying, the README's optional preparation step changes the database so it suits range fetching. It sets journal_mode to delete so page_size can actually be changed, sets page_size to 1024, optimizes any FTS tables, and vacuums to reorganize the database and apply the new page size.

pragma journal_mode = delete;
pragma page_size = 1024;

insert into ftstable(ftstable) values ('optimize');

vacuum;

If the hoster has a maximum file size, or if you want the CDN to cache only the chunks users actually touch, the README points to create_db.sh, which splits the database into chunks and generates a JSON config. The config passed to createDbWorker is either the URL to that generated config or an inline object. When the config is remote (from: 'jsonconfig'), the README says to cachebust it too. Cachebusting itself is an optional cacheBust property whose value is appended as a query parameter; setting it to a random value when the database changes avoids caching-related database corruption.

Where sql.js-httpvfs breaks: scans, memory, browser support and untested code

The failure mode is a query that cannot use an index. Any plan that reports SCAN TABLE without an index means the table is downloaded pretty much fully, and on a large database that is worse than shipping the file. This is not a bug to work around; it is the design. The README's own framing is that the library only works well when the database and indexes are structured well, which puts the burden on you before the first request.

Memory is the second constraint. The README states there is no cache eviction, so the more data is fetched the more RAM the page uses. A long session that wanders across the dataset grows without bound.

Browser support is the third. The README says the project makes no effort to support older or weird browsers, and that if the browser lacks WebAssembly and WebWorkers, nothing works. There is no fallback path described.

Testing is the fourth, and it is the one worth reading twice. Most of the complicated work is done by SQLite, which the README calls well tested, but the virtual file system part has no tests. That is the author's own statement about the component this project exists to provide. The README also answers its own production-readiness question without a clean yes.

One more practical limit: this is read-only. If your application needs to write, the whole approach is wrong.

Alternatives: wa-sqlite for a simpler VFS, absurd-sql for persistence

The README names two alternatives, and the difference is architectural rather than cosmetic.

wa-sqlite is described as a much simpler wasm wrapper for SQLite than sql.js, with different VFSes that do not require an Emscripten dependency. The README says sql.js-httpvfs could easily be reimplemented on top of it. So the choice is dependency weight: sql.js-httpvfs inherits sql.js and Emscripten's OS emulation, which is what makes the Emscripten filesystem approach possible in the first place, while wa-sqlite starts from a smaller base and expects you to pick a VFS.

absurd-sql is described as an implementation of a pretty efficient VFS that allows persistence and read/write queries by storing the database in IndexedDB. That is a different problem. sql.js-httpvfs reads a remote, read-only file and treats the network as the disk. absurd-sql keeps the database in the browser's own storage and lets you write. If your users need to modify data, or you need the database to survive a reload without re-fetching, absurd-sql addresses that and sql.js-httpvfs does not.

The README also links the general virtual file system discussion in the sql.js issue tracker and mentions a SQLite VFS as a possible future direction, alongside torrent-based VFS projects as inspiration. Those are pointers, not supported paths.

Licence, maintenance and what upgrading costs you

The package is Apache-2.0, both in package.json and in the repository's LICENSE file. That is a permissive licence with an explicit patent grant, and it permits commercial use and modification. It also carries notice and attribution obligations when you redistribute, and the licence text should be read directly rather than summarized by anyone, including this article. The repository's own metadata does not record a last push date in what is available here, so no claim about current maintenance activity can be made either way.

Upgrade cost is dominated by the wasm and worker assets. Because the README says there is no good way to package workers and wasm directly, and shows bundler-specific URL construction for webpack 5 with a note about the legacy webpack 4 file-loader form, a version bump can require revisiting that wiring. The dependency surface is small: package.json lists comlink as the only runtime dependency, with webpack, ts-loader, TypeScript and type packages in devDependencies. The published files field is dist/*, so what ships is the built output.

There is no release history available to reason about, and no migration notes. Treat the version pinned in your lockfile as the thing you are testing against.

Editorial conclusion

Adopt sql.js-httpvfs when you have a read-only SQLite file, a static host such as GitHub Pages, and queries that an index can answer without scanning tables. Do not adopt it for write workloads, for databases that need a table scan, or for browsers without WebAssembly and WebWorkers, and note the README's own warning that there is no cache eviction and no tests for the virtual file system. Before committing, verify three things: that the hoster serves HTTP range requests and a correct Content-Length, that your queries report COVERING INDEX or index lookups in explain query plan, and that the worker and wasm URLs resolve through your bundler.

Frequently asked questions

Official sources

1. Official README
2. Project repository