docfind: a Rust search engine that ships as a WebAssembly blob

A search engine sized to fit in a page

docfind is a high-performance document search engine built in Rust with WebAssembly support. The design combines full-text search using FST, finite state transducers, with FSST compression for efficient storage and fast fuzzy matching capabilities.

The pitch is a search engine that is a build artefact rather than a service. You prepare a JSON file of documents, run one command, and you get a WASM module with the index embedded in it. The browser loads that module and searches locally. There is no search server, no query endpoint and no network traffic after the initial load.

The live demo makes the scale concrete. It searches through 50,000 news articles from the AG News dataset, running entirely in the browser with WebAssembly. The numbers the project publishes for that dataset are a source size of 17.14 MB as uncompressed JSON, an index size of 11.48 MB as the WASM file, a compressed size of 5.20 MB with Brotli, an index build time of about 1.1 seconds, a load time of about 100 milliseconds depending on network and browser, and a search speed of about 1 to 3 milliseconds per query.

Whether those figures are good depends on what you are comparing against. For a documentation site with a few hundred pages, an 11 MB index is a serious download. For a news archive of 50,000 articles, it is a reasonable trade for search that works offline.

The repository layout is small: cli/ and wasm/ are the two workspace members, with core/ holding the shared implementation, plus scripts/ for the build, static/ for the demo assets and a SECURITY.md.

The three phase pipeline from JSON to a searchable page

The README documents the mechanism in three phases, and the split explains why the tool works the way it does.

The indexing phase runs in the CLI. It extracts keywords from document titles, categories and bodies, uses the RAKE algorithm to identify important multi-word phrases, assigns relevance scores based on keyword source with metadata above title above body, builds an FST mapping keywords to document indices, compresses all document strings using FSST, and serializes the index using Postcard as a binary format.

The embedding phase also runs in the CLI, and it is the unusual part. It parses the pre-compiled WASM module, expands WASM memory to accommodate the index, patches the global variables INDEX_BASE and INDEX_LEN with the actual values, and adds the index as a new data segment in the WASM binary. The index is not a separate file that JavaScript fetches. It is written into the WASM module itself, which is why the deliverable is a single .wasm file rather than a .wasm plus a blob.

The search phase happens in WASM. It deserializes the embedded index on first use, performs fuzzy matching using a Levenshtein automaton, combines results from multiple keywords with score accumulation, decompresses matching document strings on demand, and returns ranked results as JavaScript objects.

That last step, decompressing strings on demand rather than shipping all of them, is what keeps the module small. The FST holds the searchable structure, FSST holds the compressed text, and only the documents that actually match get expanded.

Building an index from a JSON file

The input is a JSON array of documents, and the shape is deliberately flat. The README's example has a title, a category, an href and a body:

[
  {
    "title": "Getting Started",
    "category": "docs",
    "href": "/docs/getting-started",
    "body": "This guide will help you get started."
    },
    {
    "title": "API Reference",
    "category": "reference",
    "href": "/docs/api",
    "body": "Complete API documentation for all search functions and configuration options."
    }
]

The href field is what turns a search result into a navigation result, and the category field is what the RAKE keyword extraction reads as metadata with the highest relevance weight. The score hierarchy in the indexing phase means a term in the category of a document is worth more than the same term in its body, which is how a search for a section name can rank above a passing mention.

Building the index and generating the module is one command:

docfind documents.json output

That produces two files in the output directory. output/docfind.js holds the JavaScript bindings, and output/docfind_bg.wasm is the WebAssembly module with the index embedded. The CLI is described as a standalone tool that builds a .wasm file out of a collection of documents with no Rust tooling required, which matters because the people who publish the site often are not the people who can install a Rust toolchain.

The browser side is a module import and one call:

<script type="module">
  import search from 'docfind.js';
  
  const documents = await search('needle');
  console.log(documents);
</script>

The search function returns the ranked results as JavaScript objects, so the consumer code deals with plain data and decides how to render it.

Installing the binary without a Rust toolchain

Installation is a shell script on macOS and Linux:

curl -fsSL https://microsoft.github.io/docfind/install.sh | sh

On Windows the equivalent is a PowerShell one-liner:

irm https://microsoft.github.io/docfind/install.ps1 | iex

The installer downloads the latest release binary for your platform, installs it to ~/.local/bin on Unix or to the .docfind directory under your home on Windows, and provides instructions for adding it to your PATH if needed.

If you would rather not pipe a script into a shell, the manual route is to download the binary for your platform from the latest release. The names encode the target: docfind-x86_64-apple-darwin and docfind-aarch64-apple-darwin for the two macOS architectures, docfind-x86_64-unknown-linux-musl and docfind-aarch64-unknown-linux-musl for Linux on x64 and ARM64, and docfind-x86_64-pc-windows-msvc.exe and docfind-aarch64-pc-windows-msvc.exe for Windows. The Linux builds are musl, which means they are static and do not depend on the glibc version of the machine. You rename the file to docfind, or docfind.exe on Windows, make it executable and put it in your PATH.

Building from source is a third option, and it needs Rust, wasm-pack and Node.js, the last because esbuild is used. The build is a single script:

./scripts/build.sh

and the compiled binary ends up at ./target/release/docfind.

What the FST and FSST pairing actually buys

The performance section makes a set of claims that follow from the data structures rather than from tuning.

Sub-millisecond search times for typical queries come from the FST. A finite state transducer is a compact structure that maps strings to values, and the fst crate is used here with Levenshtein support, which is what enables fuzzy matching. A query is walked through the automaton, and because the automaton encodes all keywords in one structure, the search is a traversal rather than a scan over a term list.

The 60 to 80 percent compression ratio for document storage comes from FSST, which is a byte-oriented compressor built for short strings. It works well here because the document bodies are short and numerous, and it is a fast decompressor, which matters when the search phase decompresses matching strings on demand inside the browser.

Instant startup with lazy index loading follows from the fact that the index is deserialized on first use rather than at load time, so the module can be fetched and initialised before anyone types. And zero network requests after initial load is a property of the deployment model rather than an optimisation.

The dependency list shows the cost. The fst crate for the transducer with Levenshtein support, fsst-rs for compression, rake for the keyword extraction, serde with postcard for serialization, wasm-bindgen for the bindings, and wasm-encoder with wasmparser for the binary manipulation that makes the embedding phase possible.

The prior art section names the projects this sits among: Algolia, a server-side search-as-a-service platform, TypeSense, an open source server-side search engine, Lunr.js, a client-side full-text search library for JavaScript, Stork Search, a WebAssembly-powered search for static sites, and Tinysearch, a minimalist WASM-based search engine. The difference in approach between docfind and the server-side options is structural, and it is the same difference as between shipping a database versus shipping a query.

Where a WASM-embedded index stops being the right answer

The first limitation is that the index is a build artefact. The CLI produces it from a JSON file, and the WASM module carries the index inside it. That means the corpus has to be known at build time. If your content changes, the pipeline is build, embed, serve again, and the page weight is paid again by every visitor unless you use a CDN with proper caching. For a documentation site that changes on deploy, that is fine. For a site with user-generated content or a catalogue that changes daily, it is the wrong shape.

The second limitation is the size of the download. The demo's 11.48 MB index for 50,000 short articles is the honest number, and it scales with corpus size. The project also publishes a 17.14 MB uncompressed source and 5.20 MB with Brotli, and the 100 ms load time is described as depending on network and browser. A documentation set of a few hundred pages will be far smaller, but a large archive will make the search box appear after a noticeable wait, and that is a real cost on mobile connections.

The third limitation is the search model itself. The index is keyword based, extracted by RAKE and matched with a Levenshtein automaton. That gives tolerance for typos and for partial words, and it is fast. It does not give you phrase matching, field-specific boosting beyond the metadata, title and body hierarchy, faceting, or relevance tuning. A user searching for an exact error message will get fuzzy keyword matches rather than an exact phrase hit.

The real alternative is a server-side search engine, and the difference in approach is the difference between shipping data and shipping a service. A hosted engine like Algolia or a self-hosted one like TypeSense gives you a live index that reflects changes immediately, a richer query language, ranking you can tune, and results that are not limited to what a single WASM download can carry. What it costs is infrastructure, a network round trip per query, and the operational work of running the service. For a static documentation site with a fixed corpus and a budget of one build step, docfind's approach wins on simplicity. For a product with a growing or changing catalogue, or with users who expect exact phrase search, the service is the honest answer.

Editorial conclusion

Adopt docfind when you are publishing a static documentation set or article archive and want search that runs entirely in the browser, because the model of a prebuilt index embedded in a WASM file means no search service to run and no network request per keystroke. Do not adopt it if you need search over data that changes at runtime, since the index is built by a command and embedded at build time, and do not adopt it for a large corpus without measuring the load time on a real connection first. Before you commit, build the index over your own documents and check the resulting WASM size against your page weight budget, and confirm that the fuzzy matching behaviour of the FST index is what you want, since it matches on approximate keywords rather than on phrases.

Frequently asked questions

Official sources

1. Issues
2. microsoft/docfind on GitHub
3. Project website
4. README
5. Releases