LensVM: bi-directional data transformation with WebAssembly

What the engine does, and what a lens is

LensVM is a bi-directional transformation engine that was originally built for a document database and is available as a standalone tool. It transforms data in both the forward and the reverse direction, using lenses that the user creates, written in any language and compiled to WebAssembly.

The bi-directional part is the heart of the idea, and it comes from a well established idea in programming: a lens is a value that focuses on one part of a data structure and provides a way to read it and a way to write it. That is the classic meaning, and this engine takes the word literally, because each lens is compiled to a module with two directions of travel and the engine can run either.

The word transformation is doing a lot of work here too. A lens is not a function from one value to another, and that distinction matters. A function takes a value and returns a value. A lens takes a value, and can be composed with other lenses, and can be inverted, and can be applied to a part of a larger structure rather than to the whole of it. The whole value of a lens is that the reverse direction is available.

The mechanics of the standalone tool are simple to state. You need two things. First, you need to choose how you will talk to the engine, through one of the engine implementations. Second, you need one or more lenses, which contain the compiled transforms you want to apply to your data.

And there is currently a single engine, written in Go, with implementations or wrappers in other languages planned rather than shipped. The readme's request in that case is specific: if you want to use lens through a language other than Go, thumb up the issue or create one. That is a good way to prioritise, and it also tells you that today the Go engine is the only path, so a project in another language would have to run the Go engine rather than embed its own.

The repository records the primary language as Go, the licence is the Mozilla Public Licence, and the last release was in June 2026.

The function contract, and how small it is

A lens is a WebAssembly module with a handful of exported and imported functions, and the readme lists them precisely. The smallness is the design.

There is a mandatory exported function to allocate a block of memory of a given size, which the engine calls. There is a mandatory imported function that lets the lens pull a pointer to the next data item from the engine. There is a mandatory exported function where the transformation actually happens. And there are two optional exported functions.

The optional one that provides the configuration is worth explaining, because it is where a lens gets its parameters. It is called once, before the lens receives any data, and it receives a pointer to the configured data and returns a pointer to an ok or error response. So the engine hands the lens its settings once, at initialisation, and the lens can hold on to them.

The transformation function is the interesting one. It can pull zero or many items from the incoming function, transform the inputs, and return a single pointer to the transformed output item. Only one output can be yielded at a time, but the lens is allowed to be stateful, which means it can cache several outputs and yield them one at a time. That is a real escape hatch, because a transformation that expands one input into several outputs has nowhere else to put them.

And the second optional function is the inverse. It is defined in exactly the same way as the transformation, and if a lens does not provide one, the engine has no way to run the data backwards through it. So the bi-directional promise of the engine is per lens, and a lens that does not implement the inverse is a one-way transformation wearing a two-way engine's clothes.

That is the whole contract. Three mandatory functions, two optional ones, and a memory allocator. It is small enough to implement in a language with no WebAssembly library of its own, which is presumably the point of specifying it in terms of raw pointers rather than in terms of a richer interface.

The wire format, and the four type identifiers

Data crosses the WebAssembly boundary in a fixed format, and the readme gives it in one line.

[TypeId][Length][Payload]

The type identifier is a signed eight byte integer and the length is an unsigned thirty-two byte integer, and the payload is whatever the type says it is.

There are four possible values of the type identifier, and the way they are defined tells you everything about the protocol's design.

The value negative one means an error. The length may be set, and an error string may be provided as the payload. So the error case is a first class result rather than an exception across the sandbox boundary, which is the right way to do it, because a WebAssembly boundary cannot throw a host exception into the guest.

The value zero means a nil item, and the length and payload do not exist at all. So the stream can carry an explicit absence, which is not the same thing as an empty payload and matters for data that can be null.

The value one means a JSON item, with the length set to the length of the payload and the payload containing the serialised item. The engine does not care about the shape of that JSON, which is what keeps the contract language neutral: a lens in Rust, in AssemblyScript, or in anything else that targets WebAssembly, speaks JSON across the boundary.

And the value one hundred and twenty-seven means the end of the data stream, that there are no more items to pull or to return. Again the length and payload do not exist. The end marker is what lets a lens pull from the engine until there is nothing left, and what lets a transformation that yields several outputs signal that it has finished.

So the format is: a signed tag, a length, and a payload, with the tag carrying the meaning and the length existing only for the one case where there is a payload. That is about as spare a protocol as you can write while still being unambiguous, and the fact that the end of stream and the error are both first class values is what makes a pipeline of lenses composeable.

The readme also says there is a Rust SDK that should remove the need to think about any of this, and it is published both on GitHub and on the package registry. For a lens author, that SDK is the difference between writing five functions and writing a transform.

The isolation model, and lenses from several parties

The sentence that explains why this is a WebAssembly engine rather than a plugin system is the second one in the readme: each lens is executed in its own independent WebAssembly environment, which allows pipelines to be safely constructed from lenses sourced from multiple parties.

That is the design goal, and it is a strong one. A transformation is a piece of code, and code from another party is untrusted. If a lens runs in the same process as the engine, it can read the engine's memory, see the other lenses' data, and corrupt the run. The usual mitigation is a process boundary, with the cost of serialising everything and paying for a context switch per operation.

The WebAssembly sandbox is a cheaper version of the same idea. The guest gets its own linear memory, it cannot reach outside it, and the only channel is the interface that the readme has just documented. So a lens written by a partner organisation, or by a user of your application, can be plugged into your pipeline without becoming a liability, and the trust boundary is enforced by the runtime rather than by review.

The isolation is what makes the bi-directional contract safe too, because the inverse function runs in exactly the same sandbox with exactly the same restrictions. A lens that has been given your data cannot keep a copy of it and leak it later, because its memory is discarded when the lens instance goes away.

And the reason the lenses are separate modules, rather than a single compiled program, is composition. A pipeline is a sequence of lenses, each one a separate module, and the engine wires them together. The readme's example lenses are written in two different languages, AssemblyScript and Rust, in the same test directory, which is the clearest possible demonstration that the language independence is real rather than aspirational.

So the design has three parts that fit together: a sandbox so untrusted transforms are safe, a tiny contract so a transform is cheap to write in any language, and an inverse so the pipeline can be run in either direction. Take away the inverse and you have a sandboxed pipeline runner. Take away the sandbox and you have a bidirectional transformation library. Together they are more than either.

The repository, the toolchain, and the test targets

The repository layout is a Go workspace with a Rust SDK and a set of test modules, and the build file shows how the pieces are assembled.

The top level holds the Go workspace file and its checksum file, the directory for the Go engine, the directory for the Rust SDK, a tests directory, a tools directory, and the make file.

The build section of the readme says that once the prerequisites are installed, you can build everything in the repository and run all the tests with a single make target from the root. The prerequisites are three toolchains: the Rust toolchain and Cargo, the Node package manager, and Go. The Rust note is specific, since on some systems you also need the compiler package for the distribution, and the readme also warns about connection errors when fetching Rust dependencies, pointing at a configuration change to make in the Git configuration. That is a real problem people hit, and a link to the exact discussion saves a lot of guessing.

The make file is short, and its targets describe the test strategy.

test:
	$(MAKE) deps:test
	$(MAKE) -C ./tests/modules test
	$(MAKE) --no-print-directory -C ./host-go test:no-deps
	$(MAKE) --no-print-directory -C ./tests/integration test:no-deps

So the test target depends on building the engine and the test modules, then runs the module tests, then the engine's own tests, then an integration suite. The continuous integration variant skips the dependency build and runs the engine and integration suites directly, on the assumption that the artefacts are already there. And there is a separate target for the JavaScript tests, which installs a browser test runner, which tells you the WebAssembly modules are exercised in a real browser as well as in the Go test harness. That is a deliberate and demanding choice for a WebAssembly project, and it is the right one, because the whole point of the engine is to run in a browser.

And there is a target for the scripts directory, which is a separate test suite again, presumably for the tooling that builds and packages the modules.

So the test story has three layers: the lenses themselves, the engine, and the integration between them, plus a browser pass and a scripts pass. For a project whose contract is five functions, that is a serious amount of verification, and it is what you would expect from a tool that is meant to run code written by other people.

Editorial conclusion

LensVM fits a team with data that has to be transformed by code other teams own, because the isolation model means a lens from one party cannot reach the memory of another, and it fits anybody who needs a transformation to be reversible. It does not fit a project that wants a single process with a function call, because the WebAssembly boundary is the price of the isolation. Verify first which language your lenses will be written in and that a toolchain for it can target WebAssembly, that the engine you want is the Go one, since the readme describes a single engine and the others are planned rather than shipped, and that the inverse of your transformation is actually defined, because a lens without an inverse is only half of what the engine is for.

Frequently asked questions

Official sources

1. License: MPL-2.0
2. Project website
3. README
4. Releases
5. sourcenetwork/lens on GitHub