lite3: zero-copy serialization where the wire format is the memory format

The buffer is the document

Lite3 is described as a zero-copy binary serialization format that encodes data as a B-tree inside a single contiguous buffer, allowing access and mutation on any arbitrary field in logarithmic time. It functions, the readme says, as a serialized dictionary.

The claim that follows from that is the whole design. Because the serialization boundary is broken, parsing or serializing in the traditional sense is no longer necessary. Lite3 structures can be read and mutated directly, similar to hash maps or binary trees, and since they exist in a single contiguous buffer they always remain ready to send.

The readme walks through what that looks like in practice. A message arrives from a socket, and without doing any parsing the user can look up keys and read values through zero-copy pointers, and insert or overwrite arbitrary key and value entries. When the work is done, the structure is transmitted as it is, with no serialization step, just a memory copy. The receiver then has the same set of operations available on it.

In a typical scenario, a distinct serializing and deserializing step would be required. Lite3 blurs the line between memory and wire formats, allowing direct access, traversal and mutation of a serialized buffer.

That framing has an obvious appeal for a service that parses the same small message shape thousands of times a second, and an obvious limit. The readme is honest that this is a format whose interface is the buffer, so anything that needs a schema, a stable versioned contract across languages, or a tool that already reads JSON will not find a friend here.

Schemaless, self-describing, and able to speak JSON

Compared with other binary formats, the readme says Lite3 is schemaless and self-describing, with no interface description language or schema definitions required, and that it supports conversion to and from JSON, which enables compatibility with existing datasets and application programming interfaces and allows easy debugging and inspecting of messages.

The self-describing property is the one that makes the schemalessness workable. A format with no schema has to describe itself, and that description is what lets a reader walk the buffer without knowing in advance which fields exist. The cost is size, and the trade is stated in the comparison with other formats: Lite3 outperforms schema-only formats, and the readme names Google Flatbuffers as the example of that class.

The JSON support is a deliberate bridge rather than a core feature. The feature list calls the JSON library an optional subdependency, and the code example shows the conversion in action, printing the structure as JSON so a human can read what a machine received.

For a system that has to interoperate with existing services, that matters more than the headline speed. A format that can be dumped as JSON can be inspected with the tools people already have, which is the difference between a wire format that is debuggable and one that is not.

The design has its own page on the documentation site, with a link to read more about the design of Lite3, which is where the trade-offs around a tree structure in a fixed buffer are set out in full.

Two APIs: the raw buffer and a context that hides the bookkeeping

The readme shows the low-level usage first, and it is worth reading because it makes the design concrete.

The caller declares a buffer and its size, initialises an object in it, and then sets values into it with calls that take the buffer, the current length, the offset, the size and the key and the value. Getting a value back passes the buffer, the length, the offset and a pointer for the result. The example prints a value it read back, and the output is a single line.

    lite3_get_i64(buf, buflen, 0, "max_retries", &max_retries);

The second API is the context API, where memory management is abstracted away from the user. The example, taken from the nesting example in the examples directory, creates a context, builds a message with an event name, a method and a duration, sets a headers object with three header values, prints the structure as JSON, and then reads one of the header values back out of the nested object. The printed output shows the JSON rendering of the message followed by the value of the user agent.

The two APIs are the same format seen from two sides. The buffer API is for the case where you already own the memory and want no allocation, and the context API is for the case where you would rather not track lengths and offsets yourself. The readme notes that error handling is omitted for brevity in both examples, which is the one thing a reader should not copy.

The examples directory in the repository is split the same way, into a buffer API directory and a context API directory, along with a JSON file that appears to be sample data.

No malloc, nine kilobytes, and a build that can drop JSON

The feature list contains the constraints that decide who can use this.

There is no malloc API, because the caller provides the buffer. The library size is 9.3 kilobytes for the core, and the core is dependency-free. It is written in C11 using GNU C syntax, which narrows the set of compilers a little. Pointer validation is built in, and the readme claims predictable latency and a low memory profile.

The optional JSON support has its own build switch, and the makefile explains it in detail. The JSON related libraries, a JSON parser and a base64 helper, are conditionally compiled, and the switch defaults to enabled. Setting it to disabled removes them, and the comment explains why you would: that feature requires malloc, and removing it prevents bloating the build target, which is useful in embedded or constrained environments.

That is the point of the switch. A format that is meant to sit in a buffer you own, in a system that may not have a heap, has to let you leave the convenience features behind.

The compiler flags in the makefile are worth one more look, because they are unusually strict for a small library: warnings on format strings, extra warnings, conversion warnings, strict aliasing at level three, cast alignment and shadowing. The comment above them mentions the option for finer grained dead code elimination and the linker flag it needs, which is the kind of note that only appears in a project where somebody has profiled the binary.

Installing it is a make and a pkg-config entry

The getting started section gives a make target table and then an installation walkthrough.

The targets are ordinary and complete: build the static library with optimisation level two as the default, build and run the tests with a verbose flag for output, build the examples, install the library into the local prefix for use with pkg-config, uninstall it, remove the build artifacts, and show the help message.

A compiler is required, and the readme names gcc or clang, because the library uses various builtins. The clone is a git clone and a change into the directory, and then there are two ways to install.

The easier one is through pkg-config. You build and install with a parallel make, then refresh the linker cache:

sudo make install -j
sudo ldconfig

After that you can check the installed version:

pkg-config --modversion lite3

and compile against it with the flags pkg-config prints, including the static variant of the library:

gcc -o main main.c $(pkg-config --libs --cflags --static lite3)

The manual route skips pkg-config: build the library with a parallel make and then link your program against the static library in the build directory and include the headers from the include directory.

The pkg-config metadata file lives in the repository root, which is the sign of a project that expects to be consumed by a build system rather than copied into a source tree.

The speed claims are large, and they are benchmark dependent

The performance paragraph in the readme is the most striking claim in the project, and it deserves a careful reading rather than a repetition.

The readme says that, thanks to the cache-friendly properties of the B-tree and the very minimalist C implementation at 9.3 kilobytes, Lite3 outperforms the fastest JSON libraries, ones that make use of SIMD, by up to 120x depending on the benchmark. It also says Lite3 outperforms schema-only formats such as Google Flatbuffers, by 242x, and concludes that Lite3 is possibly the fastest schemaless data format in the world.

Three things are worth noticing. The comparisons are against different classes of format, a JSON parser and a schema-based binary format, which are not the same problem. The multiplier is qualified as up to, and as depending on the benchmark, so the number is the best case of a suite rather than a general result. And the word possibly qualifies the world record claim, which is the correct way to state a claim that is hard to prove.

What is structurally true regardless of the numbers is that a format which does not parse avoids the work of parsing, and that a B-tree layout gives logarithmic lookups with good cache behaviour. A buffer that never leaves the process, or that is sent with a memory copy, removes an entire class of allocation and copy traffic. That is a real design advantage, and the size of the multiplier depends on what you compare against.

For an evaluator the right next step is to run the library's own tests and examples, then to benchmark your own message shapes, because a format tuned for small, hot, self-describing messages is not automatically a win for large documents or for batch workloads.

Editorial conclusion

Adopt Lite3 if you are writing a C service that receives small, frequently accessed messages and want to skip the parse and copy steps, because the whole buffer is the document and the accessor functions read in place. Do not adopt it if you need a schema-enforced format, a maintained interface description language or a format every tool on your team already reads, because the format is schemaless and young. Before you commit, run your own benchmark rather than relying on the numbers in the readme, since the speed claims are stated as up to a factor and depend on the benchmark, and check the design and limitations page, because a B-tree in a caller-provided buffer is a design with opinions about memory that the documentation covers in more detail.

Frequently asked questions

Official sources

1. fastserial/lite3 on GitHub
2. Issues
3. License: MIT
4. Project website
5. README