starwing/lua-protobuf for Lua protobuf serialization

Why starwing/lua-protobuf exists

starwing/lua-protobuf targets Lua programs that exchange messages using Google's protobuf protocol. The project offers a high-level path for converting between Lua tables and binary protobuf data, plus lower-level modules for applications that need to work with the wire format directly. Its supported runtimes include Lua 5.1, 5.2, 5.3, 5.4, and LuaJIT.

The primary use case is a service or application that already has a `.proto` schema and needs to serialize structured data without implementing protobuf encoding by hand. A Lua table can represent a protobuf message, `pb.encode()` can produce binary bytes, and `pb.decode()` can convert those bytes back into a table. The README's example uses a `Person` message containing a name, an age, an address, and repeated `Phone` contacts.

There are two schema paths. Applications that use Google's compiler can load a compiled `*.pb` file produced by `protoc`. Applications that do not want that dependency can use `protoc.lua`, a pure Lua module that translates text-based protobuf schema content into binary `*.pb` data. This distinction is important for deployment. The first path depends on schemas compiled elsewhere, while the second moves schema compilation into the Lua environment.

The layered protobuf implementation

The repository separates protobuf work into a high-level `pb` module and lower-level components. The README identifies `pb.slice` as a wire-format decoder, `pb.buffer` as a buffer for encoding basic protobuf types and supporting streaming decode, `pb.conv` as a converter for integers in protobuf wire format, and `pb.io` as an access point for standard input, standard output, or other files in binary mode.

The high-level flow requires a loaded schema. A compiled schema can be loaded into the `pb` module, after which a message type name such as `Person` identifies the message used by `pb.encode()` or `pb.decode()`. The Lua table is the application-facing representation. Binary protobuf data is represented as a Lua string, and the README also identifies `pb.Slice` and `pb.Buffer` values as acceptable data forms for relevant operations.

The pure Lua compiler operates separately from the C module. `protoc.new()` creates a compiler instance. Its `parse()` method transforms schema text into a `DescriptorProto` table, `compile()` transforms schema text into binary `*.pb` data, and `load()` loads the schema into `pb`. This separation gives an application a choice between a schema-building workflow and direct access to encoded data.

Installing the module with LuaRocks

The README documents LuaRocks as the simplest installation route. This installs the package named `lua-protobuf`, which is the package users should request when they want the project through the normal LuaRocks workflow.

luarocks install lua-protobuf

After installation, the example requires the modules as `pb` and `protoc`. If a system installation does not expose `protoc.lua` on the Lua module path, the README's source instructions show how to build the C module and copy `protoc.lua` beside `pb.so`.

The repository's default branch is `master`. It is not archived, and the last push was on 2026-07-02. The most recent listed release is 0.5.3 from 2025-05-29, after 0.4.0 from 2022-07-27 and 0.3.4 from 2022-02-22. These dates identify the available release history, but the README does not provide a detailed upgrade policy or a compatibility matrix for every intervening release. Projects should therefore check their pinned LuaRocks version and their own schema requirements when upgrading.

Building starwing/lua-protobuf from source

The source build requires a Lua header or library installation for the `pb` C module. The repository entries include `pb.c` and `pb.h`, while `protoc.lua` is the pure Lua component. The README presents several build paths, including LuaRocks, hererocks, and direct compiler commands.

For a LuaRocks source build, clone the repository and use the included rockspec:

git clone https://github.com/starwing/lua-protobuf
luarocks make rockspecs/lua-protobuf-scm-1.rockspec

The README also documents a hererocks-based setup for environments without LuaRocks:

pip install hererocks
git clone https://github.com/starwing/lua-protobuf
hererocks -j 2.0 -rlatest .
bin/luarocks make lua-protobuf/rockspecs/lua-protobuf-scm-1.rockspec CFLAGS="-fPIC -Wall -Wextra" LIBFLAGS="-shared"
cp protoc.lua pb.so ..

The final command copies the pure Lua compiler and compiled C module to the parent directory. This is a deployment detail, not a universal requirement. The LuaRocks or application packaging system must still put the resulting module where the target application can resolve it.

The README also provides direct gcc commands for macOS and Linux. On macOS it uses `-undefined dynamic_lookup`, while on Linux it uses `-fPIC`. The Windows example uses `cl`, creates `pb.dll`, and defines `Lua_BUILD_AS_DLL`. The exact flags should be selected for the host toolchain rather than copied indiscriminately.

Loading a schema and performing a first encode

The following example uses `protoc:load()` with schema text, then encodes and decodes a Lua table. The README labels the inline schema as a demonstration and recommends using `protoc.new()` in a real application. The code is reproduced from the README.

local pb = require "pb"
local protoc = require "protoc"

assert(protoc:load [[
   message Phone {
      optional string name        = 1;
      optional int64  phonenumber = 2;
   }
   message Person {
      optional string name     = 1;
      optional int32  age      = 2;
      optional string address  = 3;
      repeated Phone  contacts = 4;
   }
 ]])

local data = {
   name = "ilse",
   age  = 18,
   contacts = {
      { name = "alice", phonenumber = 12312341234 },
      { name = "bob",   phonenumber = 45645674567 }
   }
}

local bytes = assert(pb.encode("Person", data))
local data2 = assert(pb.decode("Person", bytes))

`pb.tohex(bytes)` is also provided by the README for printing the encoded bytes as hexadecimal. The decode result is a Lua table. The example uses `serpent` only for formatted display, so a real application does not need `serpent` to encode or decode messages.

The schema names are significant. `Person` is referenced by name in both operations. The type naming rules also mention `.Foo` for a type without a package and `foo.Foo` for a type declared in package `foo`, so a schema with a package declaration may require a qualified message name.

Handling imports and unknown schema types

The `protoc` module provides configurable hooks for imports and unresolved types. `unknown_import` handles a missing schema module, while `unknown_type` handles an unresolved type name. The documented values for these handlers are `true`, a Lua pattern string, or a function.

Setting a handler to `true` gives all nonexistent modules or types a default value without triggering an error. A string is interpreted as a Lua pattern that identifies which unknown names should be treated as existing. For example, the README documents this pattern:

p.unknown_type = "Foo.*"

The pattern means that names prefixed by `Foo` are treated as existing types and do not trigger errors. This behavior is convenient for selective schema compilation, but it also means that an unresolved type can pass through compilation without being checked. Projects that need strict validation should avoid permissive values unless they understand the consequence.

A function receives the unknown module or type name. An import handler should return a `DescriptorProto` table produced by a loading operation. A type handler should return a type name and type, such as `message` or `enum`. The README shows handlers that substitute `my_foo.proto` for `foo.proto` and map an unknown type to a message under `.MyType`. `include_imports` is the separate option for automatically loading imported proto files.

Where the lower-level modules fit

The lower-level modules make starwing/lua-protobuf more than a table serializer. `pb.slice` focuses on wire-format decoding. `pb.buffer` provides an encoding buffer for basic protobuf types and can support streaming decode workflows. `pb.conv` handles integer conversion in the protobuf wire representation, while `pb.io` provides binary-mode access to standard streams and files.

These APIs suit code that needs incremental input, custom processing of encoded fields, or direct access to binary representation. They also introduce more implementation responsibility than the high-level `pb.encode()` and `pb.decode()` path. A streaming consumer must still understand how it receives field data, handles incomplete messages, and interacts with the application protocol.

The README does not document a complete public API for every lower-level function, nor does it provide examples of streaming message assembly. It supports identifying these modules and their stated purpose, but not promising a particular streaming interface or claiming specific recovery behavior. The displayed `pb` API table is also truncated, so implementation details beyond the listed functions should be checked against the repository.

Limitations and appropriate alternatives

The first limitation is deployment complexity. The high-level module includes a C extension, so source builds need Lua headers or libraries and a platform-appropriate compiler. The pure Lua `protoc.lua` component removes the need for Google's compiler when loading text schema, but the README does not say that all operations in the C-based `pb` module are implemented in `protoc.lua`.

The second limitation is schema strictness. Permissive `unknown_import` and `unknown_type` values can allow unresolved names to receive defaults. That can help a build succeed in a carefully controlled setup, but it can also conceal an incomplete schema. A project with multiple services and shared imports should decide whether errors should be strict or whether custom lookup rules are required.

For teams that already have generated Lua bindings from another protobuf implementation, starwing/lua-protobuf may be unnecessary because those bindings can provide message-specific encode and decode functions without requiring this schema loading workflow. The practical difference is architectural: this project centers on loading schemas and using generic `pb.encode()` and `pb.decode()` operations, while generated bindings usually expose message-specific code. The README does not identify a named alternative, so the comparison here is limited to that approach rather than a claim about a particular competitor.

License, maintenance, and upgrade considerations

The repository is licensed under MIT. That is the relevant license identifier for an engineering review, but adopting the project still requires checking the full `LICENSE` file and any dependency licenses separately. The README's example references `serpent`, but it uses that library only to format a decoded table for printing. The core encode and decode calls shown in the README do not require `serpent`.

The repository is not archived. Its last push was on 2026-07-02, and the latest listed release is 0.5.3 from 2025-05-29. That identifies a recent code push and a named release, but the README does not document a support policy, semantic versioning policy, automated compatibility process, or migration notes for each version. It also does not state which release should be selected for a particular Lua runtime.

An upgrade should therefore be assessed at the schema and API boundary. Confirm that the selected package still loads the project's compiled `*.pb` files or text schemas, and check whether unknown imports and types are handled differently. The repository contains `CMakeLists.txt`, `pb.c`, `pb.h`, `protoc.lua`, a rockspec directory, `luaunit.lua`, `serpent.lua`, and `test.lua`. Based on that layout, the project includes both native build files and Lua-level support files, but the README does not describe the exact upgrade automation.

Editorial conclusion

Adopt starwing/lua-protobuf when a Lua application needs protobuf wire data and can build or load the C module, or when the pure Lua schema compiler is sufficient. Do not choose it without confirming your protobuf syntax, required Lua runtime, compiler access, and import behavior. The first practical check is to load the schema used by your application and confirm that `pb.encode()` and `pb.decode()` preserve the required message shape.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. starwing/lua-protobuf on GitHub