vtprotobuf: a protoc plugin that trades reflection for generated Go marshalling code

A companion plugin, not a replacement

vtprotobuf is the Vitess Protocol Buffers compiler, and the repository provides the `protoc-gen-go-vtproto` plug-in for `protoc`, which is used by Vitess to generate optimized marshall and unmarshal code.

The first thing the documentation is careful about is the relationship to the upstream generator. vtprotobuf is implemented as a helper plug-in that must be run alongside the upstream `protoc-gen-go` generator, as it generates fully-compatible auxiliary code to speed up the serialization and deserialization of Protocol Buffer messages.

That is a materially different design from the alternatives. A plugin that replaces the upstream generator would have to reimplement everything the upstream one does, including the message types, the service stubs, and the registration, and it would be permanently behind. A helper plugin that emits extra methods alongside the normal generated code can be added to an existing build without touching it, and the upstream generator keeps doing the job it is good at.

The lineage is also stated. The code generated by this compiler is based on the optimized code generated by gogo/protobuf, although this package is not a fork of the original gogo compiler, as it has been implemented to support the new ProtoBuf APIv2 packages.

That distinction is the important technical fact about the project. gogo/protobuf was a well-known approach to generating reflection-free Protobuf code by hooking into the generator, and it worked by patching the upstream generator. By the time of APIv2, the extension point had changed, so the same technique needed a reimplementation. vtprotobuf is that reimplementation, aimed at the modern Protobuf API.

The licence is BSD-3-Clause, the language is Go, and the topics include codegen, grpc, protobuf, and vitess.

Nine features, and the differences are in the semantics

The feature list is the substance of the project, and it is worth reading for the places where the generated method does not behave exactly like the reflection-based original.

The `size` feature generates a `SizeVT()` helper that behaves identically to calling `proto.Size(p)`, except the size calculation is fully unrolled and does not use reflection. It is also used by the `marshal` codegen to ensure the destination buffer is properly sized before a message is marshalled to it, which is the mechanism that makes the marshal path allocation-free.

The `equal` feature generates `EqualVT`, which behaves almost identically to `proto.Equal(this, that)` except the calculation is unrolled, and `EqualMessageVT`, which takes a `proto.Message` and returns false if the argument is not of the right type. The uniform signature of the second method is the interesting part: it allows a generic `func EqualVT(proto.Message, proto.Message) bool` to be implemented without reflection, through a type assertion.

The `marshal` feature generates `MarshalVT`, which behaves identically to `proto.Marshal(p)` except the marshalling is fully unrolled and does not use reflection or allocate memory, plus `MarshalToVT` for marshalling into an existing buffer, which panics if the buffer is too small and returns the number of bytes written, and `MarshalToSizedBufferVT`, which expects the buffer to be exactly the right size and also panics otherwise. The `MarshalToVT` variant is explicitly for memory pooling, re-using serialization buffers.

The `marshal_strict` feature is a semantic change rather than a performance one. The strict variants marshal fields in the strict order by the field numbers they were declared in the .proto file, where the normal generated code is free to write fields in a different order for efficiency. Same bytes, but a different order, which matters if something downstream depends on the wire order.

So most features are drop-in replacements that are faster, and one of them changes the output bytes order. That is the distinction a team has to check before switching it on.

The unmarshal path has a trap, and the unsafe variant has a rule

The `unmarshal` feature is where the documentation is most careful, and reading it carefully is worth the time.

It generates `UnmarshalVT(data []byte)` which behaves similarly to `proto.Unmarshal(data, p)`, except the unmarshalling is performed by unrolled codegen without using reflection and allocating as little memory as possible. Then the catch: if the receiver `p` is not fully zeroed out, the unmarshal call will actually behave like `proto.Merge(data, p)`. The reason given is that the `proto.Unmarshal` in the Protobuf API is implemented by resetting the destination message and then calling `proto.Merge` on it. To ensure proper `Unmarshal` semantics, ensure you have called `proto.Reset` on your message before calling `UnmarshalVT`, or that your message has been newly allocated.

That is a real behavioural difference, and it is the kind of thing that produces a bug that looks like a data corruption issue months later. The reflection-based `proto.Unmarshal` resets the destination for you, and the generated one does not.

The `ignoreUnknownFields` option is mentioned alongside, and it can be used to ignore unknown fields in protobuf messages and further reduce memory allocations, which is a natural pairing, since the unrolled path can skip fields it does not recognise entirely instead of storing them.

The `unmarshal_unsafe` feature is the sharpest tool in the list. It generates `UnmarshalVTUnsafe(data []byte)`, which behaves like `UnmarshalVT` except it unsafely casts slices of data to `bytes` and `string` fields instead of copying them to newly allocated arrays, so that it performs fewer allocations. The rule is stated in bold: data received from the wire has to be left untouched for the lifetime of the message. Otherwise, the message's `bytes` and `string` fields can be corrupted.

That is a zero-copy unmarshal, and it is the fastest option, and it transfers ownership of the input buffer to the message. If the buffer is pooled and reused, the message's strings change underneath it. The name says unsafe and the documentation says so plainly.

Pooling and cloning, and the cost of forgetting to return

The `pool` feature generates three methods that together implement a per-type memory pool.

`ResetVT()` behaves similarly to `proto.Reset(p)`, except it keeps as much memory as possible available on the message, so that further calls to `UnmarshalVT` on the same message will need to allocate less memory. The documentation says this is an API meant to be used with memory pools and does not need to be used directly.

`ReturnToVTPool()` returns the message to a local memory pool so it can be reused later, clearing the object properly with `ResetVT` before storing it on the pool. The warning is explicit: this method should only be used on messages that were obtained from a memory pool by calling `YourProtoFromVTPool`, and using the message after calling this method will lead to undefined behavior.

`YourProtoFromVTPool()` returns a message from the local memory pool, or allocates a new one if the pool is currently empty. The returned message is always empty and ready to be used, for example by calling `UnmarshalVT` on it. Once the message has been processed, it must be returned to the pool by calling `ReturnToVTPool()`.

And then the sentence that makes the whole design coherent: returning the message to the pool is not mandatory, it does not leak memory, but if you don't return it, that defeats the whole point of memory pooling.

That is a good API design decision, making the fast path the fast path without making the slow path wrong. Forgetting to return a pooled message costs performance, not correctness, which is the right way round for an optimisation that is easy to forget.

The `clone` feature generates `CloneVT()`, which behaves similarly to `proto.Clone(p)` except the cloning is performed by unrolled codegen without using reflection, returning a typed `nil` if the receiver is `nil`, and `CloneMessageVT()`, which provides the same uniform-signature benefit as the equal methods so a generic clone can be written without reflection.

The build vendors protoc and generates its own extensions

The Makefile shows how the project builds and, more interestingly, how it tests itself.

The two binaries are installed into a local `bin` directory, with `GOBIN` set to that path. `protoc-gen-go-vtproto` is installed with `go install -buildvcs=false -tags protolegacy ./cmd/protoc-gen-go-vtproto`, and `protoc-gen-go` is installed with `go install -tags protolegacy google.golang.org/protobuf/cmd/protoc-gen-go`. The `protolegacy` build tag on both is worth noting, since it selects the legacy API compatibility path in the Protobuf runtime, and `-buildvcs=false` is there because the build happens in an environment without the VCS metadata it would otherwise try to stamp.

The `PROTOBUF_ROOT` is set to a vendored copy of the Protobuf source at `_vendor/protobuf-21.12`. That is a specific version pin of the Protobuf source tree, used because the conformance test needs `protoc` itself and the well-known type definitions, and vendoring pins them rather than depending on whatever the machine happens to have.

The `gen-conformance` target is the most interesting. It runs the vendored `protoc` over the Protobuf conformance test message definitions and the project's own `conformance/conformance.proto`, generating both the standard Go code and the vtproto code, with the `M` mappings redirecting the Protobuf test messages into an `internal/conformance` package. Running the upstream conformance suite against the generated code is a strong correctness check, because the suite is the reference implementation's own test set for wire format behaviour.

The other targets generate the project's own extension code, `gen-include` producing `ext.proto` code into the `vtproto` directory, and `gen-wkt` for the well-known types.

The repository layout follows the same structure. There is `cmd/` for the plugin entry point, `generator/` and `features/` for the code generation itself, `codec/` and `types/` for the runtime support, `protohelpers/` for the helper methods, `include/` for the extension definitions, `conformance/` for the test suite, `testproto/` for the test messages, and `doc/` for the documentation. The `protobuf.sh` script at the root and the `go.mod` with Go 1.26.4 and `google.golang.org/protobuf` v1.36.10 complete the picture.

Where vtprotobuf is the wrong tool

The limits here are precise, and they come from the same design that gives the project its value.

First, the speedup only matters where serialisation is hot. vtprotobuf removes reflection from the marshalling path, and reflection in Go is not catastrophically slow, it is a cost that only shows up when you are doing it in a tight loop or under load. A service that marshals a few thousand messages a second will not notice. A service doing hundreds of thousands, or a Vitess node on a hot query path, will. If you cannot point at the path where it matters, the extra generated code is a cost with no benefit.

Second, it only applies to the Protobuf APIv2 packages. The project states plainly that it is not a fork of the original gogo compiler, as it has been implemented to support the new ProtoBuf APIv2 packages. A codebase still on the old `github.com/golang/protobuf` runtime is not a candidate, and migrating the runtime is a larger project than adopting this plugin.

Third, the semantics are not identical across all the features, and the differences are the kind that a test suite catches only if the test suite exercises the edge case. The unmarshal path does not reset the destination, the strict marshal writes fields in a different order, and the unsafe unmarshal aliases the input buffer. Each of those is documented, and each of those is a decision you are making rather than a change you are inheriting.

Fourth, you now have two generators. Every `protoc` invocation in the build has to run both plugins, and every developer who regenerates protos has to have both binaries available, which is why the project ships a `protobuf.sh` script and vendored protoc. Tooling that regenerates protos automatically, in CI or in a pre-commit hook, has to be updated to match. A build that silently stops running the vtproto plugin will compile and pass, and the optimisation quietly disappears, which is a slow failure rather than a loud one.

Fifth, the release cadence is slow. The most recent tags are v0.4.0 in January 2023, v0.5.0 in September 2023, and v0.6.0 in January 2024, and the last push was on 2026-07-02. So the code is being maintained while the published versions are more than two years old, and the go.mod now requires Go 1.26.4, which is much newer than the v0.6.0 tag would suggest. Anyone pinning a version is pinning a much older state of the code than the current tree.

Editorial conclusion

vtprotobuf fits a Go service on the Protobuf APIv2 that marshals or unmarshals messages in a hot path and can measure the difference, especially one already running Vitess. It is the wrong choice for a project whose Protobuf traffic is not hot, and the wrong choice for a team that cannot add a second code generator to its build. Check three things before you adopt it: that you are on the Protobuf APIv2 packages, since the plugin is written for them and the whole point is that it is not a fork of gogo, that you turn on only the features whose semantics you have checked, since several of them change behaviour rather than just speed, and that the generated code is regenerated and reviewed like any other generated artefact in your build.

Frequently asked questions

Official sources

1. Issues
2. License: BSD-3-Clause
3. planetscale/vtprotobuf on GitHub
4. README
5. Releases