parquet-go: a Parquet reader and writer built for a low memory footprint

A Parquet implementation aimed at the cost of reading

parquet-go is a high-performance Go library to manipulate parquet files, initially developed at Twilio Segment, and the package documentation describes it as a high-performance Go package to read and write Parquet files.

The motivation section is unusually concrete about why a Go implementation was needed. Parquet has been established as a powerful solution for representing columnar data on persistent storage, achieving compression and query performance that let data sets reach the petabyte scale, and having data applications share a common format creates opportunities for interoperation across toolkits. But the creation and evolution of large scale data management systems, combined with realtime expectations, brings maintenance and performance requirements that existing Go solutions to Parquet were not addressing.

So the design goal is stated in terms of cost rather than convenience: high level APIs to read and write Parquet files, while keeping a low compute and memory footprint, so that the library can be used where data volumes and cost constraints demand efficiency.

That framing matters, because it decides what the library is optimised for. A Parquet reader can be built to be simple, or to be cheap in memory and CPU, and a library aimed at the second will make different tradeoffs about buffering and concurrency than one aimed at the first. The specification section puts the format itself in context, noting that columnar storage allows Parquet to store data more efficiently than, say, JSON or Protobuf, with a link to the Parquet Format Specification.

The licence is Apache-2.0, and the releases have been frequent: v0.30.2, v0.31.0 and v0.32.0 all on 2026-08-12, with the last push on 2026-09-23.

Struct tags are where the schema lives

The schema of a Parquet file in this library is a Go struct, and the details live in struct tags.

When using Go structs to define the schema of Parquet files, struct fields may include a parquet tag to configure properties of the Parquet column such as its name, compression, encoding, and logical type. The first value in the tag sets the column name, and additional comma-separated values set options.

type Record struct {
    ID        int64     `parquet:"id,delta"`
    Name      string    `parquet:"name,dict,zstd"`
    Timestamp int64     `parquet:"timestamp,timestamp(microsecond)"`
    Score     float64   `parquet:"score,split"`
    Tags      []string  `parquet:"tags,list"`
    Optional  *string   `parquet:"optional,optional"`
}

That one line of Go carries four decisions. The column name, the encoding, whether the value is optional, and for timestamps the logical type with its unit. The ID column uses delta encoding, the name uses dictionary encoding with zstd compression, the score uses the split encoding for floating point, and the tags field is a list.

The encoding vocabulary is the part worth learning properly, because it is where a Parquet file's size comes from. Dictionary encoding for repeated strings, delta for integers, split for floats, and a logical type such as timestamp with a microsecond unit are all decisions that a hand written Parquet writer would otherwise make for you, and getting them wrong is how you end up with a file three times larger than it needs to be.

Beyond the field tag there are three more, for the nested cases: map keys and values can be configured with the parquet-key and parquet-value tags, and list elements with the parquet-element tag. The full reference of supported tags, type constraints and examples lives in the SchemaOf documentation, which is the page to read before designing a schema for files that other systems will read.

Variant columns for data that is not really a column

The library supports the Parquet VARIANT logical type for storing semi-structured data, JSON-like values, in a columnar format.

The simplest way to use variants is with the variant struct tag, which automatically marshals and unmarshals Go values to and from the variant binary format:

type Event struct {
    ID   int64 `parquet:"id"`
    Data any   `parquet:"data,variant"`
}

writer := parquet.NewGenericWriter[Event](output)
writer.Write([]Event{
    {ID: 1, Data: "hello"},
    {ID: 2, Data: int32(42)},
    {ID: 3, Data: map[string]any{"key": "value"}},
})
writer.Close()

Note what the example is doing with the any field: a string, an int32 and a map all go into the same column, and the variant encoding records enough type information for a reader to recover the original value.

There is a second, more interesting mode. For shredded variants, which store typed columns alongside the raw value for faster queries, you build a schema with parquet.ShreddedVariant():

shreddedType, err := parquet.ShreddedVariant(parquet.String())
schema := parquet.NewSchema("Record", parquet.Group{
    "data": shreddedType,
})
writer := parquet.NewGenericWriter[Record](output, schema)

The trade is explicit. A plain variant column keeps the value as bytes, so a query that filters on a field inside it has to decode everything. A shredded variant materialises typed columns next to the raw value, so filtering pushes down into the columnar data, at the cost of a wider, more specialised file.

If you need the raw bytes rather than the decoded value, the library also documents a low-level route: a struct with Metadata and Value []byte fields.

Writing files, and the step everybody forgets

The writing API has two levels, a function for the simple case and a type for everything else.

A Parquet file is a collection of rows sharing the same schema, arranged in columns to support faster scan operations on subsets of the data set. For simple use cases, the parquet.WriteFile[T] function allows the creation of Parquet files on the file system from a slice of Go values representing the rows to write to the file:

if err := parquet.WriteFile("file.parquet", []RowType{
    {FirstName: "Bob"},
    {FirstName: "Alice"},
}); err != nil {
    ...
}

The GenericWriter[T] type is the general path. It denormalizes rows into columns, then encodes the columns into a Parquet file, generating row groups, column chunks, and pages based on configurable heuristics. Those three levels, row group, column chunk and page, are the physical structure of a Parquet file, and they are what a reader navigates when it projects a subset of the columns.

The step that is easy to skip is the close, and the documentation is explicit about why it matters: closing the writer is necessary to flush buffers and write the file footer. A Parquet file without its footer is not a Parquet file that a reader can parse, so a writer that is never closed produces something that looks like a file and is not one. Any code that returns early on an error path has to close the writer too, which is why the examples put the close outside the error branch.

The other option on the writer is the explicit schema. Declaring the Parquet schema on a writer is useful when the application needs to ensure that data written to a file adheres to a predefined schema, which may differ from what the Go struct would otherwise infer.

Pre-v1 is a promise about the future, not a guarantee

The compatibility section is the one to read before you depend on this package, and it is refreshingly direct.

The package is currently released as a pre-v1 version, which gives maintainers the freedom to break backward compatibility to help improve the APIs as they learn which initial design decisions would need to be revisited to better support the use cases the library solves for. The documentation adds that these occurrences are expected to be rare in frequency, and that documentation will be produced to guide users on how to adapt their programs to breaking changes.

That is a well-reasoned position rather than a warning label. A library like this starts with a schema model, a writer and a reader, and the first version of any such thing is usually wrong in a few places that only become obvious once real files exist. The maintainers are saying they will fix those places, and the cost of that is a version number below 1.

What it means for you is concrete. A major version bump is not a signal of instability in the file format, it is a signal that the Go API changed. If your code depends on the library across a long lifetime, you will be reading migration notes rather than bumping a patch version, and the documentation of those changes is something to plan for rather than discover.

The release record shows how fast the surface can move. Three versions were published on the same day in August 2026, and the project is under active development with a CHANGELOG and a CODEOWNERS file in the repository.

What the dependency list tells you about format support

The module manifest is short and every entry corresponds to a format or a platform concern.

There is brotli from andybalholm, klauspost/compress for the other compression families, and lz4 from pierrec, so the common Parquet codecs are all there. The parquet-go/bitpack and parquet-go/jsonlite modules are the project's own helper packages for bit packing and for its JSON handling, and twpayne/go-geom covers geometry types, which is why there is a column_buffer_geometry.go in the tree. google/uuid, golang.org/x/sys and google.golang.org/protobuf round out the list.

The file layout in the repository root is a good map of the feature surface. The column buffer implementations are separate files per physical type, with amd64 and be128 assembly variants for the hot paths, and there are dedicated files for boolean, byte array, double, float, int32, int64, int96, fixed length byte array, JSON, null, optional and protobuf values. There is a bloom filter directory and bloom filter files, a buffer pool, and a bloom.go.

Two things are worth noting about the toolchain. The manifest declares go 1.24.9, while the documentation says Go 1.22 or later is required to use the package, and that is a discrepancy to check against your own build environment rather than take on trust.

The Makefile shows how the project is tested and formatted: the test target runs go test with the verbose flag, trimpath, the race detector and coverage across the whole tree, and the format target runs go fmt followed by the modernize tool through a separate go.tools.mod file so the tool dependencies stay out of the main module.

Editorial conclusion

parquet-go suits a Go team writing or reading columnar data at a scale where scanning JSON is the bottleneck, and where the team wants the Parquet format without adopting a columnar engine such as Arrow as well. It is the wrong choice for a service that writes a handful of rows and reads them back once, since the columnar machinery buys nothing there, and it is the wrong choice for a team that cannot absorb API churn, because the package is pre-v1 and the maintainers say explicitly that they will break compatibility when the design needs it. Verify three things before you commit: that the Go version your build uses matches what the module requires, since the documentation and the manifest do not agree, that the encodings and logical types your existing Parquet files use are covered by the struct tag vocabulary, and that you are comfortable with the writer API changing between minor versions.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. parquet-go/parquet-go on GitHub
3. Project website
4. README
5. Releases