klauspost/reedsolomon: Reed-Solomon erasure coding in Go at 1GB/s per core

Backblaze's JavaReedSolomon adapted to Go, with SIMD throughput above 1GB/s per core

Reed-Solomon erasure coding splits a data set into N data shards and K parity shards. Any N shards from the full set of N+K can reconstruct the original data, which means K shards can be lost before recovery fails. This property makes it common in distributed storage, where drives may disappear independently.

klauspost/reedsolomon is a Go port of the JavaReedSolomon library that Backblaze released. The README attributes additional optimizations beyond the Java source, and the primary language listed in the repository is Assembly, reflecting the SIMD-accelerated Galois field arithmetic that drives the performance. The stated throughput is speeds exceeding 1GB/s per CPU core in pure Go, though the Assembly files indicate platform-specific acceleration for amd64, arm64, and ppc64le.

For shard counts above 256, the library switches to a Leopard implementation automatically, with speed close to the original Leopard C implementation.

Installing with go get and creating an encoder with reedsolomon.New(10, 3)

The README gives the installation command as a standard Go module fetch:

go get -u github.com/klauspost/reedsolomon

Go modules are recommended. The go.mod file specifies `go 1.24` as the minimum Go version, and the only direct dependency is `github.com/klauspost/cpuid/v2 v2.3.0`, which the library uses for CPU feature detection.

Creating an encoder requires choosing the number of data shards and parity shards. The README gives 10 data and 3 parity as an example:

enc, err := reedsolomon.New(10, 3)

This encoder handles all parity operations with that distribution. For repeated operations on a fixed shard size, adding `WithAutoGoroutines(shardSize)` as an additional parameter calculates the optimal number of goroutines for that size. The slice passed to the encoder is `[][]byte` where the outer slice has a length of 13 in this example: 10 data shards followed by 3 parity shards. To encode, call:

err = enc.Encode(data)

The error from Encode is non-nil only when the data shards are not of equal size. After a successful encode, the last 3 slices contain parity data.

The encoder cannot detect corruption; invalid shards must be nil before calling Reconstruct

Reconstruction is the main use case for erasure coding, but the encoder has a fundamental constraint: it does not know which shards are invalid. If a byte changes in a shard and the caller does not detect this, there is no way to reconstruct the correct data.

Any application that uses this library for corruption recovery must therefore implement its own per-shard hash check, verify each shard before reconstruction, and nil out only those that fail. Only after identifying bad shards should the caller set them to nil and call Reconstruct:

data[3] = nil
data[7] = nil
err := enc.Reconstruct(data)

With 3 parity shards configured, reconstruction succeeds when at most 3 shards are missing or nil. Passing more than 3 nil shards returns a failure.

Two narrower reconstruction functions are also available. `ReconstructData` skips regenerating parity shards and returns only the data portion, which saves work when the parity shards are not needed after recovery. `ReconstructSome` accepts a boolean slice specifying exactly which shards to regenerate, letting the caller skip shards that are not needed for the immediate operation. Both follow the same nil-to-mark-invalid convention and have the same upper bound on missing shards.

Split pads the last shard with zeros, and the exact input size must survive the round trip

For a single large byte slice, the library provides `Split` and `Join` helpers. `Split` divides the input into the number of data shards configured in the encoder and creates empty parity shards:

bigfile, _ := os.Readfile("myfile.data")
split, err := enc.Split(bigfile)

When the input length is not divisible by the number of data shards, `Split` inserts zeros in the last shard to make the lengths equal. Keeping track of the exact input size is necessary because those padding zeros become part of the encoded data and will appear in the reconstructed output unless trimmed with the correct original length.

To reverse the process, `Join` writes the shards back out through an `io.Writer` and takes the original length as a parameter:

err = enc.Join(io.Discard, data, len(bigfile))

The original length tells Join how many bytes to write before stopping, which trims the padding zeros. Passing the wrong length produces either truncated output or extra zero bytes at the end. There is no checksum embedded by Split itself, so the caller is responsible for preserving the original size.

Aligned allocation adds roughly 2.5% throughput on AMD64 at the 100x20x10000 shard configuration

For AMD64, the alignment of shard buffers affects throughput. A benchmark in the README compares the two cases:

BenchmarkEncodeUnaligned100x20x10000-32    	  100198	     36085 ns/op	33254.75 MB/s
BenchmarkEncodeAligned100x20x10000-32      	  102711	     35207 ns/op	34084.15 MB/s

The difference is 33254.75 MB/s versus 34084.15 MB/s. The gain is modest for this configuration but this is more pronounced with odd-sized shards.

The library exposes `AllocAligned(shards, each int) [][]byte` as a package-level function and also as a method on the extended encoder interface, which allocates slices aligned to a 64-byte boundary. Using `Split` produces unaligned shards because the input slice may start at any address, and re-aligning already allocated slices is not possible. For workloads where alignment matters, the approach is to allocate aligned shards first and fill them directly rather than calling `Split` and hoping for alignment.

Shard counts above 256 switch internally to a Leopard algorithm

The standard Reed-Solomon implementation in this library covers up to 256 shards. When the total of data and parity shards exceeds 256, the encoder selects a Leopard implementation automatically. For shard counts above 256, the library uses a Leopard implementation that performs close to the original Leopard C implementation in terms of speed, which means high-shard-count configurations are not penalized heavily compared to the C reference.

The switch is automatic: callers do not pass a flag or choose between implementations. The encoder returned by `reedsolomon.New` selects the right backend based on the shard count. However, the two code paths have different internal data structures, so an encoder created with a count below 256 cannot be reused for a count above 256 or vice versa.

For applications with very large shard distributions, the practical consequence is that the 256-shard boundary is a configuration decision worth testing. Throughput, memory layout and error handling at that boundary should be verified against the actual message sizes a deployment will use.

go.mod retracts four version ranges linked to specific PRs and an issue

A close reading of go.mod turns up four retract entries, each with a link to a specific bug report:

retract (
	v1.12.2 // https://github.com/klauspost/reedsolomon/pull/283
	v1.11.6 // https://github.com/klauspost/reedsolomon/issues/240
	[v1.11.3, v1.11.5] // https://github.com/klauspost/reedsolomon/pull/238
	v1.11.2 // https://github.com/klauspost/reedsolomon/pull/229
)

The range `[v1.11.3, v1.11.5]` covers every version between v1.11.3 and v1.11.5 inclusive. Together with v1.11.2 and v1.11.6, that spans most of the v1.11 series. v1.12.2 is retracted separately. Any dependency tree pinned to one of those six versions is running retracted code, and Go modules will warn on `go get` but will not prevent installation if the version is pinned explicitly.

The current release is v1.14.2 from 2026-08-19, and the last push was on 2026-09-25. The licence is MIT, with the full terms in the LICENSE file at the repository root.

Editorial conclusion

Use this package when your Go application needs erasure coding with speeds exceeding 1GB/s per CPU core and you are prepared to hash each shard before reconstruction. Skip it if your application needs automatic corruption detection, since the encoder cannot tell a valid shard from a corrupted one and requires you to nil out any suspect shards before calling Reconstruct. Before deploying, check go.mod: v1.12.2, v1.11.6, v1.11.2, and the range v1.11.3 through v1.11.5 are all retracted with linked bugs, so any existing build pinned to those versions needs to upgrade.

Frequently asked questions

Official sources

1. Issues
2. klauspost/reedsolomon on GitHub
3. License: MIT
4. README
5. Releases