# pgzip: parallel gzip that stays a standard gzip file

> pgzip is a Go package that compresses and decompresses gzip in parallel while producing and reading ordinary gzip files, so it can replace compress/gzip with a one line import change. It suits a service moving megabytes of data through gzip, and the README is explicit that below about a megabyte the standard library is faster.

**klauspost/pgzip** — Go parallel gzip (de)compression

- Repository: https://github.com/klauspost/pgzip
- Stars: 1,212 · Forks: 91
- Language: Go
- License: MIT
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/klauspost-pgzip

## A drop in replacement that writes ordinary gzip files

pgzip is Go parallel gzip compression and decompression, and the README describes it as a fully gzip compatible drop in replacement for compress/gzip. The mechanism on the compression side is to split the input into blocks that are compressed in parallel, and the output is a standard gzip file. On the decompression side the reader is modified so that it decompresses ahead of the current reader position, which means reads are non-blocking as long as the decompressor can keep ahead of the code consuming it, and the CRC calculation runs in its own goroutine.

The compatibility point is the one that decides whether you can use it. The library creates and reads standard gzip files, so you do not have to match the compressor and the decompressor to get the speedups, and the files are fully compatible with other gzip readers and writers. There is no container format, no index, no proprietary header. A file written by pgzip can be sent to a server that decompresses it with the standard library, and a file written by the standard library can be read by pgzip.

The swap itself is an import change.

```go
import gzip "github.com/klauspost/pgzip"
```

There is a zlib variant too, github.com/klauspost/pgzip/zlib, for code that used compress/zlib. The installation is a go get of the module, and the readme notes you may need to update the dependency on klauspost/compress, which is where the actual deflate implementation lives.

```bash
go get github.com/klauspost/pgzip/...
go get -u github.com/klauspost/compress
```

## Block size and concurrency are the two knobs that matter

Everything about the performance of pgzip comes down to two parameters, and the API exposes them on the writer through SetConcurrency, which takes a block size and a number of blocks. The default is SetConcurrency(1MB, runtime.GOMAXPROCS(0)), which means blocks are split at 1 MB and as many blocks as there are CPU threads can be in flight before the writer blocks.

```go
var b bytes.Buffer
w := gzip.NewWriter(&b)
w.SetConcurrency(100000, 10)
w.Write([]byte("hello, world\n"))
w.Close()
```

The readme gives the guidance directly. To get any performance gain you should be compressing more than a megabyte at a time. The block size should be at least 100k, and the number of blocks should match the number of cores you want to use, though about twice the number of blocks is better. There is also a second order effect worth understanding, because writes to the compressor only block once the compressor is already working through the number of blocks you specified, which means you do not have to buffer the input yourself before handing it over.

For decompression the same idea appears with a different function, pgzip.NewReaderN, which takes the reader plus a block size and a number of blocks, and gives you a reader with custom block sizes. The block size is the size of each block decoded, and the blocks value is the maximum number of blocks decoded ahead.

The threshold matters as much as the tuning. The readme is blunt about it. You should only use this if you are compressing or decompressing big amounts of data, say more than 1MB at a time, otherwise you will not see any benefit, and it is likely to be faster to use the internal gzip library or the klauspost/compress package instead.

## What the README benchmarks actually compare

The performance section points at a blog post on benchmarks of Go gzip and presents a table on the Matt Mahoney 10GB corpus at compression level 6, with GOMAXPROC set to 32 on a 16 core CPU.

| Compressor | Throughput | Speedup | Size | Size overhead |
| --- | --- | --- | --- | --- |
| gzip, Go standard library | 16.91 MB/s, 1 thread | 1.0x | 4781329307 | 0% |
| gzip, klauspost/compress | 127.10 MB/s, 1 thread | 7.52x | 4885366806 | +2.17% |
| pgzip | 2085.35 MB/s | 123.34x | 4886132566 | +2.19% |
| pargzip, Go build tools | 334.04 MB/s | 19.76x | 4786890417 | +0.12% |

Read the third column with care. The 123x figure for pgzip is against the Go standard library, which is single threaded, while the second row is klauspost/compress at 7.52x against the same baseline, so the gap between pgzip and a good single threaded implementation is roughly 16x rather than 123x. The size column tells the other half of the story. pgzip and klauspost/compress produce files about 2.2 percent larger than the standard library output, because their deflate implementation trades a little ratio for speed, while pargzip stays within 0.12 percent of the baseline. So the choice is speed against output size, and the answer depends on whether you are paying for storage and bandwidth.

For decompression the readme gives a separate table on a 4 CPU machine, with the standard library at 1m28.85s and pgzip at 43.48s, and it makes the point that this is more than a 100 percent speedup even though gzip decompression is inherently single threaded apart from the CRC. The explanation is that pgzip acts as a buffer, so the reader never waits on input. With the standard library you are also waiting on IO while decompressing, and if the decoder keeps up, data is always ready.

The readme also notes the compression cost is usually about 0.2 percent with default settings at a 250k block size, and points at a spreadsheet of results for different content types and settings.

## The zlib wrapper, the sibling project and the maintenance picture

A few structural details round out the picture. The library has a zlib subpackage, so code that used compress/zlib can switch by changing the import to github.com/klauspost/pgzip/zlib, keeping the same API shape.

The readme also names a Go alternative with a different emphasis, bgzf from biogo/hts, which has the same parallel feature and additionally allows seeking in the resulting file, at the cost of slightly more overhead than pgzip or plain gzip. The readme is careful about the limitation of bgzf, since it can only decompress files created by a compatible encoder and therefore cannot be considered a generic gzip decompressor, which is why it is excluded from the decompression comparison. If you need random access into compressed data, that difference matters more than throughput.

The other dependency worth naming is klauspost/compress, which provides the underlying deflate implementation and the huffman-only compression mode that pgzip exposes, described as allowing compression at roughly 450 MB per core per second largely independent of content. The readme points at that package's changelog, because changes there are carried over, so the effective release history of pgzip is the union of the two.

The repository is small, with gzip.go and gunzip.go at the top level alongside the zlib/ directory, a testdata/ directory and a set of focused test files, including gzip_unreliable_test.go, gzip_norace_test.go and writeto_stale_offset_test.go, which are the kind of names that indicate edge cases someone hit. The go.mod declares go 1.25 and depends on klauspost/compress v1.20.3.

The release history is worth reading for a different reason. v1.2.5 was tagged on 2020-09-02, v1.2.6 on 2023-05-04 and v1.2.7 on 2026-09-30, so the project has had long gaps between patch releases and then resumed. The changes list in the readme is dated too, with entries from 2015 and 2016 covering a fix for a destination writer returning an error, better buffer reuse to generate less garbage, output that no longer changes based on write sizes, decoder support for the io.WriterTo interface, and a reduction in allocations of about 35 using sync.Pool for roughly 15 percent overall speedup. The project is MIT licensed.

## Where pgzip is the wrong tool

Three cases argue against it. The first is a service handling small payloads, which is most of them. The readme says the library only helps above about 1MB and that below that the standard library is likely faster, and that is not a marginal difference, since a parallel implementation on a small input spends its time in setup and goroutine scheduling rather than in compression. A JSON API body of 40KB is not the workload this library was built for.

The second case is a workload where output size matters more than throughput. The table shows pgzip producing files about 2.2 percent larger than the Go standard library, and if you are compressing 10GB of data that is roughly 220MB of extra storage, plus the bandwidth cost of moving it. For log archiving or transfer of already-compressed payloads, that overhead is not worth the speed.

The third case is a system that needs to seek inside a compressed stream. gzip is a stream format and cannot be seeked, and pgzip does not change that. The readme points at bgzf for the seeking case, with the caveat that it is not a generic gzip format. If your design assumes random access into compressed data, that is a storage format decision and pgzip is not the place to make it.

There is a subtler constraint worth naming. The parallelism depends on having enough data to fill the blocks, and the blocks are compressed independently, so a block boundary in the output is a place where the compressor could not have referenced earlier data. That is the mechanism behind the size overhead in the table, and it is inherent to block parallel deflate rather than a tuning problem. The trade is fixed by the format rather than by the settings.

On the state of the project, the version is 1.2.7 and the last push was on 2026-10-14, after a release on 2026-09-30, so the tree is being worked on. The gap between v1.2.6 in May 2023 and v1.2.7 in September 2026 is a long one, and a team adopting this should read the changes list in the readme and the changelog in klauspost/compress, because the behaviour of the underlying deflate implementation is what changes most between those tags.

## Conclusion

pgzip suits a Go service that compresses or decompresses payloads of a megabyte or more, especially one where the reader can work while decompression runs ahead of it. It does not suit a service handling small bodies, where the readme says the standard library is likely faster, and it does not suit a project that needs seeking in the compressed file, which is what the bgzf alternative provides. Before you adopt it, measure on your own payload size and core count, because the block and concurrency settings are what determine the benefit, and note that the last push was on 2026-10-14 with the newest release v1.2.7 from 2026-09-30.

## FAQ

### How do I use pgzip as a replacement for compress/gzip?

Change the import from compress/gzip to gzip "github.com/klauspost/pgzip" and keep the same API, or import github.com/klauspost/pgzip/zlib in place of compress/zlib. The files it writes are standard gzip files that any gzip reader can open.

### What block size and concurrency should I set?

The readme says to compress more than a megabyte to see any benefit, use a block size of at least 100k, and set the number of blocks to match the cores you want to use, or about twice that. The default is SetConcurrency(1MB, runtime.GOMAXPROCS(0)).

### Is pgzip output compatible with other gzip tools?

Yes. The readme states that the library creates and reads standard gzip files, that you do not have to match the compressor and decompressor to get the speedups, and that the files are fully compatible with other gzip readers and writers. The cost is about 2.2 percent more output size than the Go standard library in the benchmark table.

### Can I seek inside a file compressed with pgzip?

Not with pgzip itself, since it produces ordinary gzip streams. The readme points at bgzf from biogo/hts as a Go alternative with the same parallel feature that additionally allows seeking, noting that it can only read files written by a compatible encoder.

## Sources

- [Issues](https://github.com/klauspost/pgzip/issues)
- [klauspost/pgzip on GitHub](https://github.com/klauspost/pgzip)
- [License: MIT](https://github.com/klauspost/pgzip/blob/master/LICENSE)
- [README](https://github.com/klauspost/pgzip/blob/master/README.md)
- [Releases](https://github.com/klauspost/pgzip/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/klauspost-pgzip