The 100 MiB buffer cap arrived as a breaking change in 1.0

Four names for two operations, and why they exist

The API surface looks redundant until you know why. packb packs to bytes and unpackb unpacks from bytes, while dumps and loads exist as aliases for compatibility with the json and pickle modules, so code that used to call json.loads can call msgpack.loads with less thought. Then pack and dump pack to a file-like object, and unpack and load unpack from one. The reason is that the byte-oriented names are the primitive operations and the dump and load names are the stream-oriented ones, which happen to collide with the vocabulary everyone already knows from the standard library. The minimal example shows the byte form:

>>> import msgpack
>>> msgpack.packb([1, 2, 3])
'\x93\x01\x02\x03'
>>> msgpack.unpackb(_)
[1, 2, 3]

The leading 0x93 is the array header for three elements, so the entire list is four bytes on the wire. The rest of the options live in the docstrings of packb and unpackb, which the README points to rather than duplicating.

An Unpacker that cannot be reused after the wrong exception

The streaming API is Unpacker, which unpacks multiple objects from one stream or from bytes handed to its feed method. The common shape writes a hundred objects into a BytesIO buffer, seeks back to the start, and iterates the Unpacker, printing each object. The important detail is a warning rather than an example. If Unpacker.unpack() stops with an exception other than OutOfData, that Unpacker cannot be reused, and you have to create a new one when reading another stream. OutOfData is the normal way a stream ends, so the distinction is between a clean end of input and a malformed message in the middle of one. Once the unpacker has hit a real error its internal state is no longer meaningful, and reusing it would silently produce wrong values rather than failing. For anyone writing a loop that reads from a socket, that rule is the difference between a recoverable error and a corrupted stream.

Two options guard untrusted input, and one changed its default

Two options are documented for unpacking data from an unreliable source for unpacking data from an unreliable source. max_buffer_size has a default of 100*1024*1024, which is 100 MiB, and it limits both the internal buffer size and preallocated list sizes. The second half of that matters: a malicious message declaring a huge array does not get rejected, it gets a large allocation reserved for it. strict_map_key defaults to True and limits map keys to bytes and str, because the MessagePack specification itself does not restrict map key types and accepting arbitrary hashable objects opens a hash denial of service. If you genuinely need other key types, strict_map_key=False is the switch. Both defaults changed in the 1.0 release: max_buffer_size moved from 0, meaning unlimited, to 100 MiB specifically to avoid denial of service attacks. A code path that relied on the old default now fails on large messages rather than consuming memory until it does not.

use_bin_type and raw form a compatibility fork

Early MessagePack versions did not distinguish strings from binary data. The type covering both was named raw, and that ambiguity is still visible in the API as a pair of options. use_bin_type=False packs into the old specification, and raw=True unpacks from it, which is how a message containing the bytes b'spam' and the string 'eggs' comes back as the pair b'spam', b'eggs'. In 1.0 the defaults flipped: Packer now uses use_bin_type=True so bytes are encoded as the bin type, and Unpacker uses raw=False so str values are assumed to be valid UTF-8 and decoded into Python str objects. The encoding option was removed entirely because UTF-8 is always used. The practical consequence is that reading data produced by an old peer, or by a non-Python implementation that has not moved on, needs both flags set explicitly, and the same applies in reverse when you write for an old consumer.

The performance tips target the garbage collector

The performance section is short and neither tip is about the codec. The first is that CPython's garbage collector starts when the number of allocated objects grows, which means unpacking a large message can trigger collections that are not needed for correctness, and gc.disable() during a large unpack avoids them. The second is that a list is Python's default sequence type while a tuple is lighter, so use_list=False while unpacking reduces allocation when performance matters. Both are about the object graph the decoder produces rather than the byte parsing, which is a fair reflection of where a serialization library spends its time once the format is compact. The repository has a benchmark/ directory, so the measurable version of these claims is available rather than asserted, and the Makefile at the root is the entry point for running it.

The extension module skips PyPy and the fallback takes over

There are two implementations in the package and the README says which is which. A C extension, msgpack._cmsgpack, does not support PyPy, and and msgpack.fallback is the pure Python implementation used in that case. On Windows the situation is different: if you cannot use a binary distribution you need to install Visual Studio or the Windows SDK, and without the extension the pure Python implementation on CPython runs slowly. So the fallback is not just a portability nicety, it is a large performance difference on CPython, and the decision about which one you get happens at install time based on what your platform offers. Installing is one line:

pip install msgpack

The package on PyPI was named msgpack-python until version 0.5 and is now msgpack, so the upgrade note is explicit: coming from 0.4 or earlier, uninstall msgpack-python before installing the new one.

Where msgpack is the wrong choice

Four cases. The first is interop with something that only speaks JSON. MessagePack is faster and smaller, which the README says plainly, but that advantage only exists if both ends speak it, and a browser or a partner API that accepts JSON gives you no way to use it. The second is a message larger than 100 MiB. A default max_buffer_size caps both the internal buffer and preallocated list sizes, so a legitimate large payload needs the limit raised consciously, and if you raise it you have removed the protection that was added in 1.0. The third is untrusted input that needs more than two options. The security story here amounts to a buffer cap and a map key restriction, which is a real improvement over an uncapped decoder and is still a short list. The fourth is a schema that must be enforced. MessagePack carries types, not schemas, so a decoded object is a dict with whatever keys arrived, and the validation work is yours on top. The ext type helps for custom values and still does not validate anything.

Editorial conclusion

msgpack suits a Python service exchanging compact structured data with other MessagePack implementations, especially over a stream where you want to unpack one object at a time. It does not suit interop with a system that only speaks JSON, and it does not validate anything about the shape of what it decodes. Before adopting, install with pip install msgpack, leave strict_map_key at its default unless you need exotic map keys, and set max_buffer_size consciously for any message near the 100 MiB cap.

Frequently asked questions

Official sources

1. Issues
2. msgpack/msgpack-python on GitHub
3. Project website
4. README
5. Releases