pycapnp: Cap'n Proto from Python, with the C library bundled or not

What the binding gives you, and what it asks of you

pycapnp is the Python binding for the Cap'n Proto serialization and RPC system, written in Cython, with the licence BSD 2-Clause. The library's own framing is straightforward, it is Cap'n Proto for Python, and the value of Cap'n Proto itself is that the schema is the source of truth and both ends generate code from it. The cost of that is a compiler. Building pycapnp needs a C++14 supported compiler, so gcc 6.1 or newer with 5 possibly working, clang 6 with 3.4 possibly working, or Visual Studio 2017 or newer, plus cmake for the bundled Cap'n Proto, ninja on macOS and Linux or Visual Studio 2017 on Windows, and Cap'n Proto 1.0 or newer, with 0.8.0 or newer also working if you link to system libraries. You also need the Python development headers, meaning Python.h, which the official distributables from python.org include but which are usually a separate package on Linux distributions.

Installing, and the three ways to get Cap'n Proto

The short path is one command.

pip install pycapnp

You can set the CC environment variable to control which compiler is used, the example given being CC=gcc-8.2 pip install pycapnp. Or you can clone the repository and install it from there.

git clone https://github.com/capnproto/pycapnp.git
cd pycapnp
pip install .

The interesting decision is what happens to the C++ library during that build. By default the setup script will automatically use the locally installed Cap'n Proto, and if Cap'n Proto is not installed it will bundle and build the matching Cap'n Proto library. You can override the default in either direction, and forcing the bundled library is one flag.

pip install . -C force-bundled-libcapnp=True

Forcing the system library is the mirror image.

pip install . -C force-system-libcapnp=True

And there is a third option, which is to build against the latest upstream C++ Cap'n Proto rather than the pinned one.

pip install . \
    -C force-bundled-libcapnp=True \
    -C libcapnp-url="https://github.com/capnproto/capnproto/archive/master.tar.gz"

The one caveat the documentation gives about that scheme is worth remembering. The bundling system isn't that smart, so it might be necessary to clean up the bundled build when changing versions, and the command for that is a setup.py clean.

python setup.py clean

The address book example, which is the whole API in miniature

The examples directory is described as generally kept up to date with the recommended usage of the library, and the address book example is reproduced in full in the readme. The first half writes a message.

import capnp

import addressbook_capnp

def writeAddressBook(file):
    addresses = addressbook_capnp.AddressBook.new_message()
    people = addresses.init('people', 2)

    alice = people[0]
    alice.id = 123
    alice.name = 'Alice'
    alice.email = '[email protected]'
    alicePhones = alice.init('phones', 1)
    alicePhones[0].number = "555-1212"
    alicePhones[0].type = 'mobile'
    alice.employment.school = "MIT"

The shape to notice is that a message is created with new_message, a repeated field is sized with init and a count, and then the fields are assigned as plain attributes. The employment field is the union case, and you write to the arm you want, here alice.employment.school, and reading it back requires asking which arm is set. The second half reads the message and prints it, and the union handling is the part that shows the API is not just attribute access.

def printAddressBook(file):
    addresses = addressbook_capnp.AddressBook.read(file)

    for person in addresses.people:
        print(person.name, ':', person.email)
        for phone in person.phones:
            print(phone.type, ':', phone.number)

        which = person.employment.which()
        print(which)

        if which == 'unemployed':
            print('unemployed')
        elif which == 'employer':
            print('employer:', person.employment.employer)
        elif which == 'school':
            print('student at:', person.employment.school)
        elif which == 'selfEmployed':
            print('self employed')
        print()

There is a schema file next to it, examples/addressbook.capnp, which is where the message definition lives, and the generated module addressbook_capnp is what the example imports. That is the whole workflow, you write a schema, you generate a module, and you use it.

RPC, and the promise style API

The readme notes that pycapnp has gained RPC features that include pipelining and a promise style API, and it points at the calculator example for a much better demonstration than prose could give. The examples directory shows how wide that support is, with a synchronous calculator client and server, an asynchronous pair of the same, an asynchronous SSL calculator pair, an async client and async server pair, an async socket message client and server, an async reconnecting SSL client, and an async SSL client and server. The presence of both the synchronous and the asynchronous variants is the interesting part, because it means the same protocol can be served from a blocking thread or from an event loop, and the example set covers the cases people actually hit, TLS with selfsigned.cert and selfsigned.key in the directory, reconnection, and a message-oriented transport.

The packaging setup, and what the build system does

The build is not a plain setuptools build, and the pyproject.toml shows why. The build backend is a local one, with backend-path pointing at the _custom_build directory, and the requirements are setuptools, wheel, pkgconfig and cython 3.0 or newer. The custom backend exists because the build has to compile or fetch a C++ library before the extension can be built, which a stock setuptools backend cannot express. The runtime dependency list is short, with jinja2 as the only one declared, because the C++ library is where the real dependency weight sits. The version is assembled in setup.py from four constants, MAJOR 2, MINOR 2 and MICRO 4, and a separate step writes a version.py file into the capnp package so that the Python and the bundled C++ library versions can both be reported at runtime.

Testing, releasing and the parts you can rely on

The development workflow has moved off git flow, with master as the branch, and testing uses pipenv or a manual install of requirements.txt followed by pytest.

pip install pipenv
pipenv install
pipenv run pytest

The test dependency group is unusual and revealing, because it lists pytest alongside anyio, pytest-asyncio, pytest-tornasync, pytest-trio, pytest-twisted and twisted. In other words, the RPC examples are tested against several async frameworks rather than one, which is the right way to make sure the asynchronous API does not quietly depend on a single event loop. The release process is automated. Wheels and the sdist are built by the Build GitHub Actions workflow, wheels.yml, for every push including tag pushes, and a scripts/release-pypi.sh helper downloads those artifacts for a given tag or an explicit run id and uploads them to PyPI via twine.

git tag v2.2.1
git push origin v2.2.1

The helper resolves the latest successful wheels run for the tag, downloads the cibw artifacts, runs twine check, prints the file list and prompts before running twine upload, and it has a dry run mode that targets TestPyPI instead of real PyPI, which is the sort of thing that makes a release repeatable.

The bits that will trip you up

Four details are worth knowing before you commit. The first is 32-bit Linux, which requires that Cap'n Proto be compiled with -fPIC, a setting usually correct unless you are compiling Cap'n Proto yourself, and which is also written as -DCMAKE_POSITION_INDEPENDENT_CODE=1 for cmake. The second is Python versions, where 3.9 or newer is supported, and the project metadata says the same with requires-python of 3.9 or newer, so an older interpreter is out. The third is the stub files. While not directly supported by pycapnp, a tool has been created to help generate pycapnp stub files to assist with development, which the readme says is very helpful if you are new to pycapnp, and the tool lives in a separate GitLab project. The fourth is the Cython requirement, since pycapnp has additional development dependencies including cython and pytest, with the full list in requirements.txt.

Editorial conclusion

pycapnp fits a Python service that wants a compact binary format with a schema, a generated API and RPC without hand-writing a protocol, and it fits anyone who needs the same wire format in Python and in a C++ service elsewhere. It does not fit a Python-only deployment that wants no compiler on the machine, because the extension module has to be built somewhere unless a wheel covers the platform. It also does not fit a team that expects the library to keep pace with the upstream C++ implementation automatically, because the bundled Cap'n Proto is pinned to a matching version rather than tracking master. Before you install, decide between the bundled and system options, read the requirements list, because a C++14 compiler, cmake, ninja on macOS and Linux and the Python development headers are all needed to build from source, and check the Python version, since 3.9 or newer is supported.

Frequently asked questions

Official sources

1. capnproto/pycapnp on GitHub
2. Issues
3. License: BSD-2-Clause
4. README