multiprocess: the multiprocessing fork that serializes with dill

A fork whose only real change is how objects are serialized

The standard library's multiprocessing package can only hand an object to another process when pickle can handle it. Pickle handles module level functions and importable classes. It refuses a lambda, a closure, or a function assembled at runtime, and the resulting PicklingError is a familiar wall for anyone who has tried to fan work out over a Pool.

multiprocess is a fork of multiprocessing that changes that one piece. The project documentation describes it as extending multiprocessing to provide enhanced serialization, using dill, and using multiprocessing itself to support spawning processes through the API of the Python standard library's threading module. Everything else about the package, the Process class, the Pool class, queues, managers, locks, semaphores and conditions, is inherited from the standard library rather than reinvented.

That narrowness is the appeal. A program can change its import from multiprocessing to multiprocess and keep the rest of its code intact, and what it gains is the ability to send richer objects between workers. The package is also listed as a component of pathos, a framework for heterogeneous computing, so it is not designed as a standalone project with a separate design language.

The Process class keeps the threading.Thread shape

The most surprising property of the API is that Process mirrors threading.Thread, not multiprocessing.Process. The constructor takes a target and an args list, the child is started with start, and the parent waits with join. The guard clause in the example is the familiar one that keeps a spawn-based start from re-importing the main module.

Here is the example the project gives, with a string pushed onto a queue from the child and read back in the parent.

    from multiprocess import Process, Queue

    def f(q):
        q.put('hello world')

    if __name__ == '__main__':
        q = Queue()
        p = Process(target=f, args=[q])
        p.start()
        print (q.get())
        p.join()

Running it prints hello world once and exits. Note the print statement written with parentheses, which is the style Python 3 accepts but does not require. The README example for the synchronization primitives shows the same idea with a Condition object, whose printed representation changes once the lock is acquired, which is a quick way to see that the underlying primitives behave as the standard library does.

Pools and managers carry the standard library workload model

Because the fork keeps the standard library surface, a Pool of worker processes is created the same way and used the same way. The documentation shows a four worker pool, an asynchronous map over a range, and a timed get on the result handle.

    >>> from multiprocess import Pool
    >>> def f(x): return x*x
    ...
    >>> p = Pool(4)
    >>> result = p.map_async(f, range(10))
    >>> print (result.get(timeout=1))
    [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

The timeout on the get call is worth noticing, since it is the difference between a worker that eventually answers and a parent that hangs. The same class supports a manager for shared objects, either in shared memory for simple data or in a separate server process.

    >>> from multiprocess import Manager
    >>> manager = Manager()
    >>> l = manager.list(range(10))
    >>> l.reverse()
    >>> print (l)
    [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
    >>> print (repr(l))
    <Proxy[list] object at 0x00E1B3B0>

The manager returns a proxy rather than a real list, so every element access crosses back to the server process. That is the documented cost of the shared object model, and it is the reason the documentation also points at shared memory for simple data.

dill is what makes the closures example work

The last example in the project documentation is the clearest statement of the point of the fork. When dill is installed, serialization extends to most objects, and a map over a range sends a lambda that itself builds a lambda.

    >>> from multiprocess import Pool
    >>> p = Pool(4)
    >>> print (p.map(lambda x: (lambda y:y**2)(x) + x, xrange(10)))
    [0, 2, 6, 12, 20, 30, 42, 56, 72, 90]

Standard pickle refuses that function outright, so a team that hits this error in a Pool has two choices: restructure the code so everything sent across the boundary is importable, or install a serialization layer that can handle it. multiprocess takes the second route by making dill a hard requirement rather than an optional extra, listed at version 0.4.1 or newer.

There is a catch in that snippet, and it is in the documentation rather than the code. The call uses xrange, which does not exist in Python 3, while the stated requirements ask for python (or pypy) at version 3.10 or newer. Read the example as a shape of code rather than something to paste into a modern interpreter, and write your own range call instead.

The build picks a version directory and warns when it has to fall back

The package is not a single source tree that adapts at runtime. The repository root holds one directory per supported interpreter, py3.10 through py3.16, plus pypy3.10, pypy3.11 and pypy3.12, and setup.py selects one of them at build time by looking for a matching subdirectory.

If the interpreter's exact minor version has no directory, setup.py walks the minor version downward, and if it has to settle on an older one it prints a warning stating that the Python version is not currently supported and that it is reverting to the earlier minor. If nothing matches, the build raises a ValueError saying the Python version is not supported. Versions of Python before 3.10 are rejected outright at the top of setup.py with the message that versions of Python before 3.10 are not supported.

The packaging itself is a standard PEP 517 build.

[build-system]
# Further build requirements come from setup.py via the PEP 517 interface
requires = [
    "packaging",
    "setuptools>=42",
]
build-backend = "setuptools.build_meta"

The build backend is setuptools, with packaging and setuptools 42 or newer pulled in as build requirements, while setup.py remains the place where the version specific source selection happens. A practical consequence for adoption: the newest interpreter you can build against is the newest directory present in the tree, and a newer interpreter means a fallback build and a printed warning rather than a clean result.

Where the fork stops matching the standard library

The documentation states that the examples and documentation found in the Python standard library's multiprocessing documentation also apply to multiprocess, provided you import multiprocessing as multiprocess. Read carefully, that is a statement about shared behaviour, not about the import name, since the package is imported under its own name.

Two limits are worth planning around. First, the fork tracks the standard library through discrete version directories, so a new CPython minor release means either a fallback build with a warning or a wait for the project to add the directory. Second, the project documentation is not uniformly current: the installation notes still mention that a C compiler is required to build the included extension module for Python 2, while the requirements section states python (or pypy) at 3.10 or newer and the setup.py guard rejects anything older. The example code also retains xrange and the parenthesized print style, both of which signal text that was written for an earlier Python and never fully refreshed.

The project documentation points to the tests package as the best way to see real usage, and the suite can be run with python -m multiprocess.tests. It also links an examples directory under the py3.12 tree that demonstrates basic use cases and benchmarking for running Python code in parallel. No rollback procedure and no upgrade notes beyond the release history are documented.

A slow release cadence against a recent push

The repository is not archived, and the last push was on 2026-10-18, so the code is being touched even though the version history is slow. The listed releases are 0.70.19 on 2026-01-19, 0.70.18 on 2025-04-17 and 0.70.17 on 2024-09-30, which is roughly one release every year and a half. The project documentation describes the package as being in active development and asks for feedback and bug reports through its issue tracker, with a legacy tracker kept for older reports.

The licence situation is stated plainly in two places. The README says the package is distributed under a 3-clause BSD licence and is a fork of multiprocessing, and the header comment in setup.py repeats the same terms and points to the LICENSE file in the repository root for the full text. The repository root also carries a COPYING file, which is where the original multiprocessing licence notice is kept. GitHub itself labels the licence as a custom one that it cannot classify, so treat the LICENSE and COPYING files in the repository as the authoritative text rather than the badge.

The dependency floor is small and stated: dill at 0.4.1 or newer, setuptools at 42 or newer, and an interpreter of Python or PyPy at 3.10 or newer. Adding the package therefore adds one real runtime dependency beyond the standard library.

Editorial conclusion

multiprocess fits teams that already use multiprocessing and hit pickle failures on closures, lambdas or dynamically built functions, since the migration is an import swap and nothing else changes. It does not fit projects that want a lighter dependency set, that are bound to a fixed interpreter outside the listed version directories, or that need the guarantee that a process boundary will reject unpicklable objects. Before adopting it, check the newest version directory in the repository against the interpreter your team actually runs, confirm that dill and setuptools meet the stated minimums, and run the bundled test suite with python -m multiprocess.tests on your own machine.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. Releases
5. uqfoundation/multiprocess on GitHub