loky: a process pool that survives crashes and reuses its workers

A replacement for ProcessPoolExecutor, and the problem it came from

The stated aim of loky is to provide a dependable, cross-platform and cross-version implementation of the ProcessPoolExecutor class of concurrent.futures. The project description puts it more tersely, as a reusable executor for joblib.

The problem it came from is explained at the end of the README, in the section on why the project was named loky. The authors had bad experiences debugging deadlocks when using multiprocessing.Pool and concurrent.futures.ProcessPoolExecutor, especially when calling functions with non-picklable arguments or returned values. The name is a mix of a god, locks and the y that make it somehow cooler and nicer, chosen because they had dealt with so many deadlocks that they wanted some kind of invocation to repel them.

That origin explains the two things loky is actually good at. The first is crash handling. One of the major concerns in the standard multiprocessing and concurrent.futures modules is the ability of the Pool or Executor to handle crashes of worker processes, and this library intends to fix those possible deadlocks and send back meaningful errors.

The second is that the fixes were contributed upstream. The README notes that the implementation of concurrent.futures.ProcessPoolExecutor that comes with Python 3.7+ is as dependable as the executor from loky, but that the latter also works for older versions of Python.

So loky is best read as the origin of the modern ProcessPoolExecutor, still maintained as a standalone library for the cases the standard library does not cover. The primary language is Python, the licence is BSD-3-Clause, the default branch is master, and the last push was on 2026-09-30, the day after the 3.7.0 release on 2026-09-29.

Four properties, and the fork plus exec decision underneath them

The README lists four features, and each one is a deliberate difference from the standard library.

The first is consistent and safe spawn behavior. All processes are started using fork plus exec on POSIX systems, and the reason given is safety in interactions with third party libraries. On the contrary, multiprocessing.Pool uses fork without exec by default, causing third party runtimes to crash, with OpenMP and macOS Accelerate given as examples.

That is the central engineering decision in the library, and it has a cost. The plain fork approach is cheap because the child inherits the parent's memory image. The fork plus exec approach throws that image away and starts a fresh interpreter, which costs a process start and a re-import of everything the child needs. What it buys is that a native library initialised in the parent, an OpenMP runtime for example, is not in a broken state in the child, and the failure mode is a clean error rather than a hang inside a native library.

The second property is the reusable executor, described as a strategy to avoid re-spawning a complete executor every time. A singleton executor instance can be reused and dynamically resized if necessary across consecutive calls, to limit spawning and shutdown overhead, and the worker processes can be shut down automatically after a configurable idling timeout to free system resources.

The third is transparent cloudpickle integration, to call interactively defined functions and lambda expressions in parallel. It is also possible to register a custom pickler implementation to handle inter-process communications.

The fourth follows from the third. There is no need for the if __name__ == "__main__": guard in scripts, because the use of cloudpickle to call functions defined in the __main__ module means it is not required to protect the code calling parallel functions under Windows.

The dependency list in pyproject.toml reflects this. The only runtime dependency is cloudpickle, the project requires Python 3.10 or later, and the packages are loky and loky.backend. The optional tests group adds packaging, pytest and psutil.

get_reusable_executor, and what the idling timeout buys you

The usage section is one example, and it is worth reading closely because the API is a single factory function.

The basic usage of loky relies on the get_reusable_executor, which internally manages a custom ProcessPoolExecutor object, which is reused or re-spawned depending on the context.

import os
from time import sleep
from loky import get_reusable_executor


def say_hello(k):
    pid = os.getpid()
    print(f"Hello from {pid} with arg {k}")
    sleep(.01)
    return pid


# Create an executor with 4 worker processes that will
# automatically shutdown after idling for 2s
executor = get_reusable_executor(max_workers=4, timeout=2)

res = executor.submit(say_hello, 1)
print("Got results:", res.result())

results = executor.map(say_hello, range(50))
n_workers = len(set(results))
print("Number of used processes:", n_workers)
assert n_workers == 4

Two arguments carry the whole design. max_workers sets the pool size, and timeout is the idling timeout, so the four workers shut down automatically after idling for two seconds and free their resources. The example then submits one job, and maps fifty, and counts the distinct process ids in the results to check that exactly four workers were used.

That last assertion is the point of the example. A pool that respawns workers per call would still be correct, and the distinct pid count is how you notice it is not doing what you asked for.

The examples directory makes the reuse behaviour explicit with files named reuse_executor.py and reuseable_executor.py, alongside a cloudpickle_wrapper.py and a README.txt. The documentation at loky.readthedocs.io is where the advanced usage lives, per the README.

The upstream joblib project is the main consumer of this library, and the repository carries a CITATION.cff and a Zenodo DOI badge, which tells you the project is treated as a citable research dependency rather than an internal utility.

Installing it, and the psutil option worth knowing about

The recommended way to install loky is with pip.

pip install loky

Installing from sources is also documented, and it is a source install rather than an editable one.

git clone https://github.com/joblib/loky
cd loky
python setup.py install

The optional dependency is the interesting part. The README notes that loky has an optional dependency on psutil to allow early memory leak detections. That is a genuinely useful feature rather than a convenience, because the fork plus exec model means the parent process does not share the child's memory, so a leak in a worker is invisible from the parent unless the child's memory usage is read from outside. With psutil present, the executor can see a worker growing and act on it.

The test suite has its own requirements, and they are modest. To run the test suite you need the pytest, version 3 or later, and psutil modules.

    pip install -e .
    pytest .

The repository layout shows the same split between library and infrastructure. There is a loky package directory and a tests directory, plus continuous_integration, dev-requirements.txt, docs, examples, tools, a pre-commit-config.yaml, and a .codecov.yml with a .coveragerc. The pyproject.toml configures black with a line length of 79, which is a small detail that tells you the project keeps to a narrow column, and the version is dynamic, taken from the loky.__version__ attribute.

The project also carries a CHANGES.md, a TODO.md, a MANIFEST.in and a LICENSE.txt, and the README notes that the work is supported by the Center for Data Science, funded by the IDEX Paris-Saclay.

What loky does not do, and when the standard library is enough

Three boundaries are worth stating.

The first is the start-up cost, and it is the one people are surprised by. The fork plus exec model trades speed of process creation for safety with native libraries, and a workload with many short tasks and a small working set will feel the difference. If your tasks are large enough that the process start is noise, the trade is free. If they are milliseconds long, you are paying a significant fraction of your runtime on interpreter start-up, and the fix is to batch work per task rather than to abandon the executor.

The second is scope. This is a process pool for CPU work in CPython, and it has no notion of GPU contexts, shared memory between workers, or affinity pinning. If you are dispatching CUDA work, loky is the wrong layer, because a forked child does not inherit a usable GPU context and the library gives you no mechanism to manage that. There is also no scheduler, no priority, no rate limiting and no fault tolerance beyond reporting a crashed worker.

The third is that the standard library has caught up on the main axis. The README says the concurrent.futures.ProcessPoolExecutor that comes with Python 3.7+ is as dependable as the executor from loky. So for a project on a current Python, the case for loky is no longer crash resistance, which was the original motivation.

What loky still offers over the standard executor is the reusable executor with an idling timeout, the cloudpickle integration that lets you submit a closure or a lambda or a function defined in a notebook, and the custom pickler hook. In a notebook or a script with interactively defined helpers, the cloudpickle part is the decisive one, because the standard executor cannot ship a lambda at all.

The comparison with the other alternatives lines up the same way. multiprocessing.Pool is faster to start and deadlocks more often, which is where loky came from. A native extension or a vectorised library is faster still, and is the right answer when the work can be expressed as array operations rather than as independent calls. loky is for the case in between, where the work is genuinely parallel, the functions are ordinary Python, and the process lifecycle needs to be survivable.

Editorial conclusion

loky fits a Python codebase doing embarrassingly parallel work with closures, lambdas or interactively defined functions, or one that has hit the deadlocks multiprocessing.Pool is prone to, and it does not fit a project that needs GPU processes, shared memory semantics or a scheduler. Before adopting it, decide whether the spawn overhead of fork plus exec is acceptable for your workload, because that safety is exactly what makes it slower to start than a plain fork, and note that the release history jumps from 3.5.6 in 2025 to 3.7.0 in September 2026.

Frequently asked questions

Official sources

1. joblib/loky on GitHub
2. License: BSD-3-Clause
3. Project website
4. README
5. Releases