amphp/parallel: running blocking PHP code on worker processes or threads

What the library is and the two levels it offers

amphp/parallel is part of AMPHP, a collection of event-driven libraries for PHP designed with fibers and concurrency in mind. This particular package provides true parallel processing for PHP using multiple processes or threads, without blocking and with no extensions required.

That last clause is the significant one. PHP's mainstream story for parallelism has been ext-parallel, a native extension, and before that pcntl with its fork limitations. This library gets genuine parallelism from processes and threads using the standard library, and threads are available through ext-parallel when you want them.

The README describes two levels of use, and the framing is deliberate. The library comes with a collection of non-blocking concurrency tools that can be used independently as needed, and with an opinionated worker API that lets you assign units of work to a pool of worker processes. So you can take the small tools, or take the worker abstraction, and the second one is the higher-level path.

The word true in the parallelism claim matters too. It distinguishes this from a cooperative concurrency trick where a single thread interleaves work at yield points. Multiple processes or threads really do execute at the same time, which is what you need for CPU-bound work, and it is also why the mechanism is more expensive than an async call.

The requirements are PHP 8.1 or newer. Using threads instead of processes additionally needs PHP 8.2 or newer in ZTS mode and the ext-parallel extension. The README makes it clear which is the optional path.

The basic pattern: submit a task, await a future

The usage example is a set of URLs fetched concurrently by a worker pool rather than one after another:

$executions = [];
foreach ($urls as $url) {
    // FetchTask is just an example, you'll have to implement
    // the Task interface for your task.
    $executions[$url] = Worker\submit(new FetchTask($url));
}

// Each submission returns an Execution instance to allow two-way
// communication with a task. Here we're only interested in the
// task result, so we use the Future from Execution::getFuture()
$responses = Future\await(array_map(
    fn (Worker\Execution $e) => $e->getFuture(),
    $executions,
));

The shape is the important part. You submit a task to a worker, and each submission returns an Execution instance, which allows two-way communication with the task. When you only want the result, you take the Future from Execution::getFuture() and await the collection.

That separation is what makes this usable inside an event loop. The submission returns immediately with a handle, and the work happens elsewhere. The main process is not blocked while the workers run, so the event loop keeps serving other requests while the batch completes.

The README then steers you away from using this for the example it just used. FetchTask is only an example for a blocking function. If you just want to fetch multiple HTTP resources concurrently, it says, it is better to use amphp/http-client, the non-blocking HTTP client. That is the correct advice and it is worth stating plainly: parallelism is the heavy tool, and if your work is IO-bound rather than CPU-bound, an async client does the same job with none of the cost.

One constraint is flagged as a note. The functions you call must be predefined or autoloadable by Composer, so they also exist in the worker process or thread. That follows from how workers start, and it is the first thing to check when a task fails to run.

Tasks: the interface, the serialisation rule and why it matters

The Task interface has a single run() method, which is invoked in the worker to dispatch the work that needs to be done. The run() method can be written using blocking code, since the code is executed in a separate process or thread.

// FetchTask.php
// Tasks must be defined in a file which can be loaded by the composer autoloader.

use Amp\Cancellation;
use Amp\Parallel\Worker\Task;
use Amp\Sync\Channel;

class FetchTask implements Task
{
    public function __construct(
        private readonly string $url,
    ) {
    }

    public function run(Channel $channel, Cancellation $cancellation): string
    {
        return file_get_contents($this->url); // Example blocking function
    }
}

The rule that governs everything else is that task instances are serialised in the main process and unserialised in the worker, which means all data passed between the main process and a worker needs to be serialisable.

That constraint has teeth. A task carrying a database connection, a closure, an open stream or a resource handle cannot be serialised, and the failure happens at submit time rather than deep inside a worker. It is the reason the task takes a plain string URL in the example rather than a client object.

Two consequences follow. First, tasks are effectively data carriers with a class name, and the class must be loadable in the worker, which is why the file has to be autoloadable by Composer. Second, because the run() method receives a Channel, a task can send intermediate messages back to the parent, which is the two-way communication the Execution instance exposes.

Child processes or threads executing tasks may be reused to execute multiple tasks, so the startup cost is amortised across a batch rather than paid per submission.

Sharing data between tasks and cancelling them

Tasks that need to share data have a recommended pattern, and it is worth understanding why it is recommended rather than merely possible.

A Cache instance stored in a static property that is only initialised within Task::run() is the recommended strategy:

use Amp\Cache\LocalCache;
use Amp\Cancellation;
use Amp\Parallel\Worker\Task;
use Amp\Sync\Channel;

final class ExampleTask implements Task
{
    private static ?LocalCache $cache = null;
    
    public function run(Channel $channel, Cancellation $cancellation): mixed
    {
        $cache = self::$cache ??= new LocalCache();
        $cachedValue = $cache->get('cache-key');
        // Use and modify $cachedValue...
        $cache->set('cache-key', $updatedValue);
        return $updatedValue;
    }
}

The static property is what makes this work across tasks. Each worker process has its own memory, so a cache stored in a static property is shared between the tasks that run in that worker and is not shared between workers. The initialisation inside run() rather than at class load is what keeps the cache alive for the worker's lifetime rather than being rebuilt per task.

There is a caveat for the case where the value is produced by async IO. Because a worker may be executing multiple tasks, the README suggests AtomicCache instead of LocalCache when creating or updating cache values if a task uses async I/O to generate the value. AtomicCache has methods which provide mutual exclusion based on a cache key.

Cancellation is the other mechanism, and it is cooperative rather than forceful. A Cancellation provided to Worker::submit() may be used to request cancellation of the task in the worker, and when cancellation is requested in the parent, the Cancellation provided to Task::run() is cancelled. The task may choose to ignore the request and act accordingly by throwing a CancelledException from run(). And the README is explicit about the edge case: if the cancellation request is ignored, the task may continue and return a value which will be returned to the parent as though cancellation had not been requested.

So cancellation is a signal you check, not a guarantee. A task that ignores it keeps running and still delivers a result, which is a sensible design for a library that cannot safely kill a thread mid-operation.

Worker pools, the global pool and pulling a worker out

The easiest way to use workers is through a worker pool. Pools submit tasks the same way a worker does, but instead of a single worker process the pool uses multiple workers to execute tasks, which allows multiple tasks to be executed simultaneously.

The WorkerPool interface extends Worker, adding methods to get information about the pool or to pull a single Worker instance out of it. That inheritance is what makes the swap from a single worker to a pool a one-line change at the call site, which is a good sign about the API design.

Pulling a worker out of the pool is the feature with a specific use. If a set of tasks should be run within a single worker, use WorkerPool::getWorker() to pull one out, and the worker is automatically returned to the pool when the instance returned is destroyed.

That last detail is the important one. Destruction returning the worker to the pool means the pattern is safe in PHP, where there is no explicit scope-based resource management. The pool reclaims a worker even if the code path that borrowed it exits early or throws, which removes a whole class of bug where a pool slowly drains because a borrowed worker was never given back.

A global worker pool is also available, through the function Amp\Parallel\Worker\workerPool(?WorkerPool $pool = null). Passing an instance sets the global pool to that instance, and invoking the function without an instance returns the current global instance. That getter-and-setter signature is the same pattern PHP uses for things like the error handler, and it lets an application configure the pool once at start-up without threading it through every call site.

The repository has examples/ directories for contexts, processes and shared memory parcels, and a worker example directory, so the intended usage patterns are demonstrated rather than only described.

Limits: serialisation, worker startup and what parallelism costs

The limits are visible in the design and in the requirements.

The first is serialisation. Everything crossing the process boundary is serialised in the parent and unserialised in the worker, and only serialisable data can cross it. That rules out passing open resources, closures and live connections, and it means the cost of a task includes the cost of encoding and decoding its arguments. For a task that ships a large array across the boundary, that cost can exceed the work the task does.

The second is that a task must be autoloadable in the worker. The function you call must be predefined or autoloadable by Composer so it also exists in the worker process or thread. In practice that means your task class lives in your autoloaded source tree rather than in a closure defined inline, and it means the worker has to be able to find and load it.

The third is the resource cost. Processes carry their own memory, and threads share it. The README describes the library as enabling efficient multitasking, optimizing resource use and improving application responsiveness through multiple CPU threads, and the process route is the one that needs no extension at all. Either way you are adding operating system processes or threads to a machine, and the choice of how many is a tuning decision the library leaves to you.

The fourth is the requirement floor. PHP 8.1 or newer for the library at all, and PHP 8.2 or newer in ZTS mode plus ext-parallel for the thread path. An application supporting PHP 8.0 cannot use it, and an application that wants threads without asking users to install a native extension cannot have them.

The fifth is the misuse risk. The README explicitly warns that if you only want to fetch multiple HTTP resources concurrently, the non-blocking HTTP client is better. Parallelising IO-bound work with processes is a common mistake that works, performs acceptably and costs far more than it needs to. This library is for work that genuinely cannot be made non-blocking.

amphp/parallel against ext-parallel and against a plain event loop

Two comparisons make the choice clearer.

The first is against ext-parallel directly. The native extension gives you threads with much lower overhead, and it is the faster option when you need many workers doing short tasks. The difference in approach is that this library works without any extension, so it installs on any PHP 8.1 or newer host with no compilation step and no root on the server, and it uses processes by default. For a shared hosting environment, a container image you do not control, or a deployment pipeline where adding a PECL extension is a negotiation, that is the difference between a feature you can use and a feature you cannot.

The library also builds in the things you would otherwise assemble yourself: the Task interface with a serialisation contract, the Channel for two-way communication, the Cancellation signal, the pool with automatic return of borrowed workers, and the cache pattern for sharing state within a worker. The native extension gives you the primitive; this gives you the arrangement.

The second comparison is against staying with a plain event loop. If your workload is IO-bound, an async client does everything a worker pool does for these tasks, in one process, without the serialisation cost, without the memory overhead and without the worker startup cost. The README says as much in its own example. That comparison is the one to make first, because the answer is often that you do not need parallelism at all, you need a non-blocking API for the thing you are doing.

So the case for this library is narrow and real: work that is CPU-bound or that calls into a blocking library with no async equivalent, in a PHP application that is otherwise event-driven, on a host where you cannot install extensions. That is a genuine gap, and this library fills it without asking anything of the user.

Editorial conclusion

Adopt amphp/parallel when a PHP application has genuinely CPU-bound or blocking work that must not stall an event loop, and you want the parallelism without asking users to install a native extension. Do not adopt it for concurrent HTTP requests, because a non-blocking HTTP client solves that more cheaply. Verify first that your PHP is 8.1 or newer, that anything you pass into a task is serialisable, and that the function you call is autoloadable in the worker, since the worker is a separate process or thread that must find it on its own.

Frequently asked questions

Official sources

1. amphp/parallel on GitHub
2. Issues
3. License: MIT
4. README
5. Releases