Beanbun, a PHP crawler framework built on Workerman and Guzzle

A crawler with a very small surface

The library is unusually small in concept. You create a Beanbun object, set its seed property to an array of URLs, assign a callback that runs after each page is downloaded, and call start. That is the whole quick start. There is no configuration file, no plugin registration step and no scheduler. The seed array is the entry point of the crawl, and the callback is where you do the actual work, typically writing the page to disk or extracting what you came for. The example in the documentation writes the downloaded page into a file named after the MD5 of the URL, which is a reasonable pattern for a quick scrape. The point of the design is that the framework handles the link following and the queueing while you handle the meaning of the content, and the extension points are there for the cases where you want to intervene.

<?php
use Beanbun\Beanbun;
$beanbun = new Beanbun;
$beanbun->seed = [
	'http://www.950d.com/',
	'http://www.950d.com/list-1.html',
	'http://www.950d.com/list-2.html',
];
$beanbun->afterDownloadPage = function($beanbun) {
	file_put_contents(__DIR__ . '/' . md5($beanbun->url), $beanbun->page);
};
$beanbun->start();

Two ways to run: a script or a Workerman daemon

The framework supports two execution modes, and the difference matters operationally. In normal mode you run a PHP script from the command line, which is what the quick start does, and it is the mode to use for a one-off crawl or a cron-driven job. In daemon mode the framework is built on Workerman. The documentation is specific about the constraint: daemon mode is only supported on Linux servers. The practical reason is that process management behaves differently on Windows, and a crawler framework that forks a pool of workers is a natural fit for that model. The result is that a long-running crawl in daemon mode can keep its connections and its queue warm, while normal mode is the simpler choice when you only need the process to live for the length of one traversal.

$ composer require kiddyu/beanbun

The queue is pluggable, and that is where distribution comes from

Beanbun supports several queue implementations, with memory and Redis named explicitly. The memory queue is the obvious default for a single process, and the Redis queue is what lets a crawl span more than one machine, which is how the framework's distributed mode works: every worker reads from and writes to the same shared queue rather than keeping its own list. The framework's stated extension mechanism makes the queue replaceable, so a custom backend is a plugin away. That design has a consequence worth naming. A Redis-backed queue makes a distributed crawl possible, but it also means the crawl state now lives outside the process, so a worker that dies mid-traversal does not lose its place, and you can add workers to a running crawl. The alternative, a custom queue backed by something else, gives you the same property with different operational characteristics, and the documentation points to plugins as the way to build it.

$ php start.php

BFS, DFS and a URI filter to keep the crawl in bounds

Three features decide what a crawl actually does, and they are all in the feature list. The traversal order is configurable between breadth first and depth first, which is a real design decision rather than a detail: a breadth first crawl reaches the shallow pages of a site quickly and is the right choice for a site with a predictable structure, while depth first goes as deep as it can down one branch, which suits a tree-shaped site or a target set of deep pages. The URI filter is the other half of keeping a crawl sane, letting you exclude URLs by pattern so a crawler does not wander into infinite parameter spaces or into areas of a site you do not want. Between them, they are the difference between a crawler that finds what you asked for and one that spends its budget on pagination. The feature list also notes that a crawl is divided into multiple steps, and that each step supports custom actions, such as adding a proxy or changing the user agent.

Plugins, PSR-4 and the example directory

The framework follows the PSR-4 standard for autoloading, which means it drops into a composer-managed PHP project without special wiring, and the package is installed with a single composer require. Extension is the stated design goal: the feature list describes a flexible extension mechanism for making plugins for the framework, with custom queues and custom crawling methods named as examples, and there is at least one published plugin for data extraction called beanbun-parser. The repository layout backs this up, with a src/ directory holding the framework, an examples/ directory, a docs/ directory with the Chinese documentation under a chs path, and a composer.json at the root. The examples directory is worth a look before you write anything, because it contains a phantomjs.php example, which suggests a headless browser path for pages that need JavaScript, and a zhihu_user.php example, which shows the framework pointed at a real site with a specific user. The plugin list names a data extraction plugin, beanbun-parser, maintained in a separate repository.

The maintenance picture, read carefully

The version record needs care. The three visible releases are 1.0.4 from 2017-06-11, 1.0.3 from 2017-04-27 and 1.0.2 from 2017-04-27, and the version badge in the README reads 1.0.4, which matches the last tag. Yet the last push to the repository was on 2026-05-26 and the repository is not archived, so the code has been touched long after the last release. That pattern usually means the maintenance activity is small, sporadic fixes rather than a release train, and it means the version number in your composer.lock tells you almost nothing about which commits you have. The licence is MIT. The description calls the framework simple and extensible with good openness and high extensibility, and the documentation in the repository is in Chinese, with the reference at a docs/chs path. For a reader deciding whether to invest in it, the honest summary is that this is a mature small framework from the mid-2010s whose architecture still fits, sitting on libraries that have moved on since.

Editorial conclusion

Adopt Beanbun if you already write PHP and want a crawler that fits into an existing application rather than a separate Python stack, and skip it if you need an actively versioned crawler, since the last tagged release is 1.0.4 from 2017 and the framework predates the modern PHP ecosystem conventions. Verify three things before you build on it: that the Workerman daemon mode runs on Linux, since the documentation says the daemon mode is only supported on Linux servers; that the queue backend you need is available, since memory and Redis are the named options and the extension mechanism is yours to fill; and that you read the Chinese documentation in the docs directory, since the visible README is the shorter surface and the real reference lives there.

Frequently asked questions

Official sources

1. Issues
2. kiddyuchina/Beanbun on GitHub
3. License: MIT
4. README
5. Releases