php-spider, a PHP crawler built from swappable components

The parts are the design

php-spider is a configurable and extensible PHP web spider, and the feature list reads as a list of components rather than a list of features. Traversal is one component with two algorithms, breadth-first and depth-first, and the limits are separate controls, so a crawl depth limit, a queue size limit and a maximum download limit are independent. URI discovery is another, and it accepts custom discovery logic based on XPath, CSS selectors or plain PHP. Filtering is split in two, prefetch filters that look at the URI and postfetch filters that look at the resource content, and the built-in set includes robots.txt handling and domain limiting, with custom filters supported in both positions. Requests are handled by their own component, persistence by another, with memory and file handlers provided and custom ones allowed, and the crawl dispatches events so a developer can add behaviour without subclassing the spider. Statistics collection for reporting is built in, and there is a politeness policy. The audience is a PHP developer who wants to assemble these parts rather than write a crawler from scratch.

The spider does not run JavaScript

One line in the README settles the scope of the whole project, and it is worth reading before anything else. This Spider does not support Javascript. That is a hard boundary, and it decides which sites the library is for. A crawler built this way fetches HTML and reads the links out of it, with XPath and CSS selectors as the discovery mechanism, so it handles server-rendered pages and anything whose links are present in the markup. It will not execute a page's JavaScript, will not wait for a client-side router to populate the DOM, and will not see links that only exist after hydration. For a large share of the web, including most documentation sites, blogs, forums and any server-rendered application, that is entirely workable, and the same limitation is what keeps the library small enough to read in an afternoon. It also means the discovery layer has to be written against the markup you actually receive, which is where the XPath and CSS discoverer options earn their place.

A first crawl in five steps

The simple example in the README is short and shows the whole flow. You create the spider with a starting URL, and then you must add a discoverer, because without one the spider does nothing.

$spider = new Spider('http://www.dmoz.org');
$spider->addDiscoverer(new XPathExpressionDiscoverer("//div[@id='catalogs']//a"));
$spider->setMaxDepth(1);
$spider->setMaxQueueSize(10);

Then you attach a listener to the queue manager and to the spider, so that the same handler sees both the queue events and the crawl events, and you run the crawl.

$statsHandler = new StatsHandler();
$spider->getQueueManager()->getDispatcher()->addSubscriber($statsHandler);
$spider->getDispatcher()->addSubscriber($statsHandler);

$spider->crawl();

Afterwards the statistics handler reports what happened, with counts of enqueued, skipped, failed and persisted items, and the downloaded resources are reachable through the downloader's persistence handler, where the example reads the title of each one with an XPath filter.

Fluent configuration for the common settings

For the settings most crawls need, the README shows a fluent style that keeps the configuration in one place, and the method names map directly onto the components.

$spider
    ->setDownloadLimit(50)
    ->setTraversalAlgorithm(QueueManagerInterface::ALGORITHM_BREADTH_FIRST)
    ->setMaxDepth(2)
    ->setMaxQueueSize(500)
    ->setPersistenceHandler(new FileSerializedResourcePersistenceHandler(__DIR__.'/results'))
    ->addDiscoverer(new XPathExpressionDiscoverer('//a'))
    ->addFilter(new AllowedHostsFilter(['example.com']));

$spider->enablePolitenessPolicy(100);

The politeness policy takes a delay between requests to the same domain, and it is optional. The filter used here, AllowedHostsFilter, is a prefetch filter that keeps the crawl inside a set of hosts, which is the guard rail a crawler needs before it walks off into the rest of the internet. The traversal algorithm is chosen by constant rather than by string, which is a small sign of the care taken with the public API.

Caching makes a crawl incremental

The cache support is the feature that turns a one-off crawl into a repeatable job, and it is implemented as a filter plus the file persistence handler. The idea is to keep what was already downloaded and skip it next time, using a fixed spider identifier so that the cache is shared across runs.

$spiderId = 'my-spider-cache';
$spider = new Spider('http://example.com', null, null, null, $spiderId);

$resultsPath = __DIR__ . '/cache';
$spider->getDownloader()->setPersistenceHandler(
    new FileSerializedResourcePersistenceHandler($resultsPath)
);

$maxAgeSeconds = 3600;
$cacheFilter = new CachedResourceFilter($resultsPath, $spiderId, $maxAgeSeconds);
$spider->getDiscovererSet()->addFilter($cacheFilter);

The maximum age is the knob that decides how stale a cached resource may be, with a comment in the example noting that setting it to zero always uses the cache. The filter is added to the discoverer set rather than to the spider directly, which is a detail worth noticing if you write your own filters, since the two positions filter different things.

The default stops the crawl on the first broken response

There is a default in the library that will surprise anyone who points it at a real site, and the README calls it out with a note. By default, the spider stops processing when it encounters a 4XX or 5XX error response. For a link checker that is arguably the right behaviour, since a broken link is the finding, but for a crawler that is meant to keep going it means one missing page ends the run. The documented remedy is a custom request handler, and the link checker example shows it configuring the default Guzzle request handler so it does not fail on 4XX and 5XX responses. That is worth understanding as a design point rather than a workaround, because the failure mode of a crawler that dies on an error response is a crawl that silently covers a fraction of the site, and the fix is a request handler rather than a configuration flag.

The package, the standards and the examples

Installation is a single composer command, and the package is on Packagist under the vendor prefix vdb.

$ composer require vdb/php-spider

The repository is a conventional PHP library layout, with src/ for the code, tests/ and a phpunit.xml.dist, docs/ with the per filter documentation, bin/, and an example/ directory that is more useful than the README's own snippet. The examples cover the simple crawl, a complex one with logging, caching and filters, basic authentication, a link checker, a health check, persistent request parameters and a complex bootstrap, plus a lib/ directory and a cache/ folder. The contribution rules are explicit, since a pull request is expected to follow the PSR-0, PSR-1 and PSR-2 coding standards, and the tooling in the tree enforces that, with a .php-cs-fixer configuration, phpmd.xml and phpmd-tests.xml for mess detection, a .phan/ directory for static analysis and .run/ for IDE run configurations. The licence is MIT, the repository is not archived, the last push was on 2026-08-04, and the project publishes no GitHub releases, so a consumer pins a Packagist version rather than a tag.

Editorial conclusion

php-spider fits a PHP team that needs a crawler for a server-rendered site, whether that is a link checker, an archive job or a feed of pages to index, because the component boundaries let you replace the request handler or the persistence layer without rewriting the traversal. It does not fit a crawler of a modern single page application, since the project states plainly that it does not support JavaScript, and it does not fit a project that wants a distributed crawler. Verify three things before you build on it: that the default behaviour of stopping on 4XX and 5XX responses matches what you want, because a crawler that stops at the first broken link is not a crawler unless you change the request handler, which the link checker example shows how to do, that the sites you target serve their links in the HTML, and that the politeness policy delay is set to something the target site will tolerate.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. mvdbos/php-spider on GitHub
4. README