Spidermon: monitors and notifications for Scrapy spiders

A spider that reports on itself

Spidermon is described in the README as a Python 3.10 or later Scrapy extension, and the package metadata calls it a framework to build monitors for Scrapy spiders. The distinction between a tool and a framework is worth noticing, because it is the difference between a set of built-in alerts and a mechanism for defining your own.

The stated job is narrow and real. The package provides useful tools for data validation, stats monitoring and notification messages, and the README frames the value plainly: you leave the monitoring task to Spidermon and just check the reports and notifications.

That framing targets a specific failure mode in web scraping. A spider that returns HTTP 200 and writes rows to a database looks healthy to every generic check, while its output is quietly wrong, or it stopped matching anything, or it is being served a captcha page that parses without error. Those are failures of the crawl's data, and they are invisible to infrastructure monitoring. Spidermon exists to watch the spider's own behaviour and its own output rather than the machine it runs on.

The three areas map onto the three ways a crawl goes wrong. Data validation catches items that are structurally wrong, the wrong shape or the wrong values, before they reach whatever consumes them. Stats monitoring catches the crawl-level problems, an item count that collapsed, a spider that did not run, a request pattern that changed. Notifications are the delivery mechanism for both, so that somebody finds out.

The audience is therefore a team running Scrapy as a service rather than as a one-off script, and the project topics reflect that: crawling, scraping, spiders, monitoring, monitoring-tool and testing, alongside scrapinghub, which points at the company that maintains it, now Zyte.

The topics also include hacktoberfest, which is a fair signal about where contributions have come from.

Installing: a two dependency core and one large extra

The quick install is one command:

pip install spidermon

and the README points to the install section of the documentation at spidermon.readthedocs.io for the details, with the documentation also available in the docs directory of the repository.

The interesting part is what the package metadata says about the dependencies, because it splits the library into two very different shapes.

The base install has exactly two dependencies:

dependencies = [
    "jsonschema[format]>=4.21.0",
    "python-slugify>=5.0.0",
]

jsonschema with the format extras is the validation engine, and it is the reason the base install is not trivial: a validation rule in Spidermon is a JSON Schema document, so the format checker lets you write constraints on dates, times, email addresses and similar values rather than only on types. The format extra is what makes that possible, and it is a real design decision that your validation rules are schemas rather than Python code. The second dependency, python-slugify, is for turning item values into identifiers, which is how a monitor names the thing it is reporting on, and how a notification subject stays readable.

Everything else lives in an optional extra called monitoring:

monitoring = [
    "boto3>=1.10.0",
    "itemadapter>=0.1.0",
    "jinja2>=3.0.0",
    "premailer>=2.7.0",
    "requests>=2.19.0",
    "scrapinghub>=2.4.0",
    "scrapinghub-entrypoint-scrapy>=0.11.2",
    "scrapy>=2.7.0",
    "sentry_sdk",
    "slack_sdk>=3.9.0",
]

That single list tells you the integration surface without reading a line of the documentation. Slack is there through slack_sdk, error reporting through sentry_sdk, AWS through boto3, and Scrapy Cloud through the scrapinghub client and its Scrapy entrypoint package. Email is built from jinja2 for the template and premailer for inlining the CSS so the HTML looks right in a mail client, and requests for the sending itself. itemadapter is there so that a monitor can work with Scrapy items regardless of whether the project uses the older Item API or the newer dataclass-style item, and scrapy itself is in the extra rather than the base, which tells you that Spidermon expects to be installed into an environment that already has Scrapy.

Three jobs: validation, stats and notifications

The README summarises the feature set in one sentence, and unpacking the three nouns explains how the library is meant to be used.

Data validation is the JSON Schema part. The idea is that you attach a schema to an item, or to a field, and the extension checks the scraped values against it. Because the schemas are JSON Schema rather than hand-written Python, they are declarative, they can be shared between the spider and other services, and the format extra means that the standard format validators are available. The trade-off is that JSON Schema expresses constraints on structure and value rather than relationships between items, so a rule that needs to compare one field with another does not fit the model.

Stats monitoring is the crawl-level half. Scrapy already collects statistics about a run, items scraped, requests sent, errors, start and finish times, and a monitor can act on a change in those numbers. The design question is the threshold, and Spidermon leaves it to you: you define the condition, the framework does the checking. That is the right division of labour, because the useful threshold is a property of your site and your business, not of the tool.

Notifications are the third piece and the one that decides whether any of this is useful. A monitor that detects a problem and writes to a log is a dashboard nobody reads, so the delivery integrations matter, and the extra list shows the intended channels: Slack, Sentry, email, and Scrapy Cloud. The email path is the most complete of them, with a templating engine and a CSS inliner, which is a hint that the notification bodies are meant to be HTML documents rather than plain text.

The README's phrasing about leaving the monitoring task to Spidermon and just checking the reports and notifications describes the intended division of labour between the system and the human, and it is the design goal of the package in one sentence.

Documentation, the tutorial example, and the version gap

The documentation is hosted at spidermon.readthedocs.io and also lives in the docs directory in the repository, with the release notes rendered as a changelog page, so the docs are the primary interface of this project rather than the README. The README itself is short by design: an overview paragraph, an install line, and a link.

There is also an examples directory containing a tutorial, which is the quickest way to see the whole shape of the thing, since a monitoring framework is hard to grasp from an API list.

The repository is a well equipped Python project. There is a pyproject.toml using hatchling as the build backend, a tox.ini for the test matrix, a .pre-commit-config.yaml, a .readthedocs.yaml for the documentation build, a CHANGES.rst, a CONTRIBUTING.rst, and the spidermon package directory itself with a tests directory alongside it. The badges tell you the quality story: a CI workflow badge, a Codecov coverage report, the PyPI version and licence badges, the supported Python versions, and a black code style badge.

One thing needs flagging before you pin a version. The package metadata says the version is 1.27.0, while the release history the repository reports shows 1.24.0 from 2025-04-11, 1.23.0 from 2024-10-31 and 1.22.0 from 2024-05-08. The last push to the repository was on 2026-10-14, so the tree is being worked on and the working version is ahead of the newest tag. The bumpversion configuration in the pyproject explains part of it, with commit and tag both enabled and the tag name taken from the new version, so tags are cut as part of the release process rather than separately.

The practical consequence is that pip install spidermon gives you the newest published release, which may be older than what you see in the repository, and that the changelog is the place to find out what actually changed between the two.

Maintenance, licence and where it sits against the alternatives

The licence is BSD-3-Clause, with the LICENSE file at the root of the repository, and the package metadata carries the same identifier along with a License :: OSI Approved :: BSD License classifier. For a package you install into a production application, that is the most permissive of the common choices and carries no obligation back to the project.

The maintenance picture is good in the way that matters for a dependency. The last push was on 2026-10-14, the package is classified as Development Status :: 5 - Production/Stable, the supported Python versions are 3.10 through 3.14, and the project is maintained by Zyte, the company behind Scrapy Cloud and formerly Scrapinghub, at [email protected]. The relationship to Scrapy itself is the interesting part: this is a first party extension for a framework the same organisation maintains, which is a different risk profile from a third party extension with its own bus factor.

The alternative approaches to the same problem are worth spelling out, because they differ in what they can see.

The first is writing the checks yourself inside the spider, using Scrapy's signals and a validation library. That gives you full control and no new dependency, and it is what a team does when the checks are specific to one crawl. The difference is that Spidermon centralises the mechanism: the same monitor definition can apply to every spider in a project, and the notification delivery is a solved problem rather than a per-project one.

The second is using the data platform's own validation, validating on the warehouse side after the crawl has landed. That catches malformed data before it is consumed, and it is where a lot of teams end up. The difference is timing and scope. A schema check in the warehouse sees the output but not the crawl, so it cannot tell you that a spider stopped matching anything, whereas a monitor attached to the spider sees both, and it can block a bad item before it is written at all.

The third is plain infrastructure monitoring plus a dashboard. That catches the spider dying, and it is much cheaper to run. It cannot catch a spider that is alive and wrong, which is the failure mode Spidermon exists for.

So the honest framing is that Spidermon sits between the platform's validation and the infrastructure's health checks, and it is worth adopting when your crawls feed something that breaks quietly rather than loudly.

Editorial conclusion

Adopt Spidermon if you run Scrapy spiders in production and have no reliable way to know whether a spider is producing bad data or has stopped producing anything, since the validation, monitoring and notification pieces are the ones every team reinvents badly. Do not adopt it if you need a general monitoring system, since it reports on spider behaviour rather than on host or service health, and do not expect the published package version to match the working tree, because pyproject.toml says 1.27.0 while the newest GitHub release is 1.24.0 from 2025-04-11. Verify first which Python version your Scrapy project runs on, since the package requires 3.10 or later and classifies up to 3.14, which notification channels you actually need, since Slack, Sentry, AWS and Scrapy Cloud are all in the monitoring extra rather than the base install, and what the jsonschema dependency costs you in terms of how you express your validation rules.

Frequently asked questions

Official sources

1. License: BSD-3-Clause
2. Project website
3. README
4. Releases
5. scrapinghub/spidermon on GitHub