spider-admin-pro: a web console for running Scrapy spiders on Scrapyd

A management console that sits on top of Scrapyd, not beside it

Spider Admin Pro is the upgraded version of Spider Admin, described in the README as a visual management platform for Scrapy crawlers that depends on Scrapyd. The architecture is simpler than the feature list suggests: Scrapyd does the deploying and running of spiders, and this project provides the interface, the schedule and the record of what happened.

That ordering matters for evaluation. Scrapyd exposes an HTTP API for deploying a project, starting a spider and reading its logs. This project is the layer that turns that API into something a person can use, with a project list, a schedule, a dashboard and a log view. The technical stack table in the README makes the same point: the scrapyd interface is handled by the scrapyd-api library, the task scheduling by APScheduler, and the HTTP interface by Flask, with peewee as the ORM and PyJWT for tokens.

The front end is a separate story, and one an adopter has to notice. The stack is Vue 2 with Element-UI for styling, ECharts for the dashboard charts and axios for requests, and the project structure section states plainly that the back end is the public repository while the Vue front end project spider-admin-pro-web is a private repository, obtainable by sending a GitHub username to a QQ group, with one option being to sponsor the project first. The repository root also has an http directory and a web directory for static files, so the served interface and the built front end are separate concerns.

The audience is therefore a Chinese-language data team or a solo crawler operator, since the documentation, the group chats and the support channels are all Chinese, and the licence is MIT with a LICENSE file at the root.

Three installation routes, and the dependency that has to exist first

The README is explicit that before running the project you must make sure the Scrapyd service is already started. Everything else follows from that, including the requirement to install the Scrapy stack itself, which is listed as optional but is in practice not optional if you want to deploy spiders.

The first installation route is from PyPI into a virtual environment, and the commands are straightforward:

$ python3 -m venv venv
$ source venv/bin/activate
$ pip3 install spider-admin-pro

Then you serve it, and the README gives a different server for each platform. On Linux and macOS it is gunicorn, and on Windows the README says to install waitress and use it in place of gunicorn, because waitress-serve binds to 127.0.0.1 on port 8000.

The second route is from source, which clones the repository, installs the requirements and uses the Makefile:

$ git clone https://github.com/mouday/spider-admin-pro.git

$ cd spider-admin-pro

$ pip3 install -r requirements.txt 

$ make dev

The third route is Docker, and the compose-style command mounts a config file into the container and publishes port 8000:

docker run -e TZ=Asia/Shanghai -p 8000:8000 -v ./config.yml:/app/config.yml mouday/spider-admin-pro

The Makefile shows what dev and pro actually do. The dev target runs gunicorn bound to 127.0.0.1 on port 5001 with reload, and the pro target runs gunicorn bound to 127.0.0.1 on port 8000 without reload. The Dockerfile does the same thing in Alpine, on python:3.7.15-alpine3.16, and starts gunicorn bound to 0.0.0.0 on port 8000.

Note the version floor. The README says the project is developed against Python 3.7.0 and recommends 3.7.0 or newer, and the Docker base image is pinned to 3.7.15. This is a Python version that has been end of life for years, which is a maintenance fact you will have to plan around rather than discover.

Configuration is a YAML file with credentials in it

The configuration file is named config.yml, and the README explains that you create it in the directory you run the start command from, because it is read automatically from there.

The documented keys are few, and the example is short:

USERNAME: admin
PASSWORD: "123456"

COMPRESS_STATIC: True

The username and password are the login for the console, and the README attaches a strong recommendation to change the password and secret key items. That recommendation is not decorative, because the console is what triggers deployments on your machines.

The rest of the example is commented out, and the comments describe the v2 era options: a SCRAPYD_SERVER address that must not end with a slash, and SCRAPYD_USERNAME and SCRAPYD_PASSWORD for basic auth support, added since 2.0.8. The comment marks these as not supported in v3.0, which is consistent with the bigger change in v3, which is that it supports multiple Scrapyd servers where v2 supported only one. The README states plainly that v3.0 and v2.0 are not compatible and cannot be migrated smoothly, and the older documentation is kept in separate README-v1.md and README-v2.md files.

There is also an optional switch for compressing the front end static files to improve page load speed, on by default, which is the COMPRESS_STATIC setting, and the requirements list includes flask-compress and whitenoise to back it.

The README's own security advice about Scrapyd is the important part

There is a section of the README that is worth more to an adopter than any feature list, and it is a warning rather than a description.

It says that Scrapyd must not be exposed directly to the public internet, and gives two reasons. First, other people deploying through it can deploy code onto your machine, and if it runs as the root user, they can do other things on your machine while they are there. Second, configuration file information shows up in the run logs, which is a risk of information disclosure.

Both points are serious, and both are properties of Scrapyd rather than of this project. The mitigation is to keep Scrapyd on a private network and reach it through the admin console or a tunnel, and to run the Scrapyd service as an unprivileged user. The admin console itself then becomes the exposed surface, which is why the README's instruction to change the default password and secret matters.

The second point about logs is worth restating because it is easy to miss: the deploy API prints configuration values into the log stream. If your Scrapy project's settings hold credentials, they may end up in the log files that this very console displays.

A separate extension is offered for collecting run logs, the scrapy-util project, which the README says helps you gather statistics about program runs. The dependency list also includes ip-area, ua-parser and user-agents, which suggests the dashboard reports on the geographic origin and user agents of the traffic the spiders encounter.

What the project structure says about the code

The README gives a directory listing for the main project, and it is short enough to be informative.

The listing names run.py as the program entry point, then an api directory for the controller layer, a service directory, a model directory, an exceptions directory, a utils directory for utility classes, and a web directory for static web pages.

So the back end is a conventional layered Flask application, with the model layer on peewee, and a web directory holding the static front end that the separate private Vue project builds into.

The repository root also lists a test directory, an http directory, a doc directory, a version-cli directory and a Dockerfile, plus two requirements files, one named requirements.py3.6.txt and one requirements.txt, which suggests the project carries a pinned dependency set for the older Python line alongside the current one.

The requirements list is a reasonable size for what it does: Flask with Flask-Cors and Werkzeug, APScheduler, peewee and SQLAlchemy, PyJWT, python-dotenv and environs for configuration, psutil for system information, requests with session-request for outbound calls, scrapyd-api for the Scrapyd interface, ua-parser with user-agents and ip-area, gunicorn as the server, and flask-compress with whitenoise for static file handling. The setup.py describes the package as a spider admin based on vue, scrapyd api and APScheduler, and it reads its version from a version.py file found with a glob, which is a slightly unusual but workable approach for keeping the version in one place.

Where spider-admin-pro is the wrong tool

The first wrong-tool case is a team that needs the front end source. The README is explicit that the Vue project is private and that access requires sending a GitHub username to a QQ group, with sponsoring as one of the routes. A front end you cannot read, cannot fork and cannot contribute to is a real constraint for an organisation with a security review, and it means your customisation options stop at what the API exposes.

The second is an environment on a modern Python. The project is developed against Python 3.7.0, the Docker image is 3.7.15, and a separate requirements file exists for 3.6. If your organisation has standardised on a current interpreter, running this means an older base image and a dependency set that predates several security fixes in the Flask and Werkzeug lines.

The third is a team that has already standardised on a scheduler and a job UI. This project bundles APScheduler and its own dashboard because Scrapyd needed one. If you already run Airflow, Prefect, a Kubernetes CronJob or a plain systemd timer, adding a second scheduling authority is a design you have to justify rather than a convenience.

The fourth is a crawler that runs on a laptop for one person. The console, the user accounts, the tokens and the project management all exist to coordinate several people over shared machines. A single developer with two spiders does not need a web interface for them.

The fifth is anyone expecting the v2 behaviour. The README says plainly that v3.0 and v2.0 are not compatible and cannot be migrated smoothly, and that v3.0 supports multiple Scrapyd where v2.0 supported one. The older documentation still exists in the repository, so you can read both, but you are choosing a line, not upgrading within one.

Editorial conclusion

spider-admin-pro fits a team running several Scrapy crawlers on shared Scrapyd machines who want a browser-based view of projects, run logs and scheduled tasks. It is the wrong choice for a project that needs the Vue front end source, since that repository is not public, and it is the wrong choice if your team needs multi-Scrapyd support on the v2 line, since v3 and v2 are documented as incompatible. Before adopting it, read the security note in the README about not exposing Scrapyd to the public internet, and decide whether the private front end is acceptable before you commit to the project.

Frequently asked questions

Official sources

1. License: MIT
2. mouday/spider-admin-pro on GitHub
3. Project website
4. README
5. Releases