NewsCrawl: a Scrapy cluster for public opinion news

What NewsCrawl is above the spider layer

The repository presents itself as a public opinion information acquisition and visualisation platform, and the name in the README heading is NewsSpider. The useful way to describe it is as an operations frame around Scrapy rather than a set of spiders. The extraction side is delegated to GNE, the General News Extractor module, which is used to pull the body text, title, author, publication time, image addresses and the tag source of the body out of more than 400 news sites, according to the README. Everything else is the machinery for running that at scale: python, Scrapy, scrapyd, scrapydweb as the open source management platform, MySQL and Redis. On top of that the README lists a set of capabilities that are about running many crawlers rather than about crawling well: one-click operation of any number of crawlers, scheduled crawler tasks, batch deletion of crawlers, crawler monitoring and visualisation, and cluster allocation strategies for distributing spiders over nodes. The audience is a team that already knows what a public opinion monitoring job looks like and needs the plumbing rather than the parsing.

start.sh quick and start.sh full

The root of the repository ships a single start script, and the README points to docs/deployment.md as the detailed deployment document before showing the script usage. The two modes differ in what you give up. The quick mode is the local path: SQLite plus scrapyd and scrapydweb, and the README states that if the machine has no MySQL or Redis you should prefer this mode, because the script substitutes SQLite for the database service. The full mode is the Docker Compose deployment with MySQL, Redis, scrapyd and scrapydweb together.

./start.sh quick
./start.sh full
./start.sh status
./start.sh stop

Two operational notes from the README. The project expects to live in a fixed directory on the server, /home/spider_workplace/, and the deeper manual instructions repeat that path in the paths they tell you to edit. And the whole design assumes you are comfortable with Docker Compose services, because the manual path below is a series of docker-compose up -d calls in specific directories. The interesting design decision is that the quick path exists at all. A framework that expects MySQL and Redis for a local trial gets in its own way, and the SQLite substitution removes that obstacle without changing the spider code.

The manual route when a Compose file is not enough

The README also documents the deployment by hand, in four steps, and it is worth reading because it shows which piece does what. Docker and Docker Compose come first, with both an online install using yum on CentOS and an offline route that points to an external article, and the Compose binary is fetched from the GitHub releases page and given execute permission.

yum install -y yum-utils
yum install docker
systemctl restart docker
docker run hello-world

Then the database, which is a directory of Compose files under docker_yml/database that you move into the data directory and start.

mv ./docker_yml/database  /data/ && cd /data/database
docker-compose up -d
docker-compose ps

Then the crawler platform itself, from the docker_yml/news_crawl directory, where the same up and ps pair is used, and the README notes that a failed start usually means the scrapydweb configuration or the cluster entries need attention.

docker-compose up -d
docker-compose ps

Finally the configuration step, which is manual by design: you edit scrapydweb_settings_v10.py in the TLNewsSpider tree with vim, then recreate the containers and read the logs.

docker-compose down
docker-compose up -d
docker-compose logs

auto_manage_spiders.py and the two ways of splitting a cluster

Scheduling is handled by a script, auto_manage_spiders.py, which works against the public interface of scrapydweb so that a whole set of spiders can be managed without clicking through the web page one at a time. The usage output shows the option set.

usage: auto_manage_spiders.py [-h] [-host HOST] [-n {1,2}] [-t {0,1,2}] [-dp]
                              [-bsd] [-bst] [-bdt] [-bdj] [-ls]

The two things to understand here are the node and time options. The node option selects which cluster node index is the current available one, and the time option chooses the scheduled task type in hours, with zero meaning no schedule. The cluster split has two strategies defined in the code, and the README names them: group_spiders_by_chars divides the spiders evenly by their first letter, and group_spiders_by_length divides them by the number of spiders, and the README marks the second one as the one currently in use. Running a node's spiders, adding them to the schedule and deploying a new spider version are the everyday operations.

python auto_manage_spiders.py -n 1
python auto_manage_spiders.py -n 2 -t 1
python auto_manage_spiders.py -dp

The README's own notes on using the script are operational rather than promotional, and they are the part worth internalising. The host option defaults to the current main node and is required, and the host list must contain all of the cluster nodes, also required. The cron list is the hourly schedule and can be edited if you need another cadence, and that one is optional. The script can be run from anywhere as long as the network is reachable, otherwise you run it inside the container.

Middlewares, pipelines, and the limits you can put on a crawl

The crawler side is configurable in the usual Scrapy way, and the README lists what you can plug in. The middlewares include a Redis deduplication middleware, a request retry middleware, and a proxy middleware, plus a placeholder for pushing crawler detail notifications to third party mini programs such as WeChat public accounts or Feishu, which the README marks as still to be developed. The data pipelines cover preprocessing, filtering and cleaning, writing to the database, and writing into Redis for the same deduplication purpose. Beyond that, the project pushes data to Kafka, and a crawler can be limited by time range and by the number of pages turned. Those two limits are the most practically important knobs in the whole system for anyone watching traffic patterns, because they are what bounds how much a single run requests from a given site. The rest of the Scrapy feature set is available as usual. Taken together, the shape is a platform where the spider is a small part and the scheduling, deduplication and storage are the product.

Where the platform stops being a good fit

Three things will decide this project for you, and none of them are the crawling. The first is licensing. The repository's licence field is listed as unknown, and the top level entries do not include a licence file, so there is no clear statement of permission to use, modify or redistribute the code. For an internal tool that may not matter to you, but it is a real obstacle for anything shipped. The second is release hygiene. The project has no GitHub releases at all, so there is no version to pin, no changelog to read and no upgrade path; the last push was on 2026-05-23, so the code you clone is whatever is on the default branch that day. The third is operational coupling. The deployment expects a fixed server directory, a manually edited scrapydweb settings file, and a host list that you maintain by hand, and the two node indices in the script usage show that the split logic as written assumes a small cluster rather than a fleet. Add to that the fact that the 400 plus supported sites figure and the capacity claim in the README are the author's own statements rather than anything independently checkable, and the honest description of NewsCrawl is a well built internal tool with an open licence question and a support model that is one maintainer wide.

A cluster of Scrapy nodes versus one Scrapy process

The alternative worth naming is the plain one: run Scrapy directly on one machine, with the scheduler you write yourself, and skip scrapyd, scrapydweb and the cluster split entirely. The difference in approach is where the scheduling intelligence lives. In NewsCrawl it lives in a management platform that several people can share: a web interface to see what is running, a script that fans spiders out over nodes, cron style schedules per node, and a defined split policy so two nodes do not crawl the same sites. That is the right model when the site list is long and the crawl window is narrow, because the parallelism is the point and the same site should not be hit twice in one pass. The plain alternative is right when the site list is short and the schedule is simple, and it wins on everything else: no MySQL, no Redis, no scrapydweb settings file, no Compose files, and one process to reason about. The deciding factor is usually not the spiders but the number of them. If a handful of sources are enough, the framework is overhead, and if you are running hundreds on a schedule, the batching, the deduplication middleware and the node split are doing real work that you would otherwise write and maintain.

Editorial conclusion

NewsCrawl fits an operations team that has to run many news site crawlers on schedule across several machines and wants the scheduling, deduplication and storage wired together already. It does not fit a commercial deployment as it stands, because the repository has no licence file that GitHub could classify, so there is no clear permission to use or redistribute the code, and it does not fit a team that needs documented, versioned releases, since the project has none. Before you build on it, check the licence situation in the repository yourself, confirm that a recent start.sh quick run works in your environment, and read the extraction coverage note, since the claim of 400 plus supported sites is the author's own and site structures change.

Frequently asked questions

Official sources

1. casual-silva/NewsCrawl on GitHub
2. Issues
3. README