Gather Platform: a web console wrapped around the Webmagic core

The console is the product, the crawler engine is inherited

Gather Platform is described as a data collection and search platform built on the Webmagic core, with a web interface for configuring collection tasks and managing them. The split matters. Crawling itself, the part that fetches pages, walks selectors and runs a downloader, comes from Webmagic. What this repository adds is a layer on top: a task configuration interface, a task management screen, storage for what has been collected, and a set of post-processing tools. That arrangement is the whole appeal for a team that would rather configure than code. The project states that no coding is required to produce a working crawler, and it advertises a deployment in about five minutes and a first crawler in about half a minute, though those are the project's own claims rather than measured results. The other side of that trade is clear. A team that needs behaviour Webmagic has since changed, or needs control over the downloader layer, will find the console's convenience stops at the platform's own abstractions. The top level of the repository shows the split in practice: a `src/` directory, a `pom.xml` build file, an `ajaxDownloader/` directory, a `doc/` directory and an `examples/` directory of task definitions.

Template based extraction, with automatic body detection as a fallback

Collection runs from configured templates rather than from hand written extractors, and the feature list spells out three modes. A template can drive extraction directly, including on Ajax driven pages, which is what the `ajaxDownloader/` directory and the `examples/phantomjs_fetcher.js` script exist for. Where no template is configured, the platform attempts to detect the body text of a page on its own and to pull the article publication time out of it, which is the difference between pointing the tool at a URL and pointing it at a list of URLs with a schema. The third mode is mixed: fields can be extracted dynamically from a page while other fields are inserted statically at configuration time, which is how a fixed category or source label gets attached to every record from one task. The design is a sensible answer to the messiness of real sites, where a single site family mixes static structure with per article metadata. The cost is predictability. Automatic detection is convenient on uniform article pages and unreliable on the rest, and a task that quietly falls back to detected body text will produce records that differ in shape from those produced by a template, which matters if downstream code assumes one schema.

Collected records are managed, searchable and re-extractable

Once data lands, the platform treats it as a dataset rather than as a log file. Records can be searched, created, updated and deleted, and existing records can be re-processed against a new data template, which is the feature that decides whether a schema change costs a full re-crawl. Alongside management sits natural language processing over the collected text: keyword extraction, summary extraction and entity word extraction, all applied to the stored records. A recommendation module suggests related articles, and an analysis of associations between people and places in an article is offered on top of that. These are the parts of the system that no plain HTTP client gives you, and they are the reason a team would accept a console and a storage layer instead of calling a scraping library directly from its own code. Two limits are worth naming. The project does not document which language or model performs the Chinese processing, so accuracy cannot be assessed from the repository alone, and the entity relation analysis is described in a single line, with no schema for the output it produces.

Deployment rests on JDK 8, Tomcat 8.3 and an optional Elasticsearch 5.0

The stated dependencies are short, and the platform runs on Windows, Mac and Linux. The required components are JDK 8 or later and Tomcat 8.3 or later, which places the application on the older side of the servlet container generation. An optional component, Elasticsearch 5.0, appears in the list separately from the required ones, so it is used for storage or search in some configurations rather than being mandatory for every deployment. The consequence for a new user is that there are two supported shapes: a plain Tomcat deployment with whatever local storage the platform uses, and a deployment that also points at an Elasticsearch 5.0 instance. Since Elasticsearch 5.0 is a specific old major version, the second shape pins you to a cluster of that vintage unless the platform's index mapping still works against a newer one, which the repository does not say. The deployment guide, the usage manual, the secondary development manual and the FAQ were all moved out of the repository to a separate online documentation site, so the repository itself no longer holds the instructions a new operator needs.

Task definitions ship as JSON files you can copy and edit

The `examples/` directory is the most concrete part of the repository for someone deciding whether to use it. It holds JSON task definitions for specific sites, including `examples/news.163.com.json`, `examples/news.qq.com.json`, `examples/news.sohu.com.json`, `examples/www.chinanews.com.json` and `examples/www.oschina.net.json`, plus a variant named `examples/news.qq.com_time_autodetect.json` that shows the automatic time detection path in use, and `examples/phantomjs_fetcher.js` for pages that need a headless browser to render. Two things follow from this list. First, the configuration format is plain JSON, so a task can be version controlled and reviewed like any other artifact, which is a genuine advantage over clicking through a saved job in a console. Second, the examples are all Chinese news and developer community sites, and their selectors encode the current markup of those sites. Selectors rot faster than code, so a template copied from this directory in 2026 is likely to need its selectors revisited, and the project does not describe any maintenance process for the shipped examples. The `examples/README.md` is the place to look first, since it is the only English facing note in that directory.

The last push is recent, the last release tag is from 2017

The repository is not archived, and the last push was on 2026-06-03, so commits have landed within the last few months. The published release history tells a different story about shipping: V0.3 on 2017-01-09, V0.5.1 on 2017-03-07 and V0.6 on 2017-05-16 are the recent releases listed, and the description of the project page calls itself the Gather Platform 0.6 version release. A gap of roughly nine years between the newest tag and the most recent commit means the code is moving while the versioned artefacts are not, so any deployment has to be built from a commit rather than from a release. For a team that plans to run this in production, that is the central maintenance fact, more significant than the date of the last push. The licence is GPL-3.0, which matters for anyone embedding the platform in a larger product, since a distributed derivative inherits copyleft obligations on the covered code. The project also carries a Java web console, a storage layer and analysis tooling, so the covered surface is wide rather than a single jar.

Where the platform is the wrong tool

There are several cases where the console buys nothing. If you are writing a library or a service that needs to collect pages inside an existing pipeline, a console plus a servlet container plus an optional search cluster is heavy, and calling the Webmagic core directly from your own code keeps the control you would otherwise hand over to the task manager. If your collection is a handful of well understood endpoints, the template and management features are overhead, and the automatic body detection, which the project presents as a convenience, becomes the risk instead of the saving. If you need a current release with current dependencies, the tag history argues against it. The extraction of keywords, summaries and entity words is also only meaningful for Chinese text in practice, since that is the only language the examples and the stated feature set are written for, and an English language deployment would need to confirm that path before committing to it. Finally, the documentation split is a practical risk rather than a philosophical one: the manual lives on a separate documentation site, and a deployment guide that is not in the repository is a deployment guide that can disappear.

Editorial conclusion

Gather Platform suits teams that need a managed collection console, stored records and a search interface, and that prefer JSON task templates to writing crawler code. It does not suit a Java library author, a team that needs current upstream Webmagic behaviour, or anyone who wants a self-contained build, since the only published releases are the V0.3, V0.5.1 and V0.6 tags from 2017 and the deployment guide lives outside the repository. Before adopting it, check three things: whether the online documentation site is still reachable, whether your Elasticsearch 5.0 instance matches the optional component it expects, and how much of your workflow depends on the entity relation analysis that the console exposes.

Frequently asked questions

Official sources

1. gsh199449/spider on GitHub
2. Issues
3. License: GPL-3.0
4. README
5. Releases