LLM-Based-Intelligent-Public-Opinion-Analytics-Assistant: a crawler-plus-LLM stack for 26 Chinese trending boards

What problem the 26-board crawler actually solves

Most trending-topic tooling stops at a dashboard. This project's stated goal is different: it collects ranking data from 15 platforms across 26 boards, then puts a large language model in front of that data so a user can ask questions in natural language. The README lists four query modes: ranking lookup, topic-specific search, topic clustering, and sentiment analysis. The intended user is someone doing Chinese-language public opinion monitoring who wants to skip manual board-hopping. The push layer is the second half of the pitch. Instead of reading a dashboard, you receive a report through a WeCom group bot, a WeCom application that forwards to personal WeChat, a Telegram bot, or SMTP email. The README's own sample report shows the output shape: a timestamped digest with clustered findings, per-item titles, source URLs, and a summary table. Note that the sample is dated 2026-04-07 and contains claims about GPT-6, DeepSeek V4, and Anthropic revenue. Those are illustrative content from a sample run, not verified facts, and the repository does not claim to have validated them.

The crawler and the analysis agent are two separate programs

The repository layout is explicit about the split. `hotsearchcrawler` is described as the crawler cluster and as completely separate from the analysis system, which lives in `hotsearch_analysis_agent`. That separation is the main architectural decision visible in the material, and it has consequences. The crawlers write into MySQL, and the analysis system reads from the same database through parameters set in its `.env` file. Because the two halves communicate only through the database, you can run the crawler on one machine and the agent on another, and a crawler failure does not take down querying. It also means there is no streaming path: the agent sees whatever the last crawl wrote. The README does not state a crawl interval, a scheduling mechanism, or how staleness is surfaced to the user. Startup is `app.py` for the project, and the README refers to `run_spiders` as the crawler entry point that the frontend triggers, with `runspider-test` as a crawler test file. The frontend also exposes keyboard shortcuts to start and stop crawler tasks, which suggests crawling is operator-driven rather than scheduled by default. That is a real constraint for anyone expecting continuous collection.

Browser drivers and MySQL are hard prerequisites

Two dependencies dominate setup. The first is a browser driver, because the project extracts news detail pages through a real browser. The README devotes six numbered steps to this: check your Edge or Chrome version under Settings then About, download the matching ChromeDriver or EdgeDriver, find the browser install path, place the driver next to the browser executable or somewhere on PATH, add that directory to PATH, then verify with `chromedriver --version` or `msedgedriver --version`. This is the classic Selenium maintenance burden. Every browser auto-update can break the driver pairing, and the README offers no version pinning or container image to avoid it. The second prerequisite is MySQL. You install it separately and then, in the README's words, build the corresponding database and tables by referring to `init.py`. The README does not enumerate the schema, so the tables you get are whatever `init.py` creates; read that file before deploying. Python dependencies go in through `pip install -r requirements.txt` inside a virtual environment. Only after both are in place does the analysis system become usable.

Configuration keys you will actually edit

Configuration is split across three places. The crawler cluster reads `hotsearchcrawler/settings`, where you set MySQL connection parameters and, optionally, per-platform cookies. The analysis system reads a `.env` file containing MySQL parameters, the OpenAI-format model endpoint, push credentials for each channel, and two tuning values the README names directly: history memory turns and model temperature. Platform cookies for detail-page extraction live in `config/cookies`, and the README is clear that they are optional but that omitting them costs you detail-page extraction on some platforms. On the push side, the WeCom group bot needs a single `WECOM_WEBHOOK` URL, which you can get by adding a bot to any group chat and copying the generated webhook. Pushing to personal WeChat through a WeCom application needs four values: `WECOM_CORP_ID`, `WECOM_AGENT_ID`, `WECOM_SECRET`, and `WECOM_USER_ID`. The README notes that multiple recipients are separated by a pipe, as in `zhangsan|lisi|wangwu`. It also recommends email first, calling SMTP the simple option. There is a `test_push_task` file for exercising push without running the full pipeline, which is the sensible first thing to configure.

Where the documentation stops and the guessing starts

The README is a setup guide, not an operations manual. Several things a production user needs are absent. There is no stated crawl frequency, no retry or rate-limit policy for the 15 platforms, and no description of how the crawler handles a board that changes its markup. There are no releases in the repository, so there is no versioned artifact to pin and no changelog to read before upgrading; you track `main`. The model layer is described as OpenAI-format, which implies you supply the endpoint, but the README does not list which models were validated or what context length the clustering step assumes. The technical selection section recommends Huawei's Pangu model and links to an openPangu-Embedded-7B download, with claims about long-text parsing and Chinese sentiment handling. Those claims come from the project's own comparison and the README does not publish the comparison set, the prompts, or the scores, so treat them as the authors' preference rather than a measured result. The sample report's content, including its emoji-laden formatting, suggests the report template is tuned for chat-style channels rather than archival storage.

The alternative: scheduled collection with a BI layer

The obvious comparison is a conventional stack: a scheduled scraper writing to a warehouse, with a BI tool or notebook on top. That approach differs in where the intelligence sits. In this project the model is the query interface and the clustering engine, so a user asks a question and gets a synthesized answer with cited links. In a BI setup you define the metrics and dimensions up front and the dashboard is deterministic and cheap to re-run. The trade-off is real in both directions. A BI dashboard cannot answer an unanticipated question like a clustering request over today's board data without new modeling work, and it will not produce a written digest. This project cannot give you reproducible numbers, because a model at a nonzero temperature produces different phrasing on each run, and the README exposes temperature as a configurable value. If your requirement is auditable, repeatable reporting, the deterministic route is the better fit. If your requirement is exploratory question answering over fresh trending data, this project's design is aimed at exactly that.

Maintenance cost, licence, and who should walk away

The maintenance surface has three parts. Browser drivers need attention whenever Chrome or Edge updates. Platform cookies expire, and each expiry degrades detail-page extraction on that platform rather than failing loudly, if the README's optional-cookie note is read literally. The model endpoint is a recurring cost and an external dependency unless you run a local model, which the Pangu section presents as the reason for that selection. The project is MIT licensed, which permits commercial use and modification provided the copyright notice and licence text are retained; that is a summary of the licence identifier in the repository metadata, not legal advice, and you should read the LICENSE file yourself. There is no published release cadence, and the last push recorded is 2026-04-07, so you are depending on an actively moving `main`. Walk away if you need an English interface, since the README, the report template, and the configuration guidance are all Chinese. Walk away if you cannot run MySQL or a browser binary. Stay if you are already operating Chinese-language social listening and want the conversational layer without building it.

Editorial conclusion

Adopt it if you already run MySQL, are comfortable maintaining Selenium browser drivers, and need conversational querying over Chinese trending boards with a push channel. Do not adopt it if you need packaged releases, an English-language interface, or a system that runs without a browser binary. Before anything else, read init.py to see what tables it actually creates and check whether the crawler settings file expects cookies you do not have.

Official sources

1. hmmnxkl/LLM-Based-Intelligent-Public-Opinion-Analytics-Assistant on GitHub
2. Issues
3. License: MIT
4. README