NewsCrawler: a news-extractor Agent Skill for 12 platforms

What NewsCrawler is actually for

NewsCrawler began as a multi-platform news and content crawler collection, covering WeChat, Toutiao, Netease, Sohu, Tencent, Naver, Detik, Quora, BBC, CNN, Twitter and X, and the project description still lists those platforms. What the project calls its primary product today is different, and that shift is the interesting part.

The primary product is a portable news-extractor Agent Skill. It bundles the crawling code for twelve news and content platforms into one self contained directory so that an agent such as Codex or Claude Code, or anything else that understands the SKILL.md convention, can extract an article locally and get back unified JSON or Markdown.

The problem it addresses is context cost. Handing a generic agent a bare link leaves it to work out how to open the page, which block is the body text, whether to keep menus and recommendation widgets, and what order the images are in. The alternative path is to have the agent open a browser, load the DOM or the accessibility tree, identify the body, clean it up, and only then reason about it, with the page scripts, navigation, ads and duplicate links all in the context window along the way.

With the skill, the agent matches the platform, the platform code extracts the body, and the result is normalised into a NewsItem with title, metadata and ordered content. The stated point is not only saving scraping time but keeping the expensive model context for summarising, retrieving, comparing and judging.

The users are therefore agent builders, not data engineers. If you want a nightly crawl into a warehouse, this is the wrong shape.

The twelve platforms and how a URL selects an adapter

Platform selection is by URL pattern, and the documentation gives the identifier and a matching example for each one. WeChat is matched on mp.weixin.qq.com/s/..., Toutiao on toutiao.com/article/..., Netease on 163.com/news/article/..., Sohu on sohu.com/a/..., Tencent on news.qq.com/rain/a/..., Lenny's Newsletter on lennysnewsletter.com/..., Naver on any naver.com host, Detik on news.detik.com/..., Quora on any quora.com host, BBC on bbc.com/news/articles/..., CNN on a date path such as cnn.com/YYYY/MM/DD/..., and Twitter on x.com/<user>/status/<id>.

The mechanism is described in five steps. The agent identifies the news extraction task from the skill description. A detector module, named detector.py, selects the platform from the URL. The matching crawler fetches the page, and the platform specific code locates the body text and the media. A NewsItem then unifies the fields while preserving content order, and the agent reads only the JSON or the Markdown and uses the remaining context to summarise, retrieve or analyse.

The consequence is that the platform rules live in adapters that can be tested and reused, rather than being rediscovered by the agent on every run. The same design also means each platform is a separate maintenance obligation, since a site that changes its markup breaks its adapter and nothing else.

The documentation is frank about the failure modes: site structure, login strategies and anti-scraping rules change. When an extraction fails, the guidance is to check the URL shape, network access and cookie requirements first, and only then look at the corresponding adapter.

Installing the skill with npx skills or by handing over the README

The skill follows the open Agent Skills specification, so it can be discovered and installed automatically from the repository by the vercel-labs/skills tooling. The source directory is .claude/skills/news-extractor, which contains the complete SKILL.md, the scripts, the dependencies and the reference files.

The recommended route is a single command that needs no prior clone and no manual copying of directories:

npx skills add NanmiCoder/NewsCrawler --skill news-extractor -g

The command detects the agents supported on the machine and lets you choose the install target. You can also name the target explicitly, for example for Codex or for Claude Code:

# Codex
npx skills add NanmiCoder/NewsCrawler --skill news-extractor -g -a codex -y

The second documented route is to paste the project's README link into the agent and let it install the skill itself, with the instruction to run the platform listing to verify the installation and explicitly not to start Docker, MCP or the web UI.

The npx tool only installs the skill files. On first execution the agent follows SKILL.md and runs uv sync inside the skill directory to install the Python dependencies, so uv is a prerequisite.

After installation you hand the agent a link and a task in plain conversation, and it recognises the intent from the skill description and runs the matching script locally. No web UI, no FastAPI service, no MCP server and no Docker container are needed for that path.

The NewsItem shape and running the script by hand

Every platform is normalised to the same NewsItem structure, which is what makes the output usable downstream regardless of where the article came from.

The item carries a title, the news URL, a news identifier, and a meta_info object with the author name, the author URL and the publish time. The body is a contents list where each entry has a type, being text, image or video, a content field holding the paragraph text or the media URL, and a desc field for captions.

The contents list preserves the original order of text, images and videos, which matters because an article's images are usually interleaved with its paragraphs. Alongside it, the item exposes three parallel lists, texts, images and videos, so an agent or a downstream program can reach a specific media type directly without walking the ordered list.

The documentation splits the two output formats by use. Markdown suits reading, summarising and writing into a knowledge base. JSON suits retrieval, storage, analysis and workflow orchestration.

To verify by hand, you run the script in the skill directory:

uv run scripts/extract_news.py "URL" --format json --output ./output

The parameters and more examples are in the skill's own usage document, SKILL.md, inside the skill directory. The project also ships a call_example.py at the repository root, and a benchmarks directory that holds the token counting record described below.

The token measurement, and what the project says about its limits

The project publishes one concrete measurement, and it is careful about its own boundaries.

On 2026-07-20 the team ran a controlled comparison on a single real Detik News article, and the skill extracted ten paragraphs of body text and one image. All text was counted with the same tiktoken o200k_base encoder. The rendered browser DOM came to 59,788 tokens as the baseline. The browser accessibility snapshot was 6,842 tokens, which is 88.56 percent less. The visible text was 1,907 tokens, 96.81 percent less. The structured JSON from the skill was 1,730 tokens, 97.11 percent less, roughly a thirty-fourth of the DOM. The Markdown output was 633 tokens, 98.94 percent less, roughly a ninety-fourth.

The measurement method is documented too. On the browser side the same logged out session loaded the page and the team took the outer HTML, an accessibility snapshot and the visible body text. On the skill side the extraction script ran with the format flag set to both, and the counts were then taken with the same encoder.

The stated boundaries are worth reading before quoting any of it. Against the browser innerText, the JSON is only 9.28 percent smaller and the Markdown 66.81 percent, so the extra value is that the body is already segmented, the page noise is removed and the fields are stable, not merely that there are fewer characters. Against an already compressed accessibility snapshot the JSON is still 74.71 percent smaller. It is one dynamic page at one point in time and does not represent the same ratio for every site. The numbers cover the actual content payload only, excluding prompts, tool call wrappers, screenshots and model output. And different agents have different browser compression strategies, accessibility trees and tokenizers, which changes the absolute figures.

The full counts, hashes and reproduction recipe are kept in a benchmark record file under benchmarks/token-efficiency, and the original web page and article content were not copied into the repository.

The other surfaces: FastAPI backend, MCP server and Vue frontend

The repository is much larger than the skill, and the tree shows the shape of it. There is a news_crawler/ directory with the original crawler collection, a video_crawler/ directory, a libs/ directory, and three workspace members: news_extractor_core, news_extractor_backend and news_extractor_mcp. There is also a news-extractor-ui/ directory for the web interface.

The compose file wires three services on a shared network. The backend is a FastAPI service on port 8000, with a healthcheck that opens the root URL, and it persists extracted data through a bind mount of the data directory. The MCP service listens on port 8765 with its own health endpoint and shares the same data volume. The frontend is a Vue 3 application served by nginx and published on port 3021, and it depends on the backend.

The Dockerfile builds the same three targets. The Python base stage installs uv, copies the workspace files and the three members, and syncs the environment frozen without dev dependencies. A separate Node 18 alpine stage builds the frontend with npm, using npm ci when a lockfile exists and npm install otherwise. The backend runtime image exposes port 8000 with a healthcheck, and the default command starts news-extractor-backend bound to 0.0.0.0 on the documented port.

The packaging metadata shows the Python side is a uv workspace with hatchling as the build backend, and the project name is news-crawler-collection with a description that calls it a multi-platform news and content crawler collection for educational purposes. The runtime dependencies are pinned exactly, including lxml, parsel, playwright, pydantic, requests, tenacity, DrissionPage, pytz, demjson3 and curl-cffi, and the console scripts are news-extractor-backend and news-extractor-mcp. Python 3.10 or newer is required, and the uv configuration also lists a secondary index at the Tsinghua PyPI mirror.

Where NewsCrawler is the wrong tool

The first limit is the evidence. The token comparison rests on one article on one dynamic page at one point in time, and the project says so itself. A site that serves its body as a script, a paywalled outlet, or a platform that requires a logged in session will not behave the same way, and the twelve adapters are the surface area where that shows up.

The second is legal and operational, and it is not a small point. Crawling a platform's content without permission is a decision with consequences that depend on the platform and on where you are, and a GPL-3.0 licence on the crawler code says nothing about the rights in the articles it retrieves. Check the terms of each source before you point this at it.

The third is that this is a skill, not a scheduler. There is no cron, no queue, no retry policy, no incremental state and no database schema for a growing archive. The docker-compose services are for interactive use and inspection, not for a pipeline that has to keep running.

The fourth is that a site change breaks exactly one platform. The adapter pattern means a fix is local, but it also means a broken adapter fails silently for that site until someone notices.

Compare that with a general purpose browser automation tool such as Playwright driven by a script you write yourself. The difference in approach is that a browser tool works on any site the day you write the script, while NewsCrawler ships twelve site specific extractors that give far cleaner output for those twelve. Use the skill where the platform is known, and fall back to a general tool for everything else.

Licence, maintenance and the cost of keeping the adapters alive

NewsCrawler is released under the GPL-3.0 licence, with the text in the LICENSE file at the repository root, and the project documentation carries a GPL-3.0 badge. Strong copyleft matters here in a specific way: the crawler code is distributed as a package with entry points, and if you modify it and distribute your modified version, the terms require you to publish the corresponding source. Read the LICENSE file yourself for the exact conditions.

The licensing question is separate from the more common one for this kind of project, which is what you are allowed to do with the content the crawlers retrieve. GPL covers the code, not the articles, and the two questions have different answers.

On maintenance, the repository is not archived and the last push was on 2026-08-08, a little over two months before the current date, so the tree is being worked on but the pace is slower than a project with an active issue queue. There are no GitHub releases, so there is no tagged version to pin, and the package version in pyproject.toml is 0.1.0, which is the collection rather than the skill itself.

The upgrade cost sits in the adapters. Every site that changes its markup needs a new or updated crawler, and each adapter carries its own dependency on the site's login and anti-scraping behaviour. The project is candid that this changes, which is a good sign of honesty and a bad sign for the maintenance budget.

The repository does carry a few signals of a serious setup beyond the skill: a DOCKER_DEPLOYMENT.md document, an INSTALL_SKILL.md file, an AGENTS.md and a CLAUDE.md at the root, a .github/ directory, and a Dockerfile whose comments label each build stage, which is the kind of thing that makes a deployment reproducible by someone other than the author.

Editorial conclusion

NewsCrawler fits an engineer building an agent that has to read specific articles from WeChat, Toutiao, Detik, BBC, CNN, Twitter or similar sources, and who would rather not spend context on page furniture. It does not fit a bulk archive or a monitoring system, since the documented evidence is one article at one point in time and the project warns that site structure, login rules and anti-scraping measures change. Before adopting it, run the skill against a URL from each platform you care about, check the site terms of service for the sources you intend to crawl, and read the LICENSE file yourself, because the GPL-3.0 terms govern how you redistribute the crawler code.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-3.0
3. NanmiCoder/NewsCrawler on GitHub
4. README