hacker-news-digest: a static site that pre-summarizes Hacker News with an LLM

The problem: Hacker News gives you links, not reading time

Hacker News is a ranked list of URLs and comment threads. Deciding whether an article is worth opening requires opening it, and the front page turns over faster than most people can read. hacker-news-digest attacks that gap by generating the summary before you arrive. The README describes the output as extracting summaries and illustrations from Hacker News articles to provide a news scanning experience, and the live result is a static page at hackernews.betacat.io where each entry carries a short generated abstract next to the original link. The audience is people who treat the front page as a triage queue: they want the gist, the score, the comment count and the submission time in one view, then decide whether to click through. It is explicitly not a replacement for reading the article. Nothing in the README claims the summaries are verified against the source, and the site itself labels each one with the model that produced it, which is the honest framing: the summary is a model output, and the model is named.

The pipeline: crawl, extract, illustrate, summarize, render

The README lays out five periodic actions, and the order matters. First, the project parses the Hacker News page to obtain a list of news articles, so the input is the rendered HTML of the site rather than an API response. Second, it extracts the main content from each article using a score algorithm, and the README links to a companion notebook titled How to extract main content from web pages using Machine Learning. That step is the fragile one: it is a heuristic over page structure, and pages that render their body with JavaScript or bury it behind unusual markup will yield thin or empty text, which then feeds a thin summary. Third, it finds the most suitable illustration for each article and makes a local copy, so images are not hotlinked from the source. Fourth, it generates summaries using the OpenAI API, or invokes a local model as a fallback when the API is unavailable; the README names local Qwen as that fallback. Fifth, it renders a template incorporating illustrations and summaries and deploys to GitHub Pages. The data flow is therefore one-directional and batch: no server sits behind the result, and every page is a snapshot produced by the last run. The README also notes that translation is performed by the same LLM with a single extra step in the prompt, which is why a Chinese edition exists at /zh.html.

What the interface actually shows you

Each card exposes a fixed set of fields, and the README documents the icon for each: a heart for points (upvotes from the community), a person for the submitting user, a clock for submission time in human-readable form, a speech bubble for comment count that links to the comment page, a link icon for the source of the news, and a newspaper icon for the summary model. That last field is the most interesting design decision in the project. Showing which model produced a given summary turns an otherwise opaque generation step into something a reader can calibrate against: if the fallback model was used, the reader knows the summary came from a different pipeline than the primary one. The README also lists sorting by points, comment count or publication time, and filtering to the top N articles by points. Those are client-side conveniences over the generated set, not a query language; the set itself is whatever the last crawl produced.

Running it yourself: what the repository gives you

The README does not include a full installation walkthrough, so treat this as the boundary of what can be confirmed. What is visible is a Python project on the master branch, a LICENSE file, a GitHub Actions workflow at .github/workflows/static.yml whose badge appears at the top of the README, and a companion notebook for the extraction algorithm. The workflow name and the deployment target imply the intended operating model: a scheduled job runs the pipeline, renders the template, and publishes the result to GitHub Pages, which is also how the hosted site is produced. The README states the site is a static site hosted on GitHub Pages and that it performs periodic actions, so the schedule lives in that workflow rather than in a long-running process. To run it you need Python dependencies for crawling and rendering, an OpenAI API key for the primary summarization path, and enough local resources to serve Qwen if you want the fallback to work. The README does not document the exact environment variable names, the model identifiers, or the template location, so anyone forking this should read the workflow file and the source before assuming a configuration contract exists. That is a real documentation gap, not a hidden feature.

Where it breaks: extraction, cost and the missing comment layer

The score-based main-content extraction is the weakest link. It works on pages whose article body is present in the served HTML and fails, or degrades, on pages that need a real browser. The project's own TODO acknowledges this: a better way to scrap websites, maybe PhantomJS and Selenium, is the first unchecked item. Until that lands, a JavaScript-heavy blog may produce a summary of navigation chrome rather than prose, and because the pipeline is batch, you will not notice unless you compare the summary against the article. The second constraint is cost and dependency. Summaries are primarily generated by a remote LLM service, so each run spends tokens proportional to the number of articles and the length of their extracted text. The local Qwen fallback exists precisely because the remote service can be unavailable, which means the project treats LLM access as an unreliable dependency rather than a given. Running the fallback shifts the cost to local compute and may change summary quality, which the model icon makes visible. The third gap is comments. Hacker News discussion is often the reason a link is on the front page, and the README lists summarizing comments as an open TODO with a link to the discussion thread. The current output gives you a comment count, not the argument inside it.

Alternatives and the difference in approach

The obvious alternative is the official Hacker News API, which the project itself lists as a TODO: switch to Hacker News API. The difference is structural. The API returns structured JSON for items, scores and comment trees, so you get stable fields and no HTML parsing, but it returns no article text and no summaries, which means you would still need your own fetch-and-summarize stage. A second alternative is a general read-later service with a summarization feature, where you save individual links and the summary is generated on demand for the pages you chose. hacker-news-digest inverts that: it summarizes the whole front page whether or not you would have picked those links, and it publishes the result as a static artifact anyone can read. That is a better fit for scanning and a worse fit for a personal queue. A third option is simply reading the RSS feed of Hacker News, which the project also supports per issues #14 and #19; RSS gives you titles and links with no generation step, no API key and no extraction failures, at the cost of doing the triage yourself. The trade is convenience against dependency, and this project sits firmly on the convenience side.

Maintenance, licence and what to check before forking

The repository is not archived and the last push recorded is 2026-08-13, so the project is active as of that timestamp. There are no releases retrieved, which means there is no versioned artifact to pin; you track the master branch. That has a concrete consequence for anyone self-hosting: an upstream change to the prompt, the template or the crawler arrives without a changelog entry, and your generated site changes with it. Budget for reading commits rather than upgrading a dependency. The licence situation needs care because the material is inconsistent. The README badge and the description field say LGPL-3.0, while the badge image in the README is labelled GPLv3 and links to the LICENSE file. Those are different licences with different obligations for anyone who redistributes a modified version, and the badge is not the authoritative text. Read the LICENSE file in the repository and, if you plan to redistribute a fork, get your own advice rather than relying on the badge. One practical note on the generated output: the site embeds local copies of illustrations taken from the source articles, which is a separate copyright question from the code licence and is not addressed in the README.

Editorial conclusion

Adopt hacker-news-digest if you want a self-hosted, GitHub Pages-hosted digest of Hacker News where summaries are generated ahead of time and every article carries a visible model attribution. Do not adopt it if you need comment-level insight, per-user filtering, or summaries you can trust without reading the source; the README lists comment summarization and a switch to the official Hacker News API as open TODO items, and the extraction step is a score-based heuristic rather than a full renderer. Before committing, verify three things in your own fork: which remote LLM endpoint the pipeline calls and what it costs per run, whether your target sites survive the main-content score algorithm or need the PhantomJS and Selenium work listed in the TODO, and whether the LGPL-3.0 terms fit how you intend to redistribute the generated site.

Official sources

1. Issues
2. License: LGPL-3.0
3. polyrabbit/hacker-news-digest on GitHub
4. Project website
5. README