# hacker-job: turning Hacker News hiring threads into structured, searchable data > hacker-job parses the free-text comments on each month's Ask HN: Who is Hiring? thread with an LLM, writes the extracted fields into committed JSONL files, and serves a React app on top. The dataset is the git history, so there is no database to run or back up. **hacker-job/hacker-job** — Play with hackernews' "who is hiring" - Repository: https://github.com/hacker-job/hacker-job - Website: https://hacker-job.com - Stars: 754 · Forks: 38 - Language: TypeScript - License: MIT - Published: 2026-10-21 · Updated: 2026-10-21 · Language: en - Canonical page: https://hysenlabs.com/projects/hacker-job-hacker-job ## The problem is that a thread is a wall of free text Each month, Hacker News runs a Who is Hiring? thread where companies post openings as free-text comments. The README calls the result impossible to search or compare over time, which is the actual problem. A single thread holds hundreds of posts, and a year of threads is thousands of them, all unstructured, all in the same format a human wrote at midnight. hacker-job parses them into structured data with an LLM and serves a React app on top of that. The homepage is hacker-job.com, and the topics on the repository are hacker, hackernews, job and nodejs. The interesting architectural choice is not the model call, it is where the output goes. The parsed data is committed to the repository as the source of truth, which means the dataset has a version history, is diffable, and needs no database. ## One JSONL file per month, and everything else derived The README's layout section shows the structure. The data directory is the dataset and is committed to the repository, and it holds one JSONL file per month under a jobs subdirectory, a jobs/index.json manifest of available months, a trends.json file with salary and keyword series, a pending.jsonl file of fetched posts awaiting AI analysis, and a hackers.json file of GitHub sponsors. Alongside it sit a scripts directory holding a Node and TypeScript pipeline for fetch, analyze, backfill, derive and store, and a frontend directory with a Vite, React and TypeScript app that builds to frontend/dist. The manifest and the trends file are marked as derived, so they can be regenerated rather than trusted. The pending.jsonl file is the queue of posts fetched but not yet analyzed, which is what makes the pipeline resumable. Each month file holds the raw Hacker News text alongside the AI-extracted fields, which is the right call for a research dataset, since you can always re-derive your own view from the raw text without paying for the model again. ## Fetching is cheap and analysis is expensive, so they are split The daily update is split into a cheap fetch and the AI step, and the two are connected by a queue file. The three commands are the core of the pipeline. ```bash npm run jobs:fetch # current thread → queue new posts (no AI) npm run jobs:analyze # drain queue → AI-parse → write data/jobs/.json + derived files npm run jobs:update # both, in sequence ``` Splitting them this way has two benefits. The fetch half touches the network and is cheap, so it can run often without a model budget. The analyze half is where the LLM cost sits, and it only looks at posts that have not been parsed yet. Failure handling follows from the same split: posts that fail to parse stay queued and retry on the next run, so a model outage or a rate limit does not lose data. The rest of the command set covers the other jobs. ```bash npm run jobs:backfill # rebuild every month from scratch (idempotent, slow) npm run jobs:derive # regenerate manifest + trends without fetching npm run hackers:fetch # refresh sponsors (needs GITHUB_TOKEN, see below) ``` The backfill command being idempotent matters more than it sounds, because rebuilding every month from scratch is the only way to change the extraction prompt and apply it to history. ## The LLM defaults to a local LM Studio server and can be pointed elsewhere The README states that LLM extraction uses the openai client and defaults to a local LM Studio server, with overrides through the LLM_BASE_URL, LLM_API_KEY and LLM_MODEL environment variables. Using the openai client is a practical choice: the pipeline speaks the standard chat completions API, so an LM Studio server on your laptop and a hosted endpoint both work without a code change. The dependency list is short, with openai at version 6.42.0 and p-limit for concurrency control, and the dev dependencies are tsx to run the TypeScript scripts directly, TypeScript, and the Node types. There is no schema validation library and no database driver in the list, which matches the design. The important operational constraint comes later: in CI the LLM must be a hosted, internet-reachable endpoint, because the local default will not work on a runner. ## Four GitHub Actions workflows, and a deploy that needs Pages enabled once The automation table in the README lists four workflows. update.yml runs daily at 13:00 UTC and does fetch plus analyze, then commits the data directory, and it takes LLM_BASE_URL, LLM_API_KEY and LLM_MODEL as secrets. derive.yml triggers on a push to main that touches scripts/store.ts or scripts/derive.ts, and re-derives trends.json and the manifest from existing jobs, commits, and triggers the deploy. refresh-github-data.yml refreshes the sponsors file and needs a GH_PAT, a token of the sponsored account with read:user scope. deploy.yml builds the frontend and the dataset and publishes to GitHub Pages on a push to main. The derived-files workflow is a nice touch: changing how trends are computed does not require re-running the model over every month, only recomputing from the stored extractions. For GitHub Pages you enable the source once under Settings, Pages, Source: GitHub Actions, and the custom domain lives in frontend/public/CNAME. ## The app reads data at runtime, so a refresh needs no rebuild The README makes a decoupling point that decides how you extend the project. The app fetches data/* at runtime, so data and app are fully decoupled and a data refresh needs no rebuild. A daily refresh only touches the current month's file, so git diffs stay tiny. Both facts follow from the same decision to make the committed files the interface between the pipeline and the frontend. The practical consequence is that the frontend never imports the dataset at build time, so a new month of data appears on the site as soon as the workflow commits it. In development, the dev server serves the repo-root data directory at /data/*, which gives the app live data while you work, and the README notes that this requires Node 18 or newer. The develop section shows the setup. ```bash git clone --depth 1 --single-branch --branch main git@github.com:hacker-job/hacker-job.git cd frontend && npm install && npm run dev # app at http://localhost:5173 npm install # pipeline deps (from repo root) ``` Two separate installs, one in frontend and one at the repo root, because the pipeline and the app have separate dependency sets. The dataset lives in the repository, so the site is a static build plus a set of JSON files, with no server to operate. ## Conclusion hacker-job fits researchers, analysts or recruiters who want to search across months of Hacker News hiring posts and compare what companies asked for over time, and who are comfortable running a Node pipeline against an LLM endpoint. It does not fit a production system that needs a transactional store, multi-user writes or low latency queries, since the dataset is a set of committed files refreshed on a schedule. Verify first that the extracted fields match what you need, because the parsing step is model output and the README says failed parses simply stay queued, then read the README's note that CI needs a hosted LLM endpoint because the local default will not work on a runner. ## FAQ ### How do I run the hacker-job data pipeline locally? Install dependencies at the repo root with npm install, then use npm run jobs:fetch to pull the current thread into the queue and npm run jobs:analyze to drain the queue and write the month files. npm run jobs:update runs both in sequence. ### Which LLM does hacker-job use? The README says extraction uses the openai client and defaults to a local LM Studio server. You can point it elsewhere with the LLM_BASE_URL, LLM_API_KEY and LLM_MODEL environment variables, and a hosted endpoint is required in CI since the local default will not work on a runner. ### Where is the hacker-job dataset stored? In the repository itself, under data/, with one JSONL file per month holding raw Hacker News text plus AI-extracted fields, alongside a derived manifest, a derived trends.json and a pending.jsonl queue. There is no database, and the commit is the persistence. ### How does hacker-job publish the site? A deploy.yml workflow builds the frontend and the dataset and publishes to GitHub Pages on a push to main, after you enable Pages once under Settings, Pages, Source: GitHub Actions. The custom domain is configured in frontend/public/CNAME. ## Sources - [hacker-job/hacker-job on GitHub](https://github.com/hacker-job/hacker-job) - [Issues](https://github.com/hacker-job/hacker-job/issues) - [License: MIT](https://github.com/hacker-job/hacker-job/blob/main/LICENSE) - [Project website](https://hacker-job.com) - [README](https://github.com/hacker-job/hacker-job/blob/main/README.md) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/hacker-job-hacker-job