Hiring Agent: reading the resume scorer that HackerRank open sourced

The problem is volume, not judgement

The README states the motivating number plainly: HackerRank receives 50,000 to 60,000 intern applications a year, and no human can read that many resumes well. The stated goal is to rank them, deciding which resumes to read first. Resumes below the cutoff are filtered out, but the README says the cutoff is intentionally set very low so only candidates at the very bottom of the distribution are removed. Most applications still reach a human.

That framing matters because it tells you who the tool is for. It is for a team that already has a review process and wants to change the order in which a human reads the pile. It is not for a team trying to replace the first review pass. The README also pushes back on three misreadings: it is not an ATS, it was not used to screen HackerRank's own open roles, and it is not a product offered to HackerRank customers. Anyone evaluating it as a drop-in applicant tracking system is evaluating the wrong artifact.

The README adds a second disclosure that is easy to miss. The repository ships with gemma4:latest as the default because it runs locally on most laptops without a cloud API key, while actual intern resumes at HackerRank are evaluated with a top-tier Gemini model. The repository ships a demo config, not the production one. Any conclusion you draw from running the default setup is a conclusion about the demo.

Five modules, one linear path from PDF bytes to category scores

The architecture section lists a short pipeline. pymupdf_rag.py converts PDF pages to Markdown-like text. pdf.py then calls the LLM per section, using Jinja templates stored under prompts/templates. github.py fetches a profile and repositories, classifies projects, and asks the LLM to select the top seven. evaluator.py runs a strict-scored evaluation with fairness constraints. score.py orchestrates the whole thing and writes a CSV when development mode is on.

Two details in that description carry more weight than their length suggests. The first is per-section extraction. Rather than sending the whole resume in one prompt and asking for a JSON blob, the pipeline runs separate extractions against separate templates. That gives you a place to inspect and edit the instructions for one section without touching the others, which is the practical reason the prompts/ directory is called out as a key module. It also means extraction cost and latency scale with the number of sections you enable.

The second is the GitHub branch. The tool does not just read a link. It fetches profile and repository data, classifies projects, and then asks the model to pick the top seven. That selection step is where a rubric becomes opinionated: which seven repositories represent a person is a judgement, and the README's own coverage list flags the consequence. A GitHub-centric rubric disadvantages engineers whose work sits in private enterprise repositories, because there is nothing public to classify. If your candidate pool skews toward people who cannot publish their work, the enrichment step adds noise in place of signal.

Around the pipeline sit models.py for Pydantic schemas and provider interfaces, llm_utils.py for provider initialization and response cleanup, and transform.py for normalization from loose LLM JSON into JSON Resume style. That last module is an admission worth reading twice: the model's JSON is described as loose, and a separate normalization layer exists to make it usable.

Getting it running: Python 3.11, one backend, one config

The prerequisites are short. Python 3.11 or newer, with .python-version pinned to 3.11.13. Then exactly one LLM backend: either Ollama, installed from the official site and started with ollama serve, or Google Gemini with an API key from aistudio.google.com.

The documented setup is a clone, a virtual environment, and a requirements install:

git clone https://github.com/interviewstreet/hiring-agent cd hiring-agent python -m venv .venv source .venv/bin/activate pip install -r requirements.txt

The README gives the Windows activation line as .venv\Scripts\activate. After that, the README's contents list points to an Ollama models subsection and a Configuration section, and the CLI usage section describes how to invoke the pipeline. The supplied material does not include the body of those sections, so the exact model-pull command, the config keys, and the CLI flags are not something I can reproduce here. Check the Configuration and CLI usage sections of the README on the main branch before assuming a flag name.

What the material does establish about configuration is the shape of the decision. You are choosing a provider and a model, and the default in the repository is a local Ollama model chosen for laptop compatibility rather than evaluation quality. If you intend to compare your results against anything, record which model produced them, because the README is explicit that the shipped default and the production model are not the same thing.

The same resume does not get the same score

The README's coverage table is unusually candid for a project page, and it is the most useful part of the document. One linked analysis, by Dan Kinsky, runs the same resume 100 times and isolates which categories are stable and which are not. Technical skills hold up. Project quality judgements move. The article attributes the spread to LLM non-determinism. A second write-up, from Pinggy, reproduces the variance and reports a separate problem: invisible text embedded in a PDF can inflate a score significantly.

Read those two findings together and the practical constraint is clear. A single score is not a measurement. It is one draw from a distribution, and the width of that distribution differs by category. If your workflow compares two candidates whose scores differ by a few points, and the difference sits in a noisy category, you may be ranking sampling noise. The stable categories are the ones worth acting on.

The invisible-text finding is the sharper of the two, because it is not a property of the model. It is a property of the path from PDF to prompt. Text that a human reader cannot see still reaches the extractor, and the extractor has no way to know it was hidden. The README lists this as coverage of outside analysis rather than as a fixed issue, so treat it as an open question about the current code rather than a resolved one. Verify it yourself against pymupdf_rag.py before putting the pipeline in front of real applicants.

A third linked piece, from ByteIota, raises a slower problem: once the rubric is public, candidates can optimize for it. That does not break the tool today. It means the signal you are extracting degrades as awareness spreads, which is a cost you inherit by using a published rubric at all.

Fairness constraints and a low cutoff are design choices, not guarantees

The evaluator step is described as a strict-scored evaluation with fairness constraints, producing category scores, evidence, bonus points and deductions. The word evidence is doing real work here. A score with attached evidence can be audited by a human reviewer, which is the difference between a ranking aid and an oracle. If the evidence field is empty or generic for a given candidate, the score is not auditable and the fairness constraints have not bought you anything.

The intentionally low cutoff is the other design choice worth naming. It is a deliberate trade: fewer false removals, more resumes for humans to read. That trade only pays off if the human review capacity actually exists. A team that sets a low cutoff and then does not read the passing pile has kept the cost of the filter and discarded its benefit.

The README also points out that HackerRank has since shipped AI Interviewer (Chakra) to automate the first interview round, so candidates are no longer assessed on the resume alone. That is context about the originating organisation, not a feature of this repository. It does tell you something about the tool's intended position: a resume score was a first-pass ranking signal inside a larger process, not a terminal verdict.

Where it is the wrong tool, and what to use instead

This is the wrong tool if you need deterministic output. Nothing in the material suggests a mechanism for pinning a score across runs: no seed control is described, no ensemble step, no cached evaluation keyed by input hash. The variance analyses in the README's own coverage list are the evidence. If your process requires that the same resume receive the same score twice, this pipeline as documented does not provide that, and you would be building the guarantee yourself.

It is also the wrong tool for candidate pools whose work is not publicly visible. The GitHub enrichment step fetches a profile and repositories and asks the model to select the top seven. A candidate with no public repositories does not get a neutral score on that dimension. They get an absence, and the rubric has to interpret it.

For a concrete alternative with a genuinely different approach, look at structured application forms with explicit, human-authored scoring rules, where the fields are fixed and the weights are set by the hiring team. The difference is not the model. It is where the judgement lives. In this pipeline, a language model reads free-form prose and decides what the sections mean before any scoring happens. In a structured form, the candidate declares the fields and the scoring is arithmetic over declared values. You lose the ability to accept any resume format. You gain reproducibility and a rubric you can defend line by line. Teams that need the second property should not reach for this repository first.

A middle path the material supports: use this pipeline for extraction only. Run pymupdf_rag.py and the per-section prompts to turn PDFs into JSON Resume shaped records via transform.py, then score those records with your own deterministic rules. You keep the parsing work, which is the tedious part, and drop the part whose variance is documented.

Maintenance cost and the MIT licence

The repository is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is the whole of what the licence identifier tells you, and it is not legal advice: if you plan to run automated scoring over applicants in a jurisdiction with rules about automated decision-making, the licence does not address that question, and the README's coverage list shows the Hacker News thread raising GDPR Article 22 as an unresolved point of discussion.

The maintenance surface is small and mostly external. There are no releases retrieved for this repository, so there is no versioned artifact to pin against; you track the main branch. The pipeline depends on Python 3.11, on an LLM backend, and on the prompts under prompts/templates. The prompts are the part most likely to need your attention, because they encode the rubric, and the README's own coverage notes that a public rubric invites optimization. If you fork this, expect to maintain your prompt templates as a living document rather than a vendored dependency.

The provider choice is the recurring cost. Ollama keeps marginal evaluation cost near zero and keeps resume text on your machine, at the price of a weaker model. Gemini moves the cost to per-call API pricing and sends resume contents to a third party. The README's disclosure that production used a top-tier Gemini model while the repository ships a local default is the clearest signal about which trade the maintainers made for the demo, and it is a trade you have to make again for yourself.

Editorial conclusion

Adopt it if you want a readable, MIT-licensed reference implementation of LLM-based resume extraction and scoring that you can run offline with Ollama, and if you are prepared to treat the output as a sorting aid rather than a decision. Do not adopt it if you need reproducible scores for the same input, if your candidates' work lives in private repositories, or if you want a hosted product with support. Before running it on real applicants, verify three things in the code: which model your config actually points at, how many times the evaluation step is called per resume, and whether the PDF text extraction path strips invisible or zero-width text before the text reaches the prompt.

Official sources

1. interviewstreet/hiring-agent on GitHub
2. Issues
3. License: MIT
4. README