gkamradt/needle-in-a-haystack: a sweep runner for long-context retrieval, with a recipe for every row
Doing simple retrieval from LLM models at various context lengths to measure accuracy
At a glance
- What is it?
- The niah CLI runs a grid of context lengths against needle depths, scores each response, and writes one JSONL row per cell. Its distinguishing choice is storing a reconstruction recipe instead of the rendered prompt, which keeps result files small but makes every surprising score auditable.
- Who is it for?
- Adopt niah if you need a repeatable sweep of context length against needle depth and want the exact prompt recoverable from the result file rather than stored inside it. Skip it if you need latency percentiles, streaming behaviour, or a hosted dashboard, since the output is JSONL and nothing else.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 99 days ago.
- What is it written in?
- Mainly Jupyter Notebook, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap niah fills between a one-off prompt test and a context-length claim
Most long-context claims come from a single prompt at a single length. That tells you almost nothing, because retrieval accuracy in a long context depends on where the fact sits, not just how many tokens surround it. A fact at the start of a 32k-token context and the same fact at the midpoint are different experiments. niah exists to run that grid explicitly: the README describes it as a sweep of (context length x needle depth) cells against any configured model, with one result row written per cell to a JSONL file. The audience is whoever has to defend a number, whether that is a team picking between two models for a retrieval feature or an engineer checking whether a new checkpoint actually improved recall at depth 80 percent. It is not an evaluation suite in the broad sense. There is no reasoning benchmark here, no code generation, no instruction-following battery. The scope is retrieval from a long context, and the built-in tasks stay inside that scope.
What the sweep actually varies, and why the sigmoid depth grid matters
The run config separates the two axes. context_lengths takes min, max, num and a scale, and depth_percents takes the same four keys. In the example config the lengths are linear from 2000 to 32000 across 8 steps, while the depths run 0 to 100 across 11 steps on a sigmoid scale. That asymmetry is the interesting part. Linear spacing at the edges of a context is where models usually hold up, and the middle is where recall tends to degrade, so a sigmoid spacing concentrates samples where the curve is likely to bend. A linear depth grid would spend most of its cells near the boundaries and produce a flattering plot. The seeds list adds a third dimension: each cell is repeated per seed, so a 8 x 11 grid with three seeds is 264 model calls, not 88. That number is worth computing before you launch a run, because the per-cell cost is recorded in cost_usd and the model config carries input and output pricing in USD per 1M tokens.
The four built-in tasks and what uuid_chain asks of the model
single places one fact at one depth and scores by exact match. multi spreads N facts evenly and returns a fractional score. uuid places one freshly generated identifier at one depth and the model has to repeat it, which removes any chance that the model memorised the answer from pretraining. uuid_chain is the one that goes beyond lookup. The README describes a chain of A to B to C links spread through the context, with a question that asks what value is associated with A without revealing the chain structure. The model has to discover the hops itself. The score details in the sample row make the grading visible: hops_correct 3 out of chain_length 5 yields a value of 0.6. That is a partial-credit scheme, not pass or fail, and it means a uuid_chain run reports how far along the chain the model got before losing the thread. For a retrieval feature in production, that distinction matters more than a binary hit rate, since a model that resolves two hops and stops is failing differently from one that never finds the first link.
Getting a first result: the fake provider, then a real key
The install path is short. pip install needlehaystack, then niah demo --fake runs a 2 x 3 sweep against an in-process fake model in about a second and writes results.jsonl, with no API key required. That is the fastest way to confirm the pipeline works on your machine. To hit a real model, the README shows writing OPENAI_API_KEY=sk-... into a .env file, which niah loads automatically, and then running niah demo, described as defaulting to gpt-4o-mini at roughly $0.01. Provider selection is a flag: niah demo --provider anthropic needs ANTHROPIC_API_KEY, and --provider cohere needs COHERE_API_KEY. For your own sweep you supply two YAML files, one run config that references one model config, and the CLI offers niah validate my-run.yaml to parse and resolve without making model calls, plus niah run my-run.yaml to execute and append to JSONL. The --dry-run flag on run does the same resolution, prints the plan, and exits, which is the cheap way to catch a bad model id before spending anything.
Model configs pass request keys straight through to the SDK
The model config is deliberately thin. An id, a runtime block naming the sdk and api, a client block with api_key_env, a request block, and a pricing block. Everything under request is forwarded verbatim to the provider SDK, so provider-specific knobs such as thinking, reasoning_effort or top_p can be added without touching code. The example model config sets max_tokens to 120000 and includes a thinking block with type adaptive and an output_config with effort set to medium. That passthrough design is the right call for a harness that wants to track provider features as they ship, but it has a consequence worth naming: niah does not validate the contents of request. A typo in a nested key will reach the SDK and surface as a provider error, not as a config error from niah. The pricing block is also manual. Input and output rates are numbers you write into YAML, so cost_usd is only as accurate as those two fields, and a stale rate quietly produces wrong cost totals without any warning.
Result rows store a recipe, not the prompt
Each JSONL row is a few KB regardless of context size, because niah does not store the rendered context. The README is explicit that storing a 200k-token context per row would balloon a single sweep into gigabytes. Instead the row carries a recipe: the haystack descriptor, the inserter name, a list of needle placements with insertion_token_index and actual_depth_percent, and a final_context_token_count. Alongside that sit the expected answer, the prompt question, the response, the score with its details, token usage, cost, duration, status, seed and timestamp. The niah reconstruct command walks that recipe and produces a byte-identical string of what the model saw, selectable by row with --row N and writable with --out. This is the strongest design decision in the project. When a score looks wrong, you can read the exact prompt rather than guessing, and the storage cost stays flat. The trade-off is that reconstruction is a computation, not a lookup. If the haystack files change on disk after the run, the recipe points at a source that no longer matches, and the README does not describe any hashing or snapshot step that would detect this.
Where niah is the wrong tool
niah measures whether a model can retrieve a planted fact. It does not measure whether the model can use that fact well, and it has no notion of latency percentiles, time to first token, or streaming behaviour, which are often what actually decides a long-context deployment. The runner config exposes concurrency and retries and a resume flag, so throughput is tunable, but the output is a JSONL file and nothing else. There is no dashboard, no aggregation command, and no built-in comparison across two runs. If you want a table of accuracy by depth, you write the code that reads the JSONL. There is also a scoring constraint: single and uuid are exact-match, so a model that answers correctly with different casing or extra words may be marked wrong unless the task's scorer normalises it. The README does not describe the normalisation rules, so that is something to check in the task source before treating a low score as a retrieval failure. Finally, the provider list is OpenAI, Anthropic and Cohere out of the box. Anything else means writing a ModelProvider and registering it, which the README frames as a small plugin but is still work you own.
How this differs from a general evaluation framework
Tools such as lm-evaluation-harness cover many benchmarks across many task families, with a shared scoring and reporting layer. niah goes the other direction. It covers one family of questions, retrieval from a long context, and pushes variation into two YAML files rather than into a benchmark registry. The practical difference shows up when you want to change the experiment. In a general harness you often add a task by writing a dataset adapter that conforms to an existing interface. Here you write a class with generate_needle, insert, question and score, then call register_task, and the README states that nothing in the runner needs to change. The same registry pattern applies to providers, haystack sources and scorers, each behind a small Protocol. That is a narrower commitment with a lower ceiling: you get exactly the retrieval sweep you configured, and you get no cross-benchmark reporting for free. If your question is whether a model is good in general, this is the wrong instrument. If your question is whether it finds your fact at token 15876, it is the right one.
Maintenance, licence and what to check before a long run
The project is active, with a v2.0.0 release on 2026-05-30 described as a clean refactor and a last push in June 2026. The result rows carry schema_version, currently 2, which is the signal that the recipe format has changed across versions and that older result files may not reconstruct under a newer install. Pin the version you run against if you intend to keep result files for comparison. The repository lists its licence as NOASSERTION, which means GitHub could not map the licence file to a known identifier. That is not a statement about the terms themselves, and it is not legal advice, but before shipping niah inside a commercial pipeline you should read the actual licence file in the repository and confirm the terms with whoever handles licensing on your side. On cost, the harness makes spend easy to see and easy to underestimate: an 8 x 11 grid with three seeds is 264 calls, each billed at the rates you typed into the model config, and the resume flag exists precisely because a long sweep may need to be restarted. Run niah validate first, then niah run with --dry-run to see the resolved plan and cell count, and only then commit to the full sweep.
Editorial conclusion
Adopt niah if you need a repeatable sweep of context length against needle depth and want the exact prompt recoverable from the result file rather than stored inside it. Skip it if you need latency percentiles, streaming behaviour, or a hosted dashboard, since the output is JSONL and nothing else. Before trusting a run, execute niah validate on your run config, then niah run with --dry-run to confirm the resolved model and cell count, and check the schema_version field in the first row so you know which recipe format you are reconstructing against.
Community notes