livenerf: Tracking Claude Opus 5.5 Capability Drift

Why livenerf Exists for Claude Opus 5.5 Followers

livenerf addresses a narrow problem: determining whether a named frontier model becomes less capable after launch. The project was created around reports that Anthropic might reduce model quality after release through quantization, routing, lower inference effort, or substitution. The README also recognizes the opposite possibility: observed differences may be noise or ordinary variation. Without a baseline captured near launch, later assessments cannot cleanly distinguish a sustained change from model-to-model sampling variation.

The intended users are people who already suspect post-launch regression and need a dated measurement trail rather than an anecdotal comparison. Model providers, evaluation engineers, and teams operating long-lived Claude workflows fall into that group. livenerf is not a general benchmark that ranks many models. It follows Claude Opus 5.5 across a fixed schedule, using the same questions and scoring path after launch.

The series began on 2026-09-24, approximately 2.5 days after Claude Opus 5.5 was released on 2026-09-22. The schedule covers 30 daily runs. Days 1 through 10 form the baseline, followed by two 10-day comparison windows. This structure limits the analysis to specific periods and does not establish continuous monitoring beyond the documented series. It also means the first supported comparison appears only after enough observations have been collected.

livenerf's Frozen Panel and Daily Scoring Flow

The benchmark begins with a fixed panel of 78 questions selected from 2,336 questions drawn from GPQA Diamond, MMLU-Pro, competition mathematics, and AIME 2025 to 2026. Candidate questions were screened with four samples each. Claude Opus 5.5 answered roughly 93% correctly during that screening. Most questions were always correct or always incorrect, so including them would add little information about small changes. The 78 questions labeled as sometimes correct form the panel.

Each day is intended to run the full panel with 90 samples. A run passes through a pinned Claude Code command-line interface, records model outputs and token usage, applies exact graders, and appends raw evaluation logs. The README says the harness hash and CLI version remain fixed so that changes in the measurement setup do not become confused with model changes. The documented pinned CLI version is `2.1.280`, and all 21 reported completed runs used harness hash `461391b6fce64167`.

Results are analyzed as daily scores, baseline distributions, and 10-day windows rather than as isolated anecdotes. Token use provides a second signal because validation found effort reductions were more visible in output tokens than in accuracy. The project uses Inspect as its evaluation framework and follows the statistical approach described in Anthropic's `Adding Error Bars to Evals`. This does not make the hosted model deterministic. Sampling parameters are unavailable, thinking cannot be disabled, and the project cannot remove all server-side variation.

livenerf Installation Evidence and Plot Regeneration

The README does not present a complete installation and first-run tutorial for a new user. The `pyproject.toml` file requires Python 3.11 or later and declares dependencies including `inspect_ai==0.3.266`, `pandas>=2.0`, `pyarrow>=15`, and `statsmodels>=0.14`. The repository includes `uv.lock`, but the README does not provide an exact installation command, so one should not be inferred from the file's presence.

The documented first operation available in the README is plot regeneration from existing `.eval` logs. It runs the `livenerf.plot` module through Python. This is useful for inspecting benchmark output, but it is not a complete benchmark setup because the README does not give commands for configuring Claude credentials, invoking a daily evaluation, or building a fresh evaluation dataset.

The plotting command below regenerates the score visualization from the existing evaluation logs. The README notes that the daily run also redraws it:

python -m livenerf.plot

A user seeking a standalone collector should treat the missing setup instructions as a material adoption constraint. The repository contains `scripts/`, `prompts/`, `data/`, and the `livenerf/` package, but their presence alone does not establish a supported end-to-end command sequence. That sequence must be confirmed in the repository's current operational documentation or with its maintainers before this is treated as a turnkey benchmark.

What livenerf Can Detect and What It Cannot

The documented statistical design detects an accuracy shift of about 7.5 percentage points per 10-day window using one daily run of the full panel. The realized minimum detectable effect reported from the baseline is 6.6 points. Its power is therefore tied to a specific sample count and schedule. It is not evidence that livenerf can identify every smaller degradation.

The clearest known limitation emerged during validation. Replacing Claude Opus 5.5 with Opus 5 produced a reported change of minus 3.8 plus or minus 6.3 accuracy points and 23% fewer tokens. That difference was not distinguishable from Opus 5.5 at the documented 99% criterion. Although a 10-day window contains about 2.5 times as many samples as the validation run, the project does not claim that this would be enough.

The question set introduces another boundary. A report-only audit found eight answer keys that appear wrong and 30 ambiguous questions among the 78 panel questions, plus two later exclusions. These questions were retained, and a pre-registered sensitivity analysis is specified to rerun results without them. This is a transparent limitation, but it also means interpretation depends on accepting a panel built from questions a strong model sometimes answers incorrectly.

Serving behavior can also interrupt measurements. According to the README, the safety classifier sometimes responds with Opus 5, refuses some biology or mathematics questions, or otherwise interferes with a sample. Those samples are rejected and counted, and affected questions are excluded. A benchmark intended to catch hidden model substitution must therefore treat classifier routing as a measurable deviation, not silently blend it into normal sampling noise.

livenerf Compared with Snapshot Evaluations

livenerf differs from a conventional snapshot evaluation in time. A snapshot assessment can evaluate the model available at one moment and record its score. livenerf instead establishes a launch-adjacent baseline, repeats a locked panel on a daily schedule, and compares later windows against that baseline. This makes it better suited to the specific question of post-launch capability drift.

That advantage comes with operational constraints. A snapshot can be run when needed. livenerf depends on repeated access through the documented Claude Max subscription and headless Claude Code, with one reported day using Anthropic API credits because the Max weekly meter was exhausted. API spending, subscription limits, daily execution, and log retention therefore affect whether the time series remains comparable.

The benchmark is also much narrower than a general evaluation suite. Its panel emphasizes questions with uncertain initial answers and intentionally avoids questions the model always got right or always got wrong. This increases sensitivity around its target change, but it does not measure broad coding reliability, instruction following, safety behavior, latency, cost at scale, or performance on everyday production tasks. Those areas require separate evaluations with their own datasets and acceptance thresholds.

livenerf Maintenance, License, and Upgrade Constraints

The repository is not archived, and its last push was on 2026-10-16. Progress reported on that date states that 21 of 30 daily runs had completed with none missed. The baseline and first comparison window were complete, while window 2 had started. The README also records operational deviations rather than hiding them: days 5 and 12 used a budget-guard override, day 12 had an out-of-memory attempt that was excluded in full, and day 19 used Anthropic API credits instead of the Max plan.

The project uses the MIT license. That identifier permits use and modification under the license's terms, but repository licensing does not grant access to Anthropic models, Claude subscription capacity, or third-party benchmark datasets. An organization must separately account for model access, data licensing, and any requirements attached to external evaluation material. This is a technical and operational consideration, not legal advice.

There are no GitHub releases, so versioned upgrade packages are not documented in the repository information. The project version in `pyproject.toml` is `0.1.0`, and the executable dependencies are pinned or bounded. Because the benchmark depends on a fixed CLI, harness hash, graders, and question panel, adopting a later commit is not automatically a routine dependency upgrade. Any code change must be checked against `PREREGISTRATION.md`, `PLAN.md`, the deviations record, and the calibration documentation before mixing runs into the same series.

Editorial conclusion

Adopt livenerf if you need a reproducible record of Claude Opus 5.5 performance over a defined 30-day period and can retain raw logs and study serving-path deviations. Do not treat it as a general model leaderboard or as sufficient evidence against a same-family model swap. Before adopting, verify access to Claude Max or API credits, review the locked question panel and graders in `docs/`, and confirm that the statistical power is appropriate for the degradation you need to detect. On the documented limits, Claude Opus 5 substitution was not distinguishable from Opus 5.5 in validation.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. ninjahawk/livenerf on GitHub
4. README