lechmazur/writing: an LLM story benchmark scored by model judges

What the lechmazur/writing benchmark actually measures

This is not a writing tool. It is a measurement rig for comparing language models on short creative fiction. Each story must meaningfully incorporate ten required elements: character, object, concept, attribute, action, method, setting, timeframe, motivation and tone. The README describes the pipeline for building those briefs: candidate combinations are proposed for coherence and originality, then independently rated, and the strongest set for each seed becomes a fixed brief used by every writer model. A typical example given in the README combines a neutron-star researcher, butterfly-wing dust, gradual change, a storm-damaged greenhouse, "after the flood," and kindled humility. Both stories in every comparison answer the same brief, which holds prompt difficulty constant. The intended audience is anyone choosing between models for fiction, or tracking whether a new release actually writes better than the previous one. The scores are relative, not grades, and the README says zero is near the middle of the comparison set.

How the pairwise judging pipeline is assembled

The mechanism is comparative rather than absolute. Stories are generated in the benchmark format, matched comparison prompts are built for models that wrote to the same required elements, and each pair is shown in both story orders to reduce first- or second-position effects. Evaluator models then read matched pairs and choose which story is better. Those choices are combined into a relative comparison score, and the README states the rating combines compatible evaluator-v2 and evaluator-v3 evidence after bridge validation. Evaluators reward integration rather than keyword inclusion: the required object should affect the plot, the motivation should produce a consequential choice, and the tone should shape the story's development. They also weigh prose, coherence, character, originality and overall effectiveness. The current comparison set, as listed, covers 50 rated models, 773 direct model pairings and 79,507 evaluator judgments. The repository layout matches that description: data/, images/, prompts_wc/, reports/ and stories_wc/ sit alongside the README.

Installing nothing: reading the leaderboard and its diagnostics

There is no package to install and the README gives no installation steps, so the practical first use is reading the published artifacts. The leaderboard table ranks models by comparison score with an estimated win chance and an uncertainty range. Higher scores mean stronger performance against the other models tested, and overlapping uncertainty ranges can indicate similarly rated models. Two diagnostics deserve attention before you quote a rank. The evaluator agreement matrix shows how similarly the evaluator models scored the same story pairs, with values closer to 1 meaning stronger agreement and negative values meaning opposing scoring patterns. The word count chart plots each story as a dot and each model average as a diamond against a shaded 600-800-word target band, and the README notes it measures story length, not writing quality. The coverage note is the other thing to read: for example, the README states that Claude Opus 4.7 completed 347 of 400 stories and only completed stories were compared. If you want a machine-readable starting point, the repository ships a data/ directory and a release named benchmark-data-2026-08-23, so the scores travel with the repository rather than living only in a chart.

Where the benchmark stops being the right instrument

The most important limitation is stated in the README itself: because the combinations are pre-screened for creative potential, the benchmark measures story construction under deliberately combinable constraints, not completely free-form writing or recovery from arbitrary incoherent prompts. If your use case is a messy user prompt with contradictory requirements, this benchmark does not model it. A second limitation is the judge. Evaluator models decide the winner, and the agreement matrix exists precisely because those judges can disagree; a model that writes in a style the evaluators favor will rank above one that does not. Third, the score is relative. A model at -4.1 is not four points worse in any absolute sense; it is far down this particular comparison set, and adding or removing models shifts the scale. Fourth, coverage varies. Several leaderboard entries carry markers for completing fewer than 400 stories, and the README notes audited recovery passes for two of them, which means the comparison for those models rests on a slightly different story population.

Chatbot Arena and this benchmark differ in what they hold constant

The obvious alternative is Chatbot Arena, which also produces relative model rankings from pairwise human preference. The difference in approach is the control. Arena uses open prompts from real users, so the prompt distribution is uncontrolled and reflects whatever people happen to ask. This benchmark fixes the brief: every writer model answers the same ten required elements, and the README states this holds prompt difficulty constant. That makes the comparison cleaner for story construction but narrower, since the briefs are pre-screened for combinability. The judges differ too: Arena relies on human voters, while this benchmark uses evaluator models, which the README tracks with an agreement matrix and a bridge validation step between evaluator versions. If you want to know what people prefer when they ask anything, Arena answers that. If you want to know which model integrates a mandated object, motivation and tone more convincingly under identical constraints, this benchmark is the closer fit.

Maintenance, licensing and the cost of tracking new models

The repository is not archived and the last push was on 2026-09-05, which is recent. The most recent release listed is benchmark-data-2026-08-23, dated 2026-08-23, and the README describes a current comparison set of 50 rated models. The upgrade cost is therefore not code maintenance on your side; it is re-reading the leaderboard when the dataset release changes, because the rating combines evaluator-v2 and evaluator-v3 evidence and the README notes that bridge validation was needed to make those compatible. The chart focuses on selected current models while the table retains all rated models for historical comparison, so older numbers stay visible but are not the same thing as current standing. The repository does not state a licence in the files available, and the README does not document terms for reusing the stories or judgments. Treat the licence as unresolved and check the repository files before republishing the data or the images.

Editorial conclusion

Adopt this benchmark if you need a relative ranking of story-writing models on constrained briefs, and you accept evaluator models as the judges. Skip it if you need absolute quality grades, free-form prompt testing, or a benchmark you can run without the published data. Before relying on a number, check the coverage note for the model you care about: several entries completed fewer than 400 stories, and only completed stories were compared.

Frequently asked questions

Official sources

1. Issues
2. lechmazur/writing on GitHub
3. README
4. Releases