Web Codegen Scorer: Evaluating LLM-Generated Web Code with Build, Runtime and Accessibility Checks

The gap Web Codegen Scorer targets: web code quality, not general coding ability

Most public LLM coding benchmarks ask a model to solve algorithmic puzzles or patch isolated functions. Web Codegen Scorer starts from a different premise: the thing you want to measure is whether the model can produce a web application that builds, runs, and passes checks a web engineer would recognise. The README states the tool "focuses specifically on _web_ code and relies primarily on well-established measures of code quality," which is the clearest statement of intent in the repository.

The audience is narrow and identifiable. The README lists three use cases: iterating on a system prompt to find effective instructions for your project, comparing code quality across models, and monitoring generated code quality over time as models and agents evolve. Those are tasks performed by a platform or developer-experience engineer who owns an internal codegen workflow, not by someone shopping for a chatbot. If your question is "which model writes better React components for our design system," this tool is shaped for you. If your question is "which model is smarter," it is not.

How an eval is structured: environment config, runner, checks, repair loop, report

The unit of work is an environment, loaded from a path passed to `--env`. The README points to a separate document, `docs/environment-reference.md`, for the full schema, so the exact config keys are not visible in the README itself. What is visible is the shape of a run: a set of application prompts, a model to generate code, a runner to execute the eval, and a set of checks applied to the result.

The runner is selectable with `--runner=<name>`, and the supported values are `ai-sdk` (the default), `gemini-cli`, `claude-code` and `codex`. That is a meaningful design choice. Rather than only calling a model API directly, the tool can drive agentic CLI harnesses, which means the code under evaluation may have been produced through tool calls, file edits and retries rather than a single completion. The README does not describe how each runner differs in behaviour beyond the names, and that is a gap worth noting.

Checks are the scoring layer. The README lists built-in checks for build success, runtime errors, accessibility, security, LLM rating (via `--autorater-model`) and coding best practices, with the note that more are coming. There is also an automatic repair path: `--max-build-repair-attempts` defaults to `1`, meaning a failed build triggers one repair attempt before the result is recorded. That repair loop is the part most likely to confuse a first-time user, because a report can reflect a build that only succeeded after the model was handed its own error output. The `--local` flag exists precisely so you can re-run assessments or debug the build and repair process without paying for the initial generation again.

Getting it running: install, API keys, and the first eval

Setup is three steps in the README. Install globally with `npm install -g web-codegen-scorer`. Export API keys for the providers you intend to use: `GEMINI_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` or `XAI_API_KEY`. Then run the bundled Angular example:

web-codegen-scorer eval --env=angular-example

To build your own environment instead, `web-codegen-scorer init` walks you through the process interactively. Once an app has been evaluated, `web-codegen-scorer run --env=angular-example --prompt=<name of the prompt you want to run>` starts that generated app locally so you can inspect it by hand.

The flags that matter most in day-to-day use are `--model`, `--autorater-model`, `--limit` (default `5` prompts), `--concurrency` (default `5`, defined as `DEFAULT_CONCURRENCY` in `src/config.ts`), `--output-directory` for keeping generated code instead of writing to a temp directory, `--report-name` for naming the report directory instead of using a timestamp, `--prompt-filter` for debugging a single prompt, `--labels` for attaching metadata to a run, and `--skip-screenshots` for turning off app screenshots. There is also `--rag-endpoint=<url>`, which requires the URL to contain a `PROMPT` substring that gets substituted with the user prompt, and `--mcp`, which starts an MCP server for the evaluation. The README does not explain what the MCP server exposes, so treat that flag as undocumented beyond its existence.

The --local flag is the most useful and least obvious part of the workflow

Running evals costs money and time, and the initial code generation is the expensive half. The `--local` flag changes that: instead of calling the LLM for the initial generation request, it reads the code from `.web-codegen-scorer/llm-output` (the README gives `.web-codegen-scorer/llm-output/todo-app.ts` as an example path). The README is explicit that you typically need one non-local run first to populate that directory, and that `web-codegen-scorer eval:local` is a shortcut for `web-codegen-scorer eval --local`.

This matters because it separates two questions that are otherwise entangled: did the model produce good code, and does my check configuration behave the way I expect? With `--local` you can change checks, repair-attempt counts or report naming and re-run against a frozen set of model outputs. Without it, every configuration tweak costs another round of API calls and introduces model nondeterminism into the comparison. Anyone evaluating this tool should test the local path early, because if the cached output format does not match what your environment expects, the whole iteration loop becomes expensive again.

Where it does not fit: no released versions, thin docs, and a hard web focus

The repository has no retrieved releases, so there is no versioned changelog to pin against and no compatibility matrix for the environment config schema. The README defers the schema to `docs/environment-reference.md` and model setup to `docs/model-setup.md`; if those documents are as terse as the README's pointer suggests, expect to read the source for anything beyond the examples. The last push recorded is 2026-05-05, so the project is active, but activity is not the same as a stable interface.

The web focus is a real boundary, not a marketing line. A team generating backend services, data pipelines or CLI tools will find the built-in checks (build success, runtime errors, accessibility, security, LLM rating, best practices) oriented toward browser-delivered applications. The accessibility and screenshot checks in particular have no meaning for a server process. Second, the autorater is itself an LLM, configured with `--autorater-model`, and the README does not describe how its ratings are calibrated or whether they are stable across runs. Treat the LLM rating as one signal among several, and prefer the deterministic checks (build, runtime) when you need a number you can defend.

Third, the repair loop changes what is being measured. A model that fails a build but succeeds after one repair attempt scores differently from one that builds cleanly on the first try, yet both may appear as passing runs unless you inspect the report. The README does not state whether repair attempts are surfaced as a distinct metric.

Alternatives and the actual difference in approach

The obvious comparison is a general coding benchmark suite, which typically ships a fixed set of tasks and a fixed scoring script so that results are comparable between anyone who runs it. Web Codegen Scorer inverts that: the environment is yours, the prompts are yours, the model and autorater are yours. The trade is comparability for relevance. You cannot meaningfully compare your report to another team's report, but you can compare two models against the prompts your team actually writes.

Within the LLM evaluation space, the other common approach is a hosted observability platform that traces prompts and responses from production traffic and lets you attach custom scorers. That gives you real user prompts and longitudinal data without writing a harness, but it scores whatever your users happened to ask, and it rarely runs the generated code through a build, a browser and an accessibility audit. Web Codegen Scorer sits at the opposite end: a controlled, offline, repeatable run that executes the artifact. If you need production-traffic evaluation, a tracing platform is the better fit. If you need to know whether a model can produce an app that compiles and passes an accessibility check, this tool is doing something the tracing platforms generally are not.

Maintenance cost, licensing, and what the repository commits to

The licence is MIT, which permits commercial and internal use, modification and redistribution with the licence and copyright notice retained. That is a permissive baseline, and nothing in the README suggests additional terms, telemetry restrictions or a separate commercial tier. This is not legal advice; if you are embedding the tool in a product or a regulated pipeline, have counsel read the LICENSE file in the repository rather than this summary.

The maintenance cost sits in the environment config, not the install. Because the config schema lives in a separate reference document and the tool has no retrieved releases, upgrades may require re-reading that document rather than following a migration guide. The `--local` cache under `.web-codegen-scorer/llm-output` is another artifact you will need to keep in sync with your prompt set; the README implies a one-to-one mapping between prompts and cached files, so adding a prompt means another non-local run before local iteration is available for it. For local development on the tool itself, the README specifies `pnpm install`, then `pnpm run release-build` for a `dist` build, `pnpm run npm-publish` to build and publish, and `pnpm run eval` to run an eval from source. Budget for reading source rather than documentation when you hit the edges of the config schema.

Editorial conclusion

Adopt Web Codegen Scorer if you are choosing between LLMs for a web codebase, tuning a system prompt, or tracking generated-code quality across model releases, and you are willing to write an environment config (mjs) and supply provider API keys. Do not adopt it if you need a hosted dashboard, a score that is comparable across teams, or an evaluation of non-web code. Before relying on any number it produces, run `web-codegen-scorer eval --env=angular-example` once, then re-run it with `--local` to confirm the cached output in `.web-codegen-scorer/llm-output` reproduces the same checks, and verify which runner and autorater model your report actually used.

Official sources

1. angular/web-codegen-scorer on GitHub
2. Issues
3. License: MIT
4. README