pyre-code is a local judge for writing PyTorch internals yourself

A judge that grades PyTorch code against test cases

pyre-code turns a specific kind of self-study into a graded exercise. The idea stated in the project is simple: you write the implementation, a local grading service runs the tests, and you see what broke. The subject matter is the internals of modern machine learning systems, with the project naming Transformers, vLLM, TRL and diffusion models as the territory, and the readme describing the problems as covering what is actually inside those systems rather than how to call them.

The audience is stated in three lines and each one is a different use case. Interview candidates practise implementing core components under test rather than reading about them. Learners who think by building are told this is their gym. And developers who have used a ready-made attention module are invited to write it themselves. The tagline captures the constraint: read the paper, then write the code, and no GPU is required.

The scale of the problem set is the other thing to know. The readme states 76 problems organised by category, running from fundamentals such as ReLU, Softmax, GELU and Kaiming initialisation, through normalisation, attention variants including GQA, sliding window, Flash and MLA, position encodings, architecture blocks, training tricks, distributed strategies such as tensor parallelism, FSDP and ring attention, inference topics like KV cache, beam search and paged attention, loss and alignment methods including DPO, GRPO and PPO, diffusion, LoRA and QLoRA, reasoning, state space models, and a graph neural network track. The project description and an April news entry say 68 problems, and a later entry adds 8 GNN problems, so 76 is the current figure and 68 was the earlier one.

Monaco in the browser, FastAPI behind it, torch_judge underneath

The stack is laid out as a table in the project documentation, and each layer has a clear job. The frontend is Next.js with the Monaco editor and Tailwind CSS, so the coding surface is a browser editor with Python syntax highlighting and no IDE setup. The backend is a FastAPI grading service. The judge engine is torch_judge, the project that executes and validates submissions, linked from this repository. Storage is SQLite, used for progress tracking.

The repository layout mirrors that division. The grading_service directory holds the FastAPI application, torch_judge sits in its own directory and is the package the setuptools configuration ships, web holds the Next.js frontend, and scripts holds the build helpers. There are two Dockerfiles, grading.Dockerfile and web.Dockerfile, and a docker-compose.yml that ties the two together.

The npm scripts show how the pieces are run in development. The dev script starts the backend and the frontend together under concurrently, with the backend running uvicorn against grading_service.main:app on port 8000 with reload enabled, and the frontend running inside the web directory. Two build scripts exist for the content, one that exports the problems and one that builds the solutions, and both are driven through a node script that shells out to Python, so a problem set can be regenerated without hand-editing files. A predev hook checks that torch_judge is importable before anything starts, which is the first thing that fails when the Python environment is wrong.

One script to set up, then two local ports

The prerequisites are Python 3.11 or later and Node.js 18 or later, and the recommended install is a clone followed by a setup script:

git clone https://github.com/whwangovo/pyre-code.git
cd pyre-code
./setup.sh
npm run dev

What the setup script does is worth knowing, because it is the part that usually goes wrong on a new machine. It creates a .venv Python environment, preferring uv and falling back to python -m venv, installs the dependencies, and prints the start command. On Windows there are equivalent scripts, setup.ps1 for PowerShell and setup.bat for the command prompt.

There is a detail in the Node scripts that saves a lot of confusion later. When .venv exists, npm run dev prefers that project-local Python automatically, and if it is missing the dev command falls back to the python on the current shell. In other words, an environment created by the setup script is picked up without you activating it, and an environment you forgot about is used instead, with a different set of packages.

Once it is running there are two addresses to keep straight. The grading service answers on http://localhost:8000 and the web app on http://localhost:3000. The manual install path exists for people who do not want the script, and it can be done with conda instead:

conda create -n pyre python=3.11 -y && conda activate pyre
pip install -e ".[dev]"
npm install
npm run dev   # run with conda env activated

Docker keeps the grader on an internal network

The compose file is the part of the project with the most explicit security posture. The grading service is built from grading.Dockerfile and is attached to a network declared internal, which means it has no route to the outside, and the web container is attached to both that internal network and the default one so that browsers can still reach it. The two containers are linked with a health check on the grading service at http://localhost:8000/health, with a ten second interval, a five second timeout, five retries and a fifteen second start period, and the web service waits for the grader to be healthy before it starts.

Only the web container publishes a port, and the environment variable that tells it where the grader lives is part of the compose definition:

  web:
    build:
      context: .
      dockerfile: web.Dockerfile
    ports:
      - "3000:3000"
    environment:
      - GRADING_SERVICE_URL=http://grading:8000

The compose file also caps the grading container at three CPUs and 4G of memory, which is a sensible limit for a process that runs student code, and progress is persisted in a named volume, so the two commands that matter are:

docker compose up --build
docker compose down -v

The first starts the stack, and the second tears it down and deletes the volume, which the documentation describes as the way to reset your progress.

AI Help stays switched off until you supply a key

The hints feature is optional and server-side configured, and the documentation is explicit that it is off unless you turn it on. To enable it you copy web/.env.example to web/.env and fill in three values:

AI_HELP_BASE_URL=https://api.openai.com/v1
AI_HELP_API_KEY=sk-...
AI_HELP_MODEL=gpt-4o-mini

The base URL is the point of flexibility. Any OpenAI-compatible endpoint works, and the documentation names OpenAI, Anthropic through a proxy, and Ollama as examples, so the same code can talk to a local model or a hosted one. Users can also configure their own key in the interface when no server-side configuration exists, which matters for a shared instance.

The rest of the platform is local. The feature list says nothing leaves the machine unless you opt into AI Help, and the Docker arrangement backs that up with the internal network, so the grading service has no outbound path. That is a meaningful property for a platform that runs code submitted by a user.

The other features are smaller and worth listing because they shape the practice loop. There is a browser editor with Python syntax highlighting, per test case pass and fail feedback within seconds, reference solutions to compare after your own attempt, progress tracking with a solved count and an attempt history persisted across sessions, and print output capture so that print statements from your solution show up in the test results instead of disappearing.

What a grading service can and cannot check

The limits of the approach are worth stating before anyone builds a curriculum on it. A judge can check that your attention implementation returns the right numbers, that shapes match, that a loss decreases. It cannot check that your implementation is the one described in the paper, that your training loop would converge on a real workload, or that your design choices are sensible. A problem set that asks you to write Flash attention is checking a contract, not a method.

There is a second limit that is about the content rather than the machinery. The reference solutions and the test cases are written by the project, and the project has no GitHub releases, so there is no published version for the problem set that you can pin. The readme itself is inconsistent about the count, with 76 in the problem set table and 68 in the news entry and in the project description, and the reconciliation is only visible by reading the dates. A curriculum built on this should expect to review the problems and the reference solutions before trusting them, and to expect occasional corrections.

Licensing is a third point. The repository licence is one that GitHub cannot classify, and it says to see the LICENSE file, while the npm manifest and the Python project metadata both declare MIT. Those two statements are not the same thing, and the file in the repository is the one to read before redistributing the code or the problem set.

Where a notebook repository fits better

The alternative approach is the one most people already use: a repository of notebooks, or a folder of exercise files, where you write the implementation and check it against a reference you read yourself. The difference in approach is the feedback loop. A judge runs your code against hidden test cases and tells you pass or fail per case, which is fast, unambiguous and repeatable, and it means nobody has to review your work. A notebook repository gives you a narrative, a worked solution to compare against and a place to experiment, and its feedback is your own reading of the output.

The consequences follow from that. A judge is better for interview preparation, because interviews test correctness under constraints and a test case is the same kind of judgement. A notebook repository is better for understanding, because the explanation next to the code is usually where the learning happens, and pyre-code only gives you the explanation after you have already attempted the problem. The platform does have reference solutions, but comparing them after an attempt is a different activity from being walked through one.

The other difference is operational. pyre-code needs a Python environment with torch installed, a Node toolchain, and a grading service that executes untrusted code on your machine. A notebook repository needs the same Python environment and nothing else, because the checking happens in your own process. For a single learner on a laptop that is one extra service, and for a class or a team it is the difference between a self-contained exercise set and a service that has to be hosted, health-checked and kept alive.

Editorial conclusion

pyre-code fits a learner or interview candidate who wants to write attention, normalisation and alignment components under test without a GPU, and who is willing to run a grading service on their own machine. It does not fit a curriculum that needs a maintained reference implementation, and it does not fit a team that cannot host a Python service with torch installed. Verify first that the problem count and the reference solutions match what you intend to teach, that the grading service health check passes on your machine, and that the licence file in the repository agrees with the MIT identifier in the package metadata before you redistribute it.

Frequently asked questions

Official sources

1. Issues
2. README
3. whwangovo/pyre-code on GitHub