cmdchallenge: running a command line in a container and grading the answer

Every command runs in a container started on demand

CMD Challenge is a site where a task is posed and the answer is a command, and the site checks whether the output matches. The repository holds the code for cmdchallenge.com, and the homepage is about.cmdchallenge.com, with the primary language of the repository listed as Go.

The execution model is the interesting part. Two runner images exist, cmd and cmd-no-bin, and the README says they are runner images launched on demand by the Go server. The Compose file puts both under a profile called runner-images, so a normal docker-compose up starts only the web server and Caddy, and the runner images have to be built explicitly because they are not long-running services.

That design is what makes the site safe to use. Each submitted command gets a fresh container rather than sharing a long-lived shell, so a command that fills the disk or hangs cannot poison the next user. The trade-off is startup cost per request, and the caching layer in the database is what pays for it.

The two images differ in what they contain. The cmd image is the standard one and includes manual pages for its installed utilities, so a user can run man grep, man ls or man bash, with the interactive pager disabled so the manual renders in the browser output instead. The cmd-no-bin image is the stripped down one used by the Oops challenges and does not include man at all.

The Compose stack also shows the security posture directly. The runcmd service mounts the Docker socket at /var/run/docker.sock, which is how it can start containers at all, and it sets CMD_SET_RATE_LIMIT=true. The caddy service sits in front on port 8100 and the Go server listens on 8181.

The dist volume is why a fresh checkout sometimes shows the old frontend

There is a bug-shaped behaviour in this project that the README spends a paragraph explaining, and the explanation is more useful than the fix.

After switching commits or branches, including checking out an older version, docker-compose up can still serve the frontend from the previous version. The reasons are three. Existing images are reused unless you explicitly rebuild them. The named dist volume is shared by the Go server and Caddy at /app/dist, and it persists between runs. And because that volume hides the frontend files in the image, even docker-compose up --build does not refresh an existing volume.

The fix is to remove the volume, rebuild the images and start the web services:

docker-compose down -v
docker-compose build runcmd cmd cmd-no-bin
docker-compose up runcmd caddy --remove-orphans

The README also corrects a common misunderstanding about the flags. down -v deletes Compose-managed volumes, and this configuration declares only the dist frontend volume, which Docker then repopulates from the rebuilt image on startup. The up -V flag only renews anonymous volumes and does not reset the named dist volume.

If the browser still shows the previous interface after that, the answer is a hard refresh of the page. And the same reset is needed when switching back to a newer version, which is the detail that catches people who assume the one reset covers both directions.

This is a genuinely instructive piece of Docker behaviour. A named volume mounted over a path in an image is not a cache, it is a permanent override, and the only reliable way to see the image contents again is to delete it.

Grading is a cached result keyed by the command output

The API for talking to the service is small, and the README gives two curl examples that show the shape of the grading contract.

Testing a single command is a POST to a /c/r path with a slug and a cmd field:

curl  http://localhost:8181/c/r -X POST -F slug=hello_world -F cmd="echo hello world"

And fetching solutions is a GET on a /c/s path with a slug query parameter:

curl http://localhost:8181/c/s?slug=hello_world

The first returns the result of running the command and comparing it, and the second returns the stored solutions for a challenge, which is the reference output set the grading is done against. The challenges themselves are described in a challenges.yaml file at the repository root, so the task definitions and the reference answers live in the repository rather than in the database.

The result cache is what makes repeated commands cheap, and it is also the source of a maintenance wrinkle the README describes at length. After upgrading an older installation, you have to rebuild the cmd image, because existing cached responses can still contain Ubuntu's minimized-system notice, the banner that a system running in a container prints.

The repair is a one-time migration run on the host holding the SQLite database:

bash bin/refresh-man-cache /var/opt/cmdchallenge/db.sqlite3

The script needs Python 3 and write permission on the database. It lists the matching commands, challenge slugs, versions and submission counts, then asks you to type delete to confirm, and any other answer or end of file cancels. The preview and the deletion share a transaction that blocks other writers while it waits for your answer, so the README says to run it during a brief maintenance window with the backend stopped.

The SQL behind it is in migrations/restore-man-pages.sql, and it only deletes unsuccessful results whose entire output matches the minimized-system notice. Other cached results and successful solutions are preserved and rerunning the script is safe. The last line of the section is a trap worth noting: resetting the frontend dist volume does not refresh the command-result cache, because the two are unrelated stores.

The frontend is plain DOM and ES modules, not a framework

The site frontend is described as using native DOM APIs and ES modules. There is no component framework, and the file names describe the split: main.js loads styles and starts app.js, which coordinates challenge state, navigation and requests, while command-input.js handles the command form with history, completion, keyboard handling and busy state. ui.js renders challenge content and feedback, and api.js, storage.js and router.js handle the browser interfaces.

The keyboard behaviour is specified in enough detail to be useful, since this is a terminal in a browser and the conventions are the point. Enter runs a command, Shift+Enter inserts a new line, Up and Down recall history, and Tab completes a word, with a double Tab displaying matching candidates. Shift+Tab leaves the command field, and Tab on an empty field moves to the Run button. Existing correct_answers progress and jQuery Terminal history remain available, which is a note about what was kept from an earlier implementation.

There is also a small set of readline-like shortcuts. Typing clear, or pressing Ctrl+L, clears the command field and the displayed output locally. Ctrl+A and Ctrl+E move to the start and end of the current line, and Ctrl+C cancels the current input when no text is selected, copying selected text normally. These shortcuts apply only while the command field has focus, which is the detail that keeps them from fighting with the rest of the page.

The build script is build.js, and the README says it bundles browser JavaScript and CSS with esbuild, compiles Sass, copies public/ and writes dist/index.html with content-hashed asset URLs, and that production builds include precompressed gzip files for Caddy. Watch builds include source maps and skip minification and compression, and you restart the watcher after changing build.js or installing dependencies.

The final image gives a user membership of the docker group

The Dockerfile is three stages, and the third one is where the operational detail lives.

The first stage builds the Go server from golang:1.26, with the target architecture written to a temporary file and passed to the build as GOARCH, producing a static binary with CGO disabled and the linker flag to strip symbols. The second stage builds the frontend from node:20.2.0-bullseye-slim with npm ci and npm run build. The third and final stage is debian:bookworm-slim, which copies the runcmd binary and the built dist directory out of the earlier stages.

The final stage then installs docker.io, creates a group with the fixed id 500, creates a user with the fixed id 510 added to that group, and gives it ownership of the app directory. The command at the end runs the server binary directly.

That group membership is the thing to be careful about, and it is a general Docker fact rather than a flaw in this project. Membership of the docker group is equivalent to root access on the host, because the Docker socket allows creating a container that mounts the host filesystem. The pattern here is deliberate, since the service needs to start runner containers on demand, but it means the container's user has a great deal of authority over the machine it runs on. Anyone deploying this on a shared host should treat the Docker socket mount as the primary risk rather than the challenges themselves.

The architecture is also parameterised, since the final stage uses --platform with the build argument, and the Compose file maps that argument to linux with a default of amd64, so an ARM machine such as an M1 Mac sets BUILD_ARCH=arm64 for both build and startup. The README stresses that the same architecture has to be used for both.

Editorial conclusion

cmdchallenge is worth reading if you are building anything that executes untrusted commands, because the architecture is a small, complete example of a server that starts a fresh container per request, grades the output, caches the result and serves a browser interface over it. It is the wrong tool if you want a shell emulator that runs in a page, since every command really executes on a server, and it is the wrong tool for anything where the container boundary is not enough. Before you deploy it, verify three things yourself: that the runcmd service can reach the Docker socket, since it mounts /var/run/docker.sock and that is what lets it launch the runner images on demand, that the rate limit is on, since the compose file sets CMD_SET_RATE_LIMIT=true and a public instance without it is an invitation, and that you understand the Docker group membership the image grants, since the final image adds a user to the docker group and that is effectively root on the host.

Frequently asked questions

Official sources

1. Issues
2. jarv/cmdchallenge on GitHub
3. License: MIT
4. Project website
5. README