judge-server: the sandboxed grading backend behind the DMOJ online judge

The four kinds of problem the judge can grade

The judge is described as the contest judge backend for the DMOJ site interface, and it implements secure grading on Linux and FreeBSD machines. The feature that matters most is the set of problem types it understands.

An IO-based task is the ordinary one, where the program reads input and writes output and the two are compared. An interactive task is where the judge speaks to the submitted program over a channel, which is how problem types that need to interrogate the program are handled. A signature-graded task is where the program is not compared against an expected output at all but against a checker, which is how construction and interactive-style problems are judged. Alongside those, the judge supports runtime data generators, so the inputs are produced rather than stored, and custom output validators, so a submission can be checked by a program of your own rather than by a diff.

Together those five capabilities are what separate a real competitive programming platform from a script that compiles and runs a test case. The interesting engineering is not in running the compiler, it is in the containment around it, because the code being run is submitted by strangers.

The runtime coverage is broad. The languages listed as supported include C++ 11 through 23 with both GCC and Clang, C 99 and 11, Java 8 and 25, Python 2 and 3, PyPy, Pascal and Mono C#, F# and VB. Beyond that, the judge can also grade a long tail including Ada, Algol 68, AWK, COBOL, D, Dart, Forth, Fortran, Go, Groovy, Haskell, INTERCAL, JavaScript through Node.js and V8, Kotlin, Lean 4, LLVM IR, Lua, NASM, Objective-C, OCaml, Perl, PHP, Pike, Prolog, Racket, Ruby, Rust, Scala 3, Scheme, sed, Common Lisp, Swift, Tcl, Turing and Zig.

Linux is the supported target, and the matrix says so

The platform support table is short and honest, and it is worth reading as the first filter for your own deployment.

On Linux, all four architectures are covered: x64, x86, x32 and ARM. On FreeBSD, only x64 is supported, with x86, x32 and ARM all marked as not supported. Windows was supported up to and including version 1.4.0 and is no longer part of the picture, and versions up to and including 3.0.2 also supported grading with pure ptrace without seccomp, which the documentation notes is useful on Linux kernel versions before 4.8. So on an old kernel you have a fallback, and on anything current you have the seccomp sandbox.

One reassurance is worth repeating because it is unusual for a judging system: the DMOJ judge does not need a root user on Linux, and it runs fine under a normal user. A judge that executes untrusted code is normally a root-privileged component, so the fact that this one is not is a meaningful design property, and it makes the operational story much easier for an institution that does not want to hand root to a sandbox.

The distribution story is where the caveats live. The documentation states that the only Linux distribution with first-class support is the latest Debian, with the default package versions of all runtimes, and that this is what the project runs on dmoj.ca. The judge will likely still work elsewhere, but some runtimes may fail to initialise, and the documented remedy is to file an issue rather than to work around it quietly.

Two ways to install, and the difference between them

Installation produces two executables in your Python script directory. The first, dmoj, connects a judge to a DMOJ site instance, and the second, dmoj-cli, gives a command-line interface to a local judge, which the documentation describes as useful for testing problems without a site in the loop.

The easy route is the package on PyPI:

$ pip install dmoj

The documentation is candid about the trade. Builds are published periodically, so this is the easiest way to get going, but it may not contain all the latest features and improvements.

The alternative is the version that runs live on dmoj.ca:

$ git clone --recursive https://github.com/DMOJ/judge-server.git
$ cd judge-server
$ pip install -e .

The recursive flag matters, because the repository has a .gitmodules file and the sandbox has components that live in submodules. The editable install means the checkout is what runs, which is what you want when you are tracking the live behaviour.

The build itself is not pure Python. The project uses Cython, and the packaging configuration requires setuptools, wheel and Cython as build requirements, with black configured at a line length of 120 and string normalisation disabled, and the testsuite excluded from formatting. The runtime requirements are short:

cython
pylru
pygments
pyyaml
setproctitle
termcolor
watchdog

The setproctitle entry exists so the judge can label its processes, and the watchdog entry backs the mechanism that watches problem directories, which the command line can switch off with the no-watchdog option.

The build refuses Windows, and the sandbox compiles C

The packaging script is where two facts about the project become concrete. The first is that Windows is not supported, and this is enforced rather than documented. The setup script checks the operating system name and, on Windows, prints a message that DMOJ is unsupported on Windows and exits.

The second is that the judge compiles a native component. The script looks for the Cython source of the sandbox, checking for its presence in the judge package directory, and refuses to continue without Cython installed if that file exists, telling you to install Cython with pip. It also reads an environment variable, DMOJ_PARALLEL, to decide how many compilation jobs to run in parallel, falling back to the CPU count when the variable is absent or set to zero.

There is a second environment variable, DMOJ_TARGET_ARCH, and it controls the architecture the sandbox is compiled for through the compiler's architecture flag. The default is usually native, but the documentation notes it will not be specified on ARM unless the variable is set, in which case a generic and slower build is compiled instead. On a Raspberry Pi or an ARM server that is the difference between a judge that starts quickly and one that spends a long time in the compiler.

The formatting and tooling configuration in the repository gives a sense of the project's engineering habits: a clang-format file, a flake8 configuration, a coveragerc and a codecov configuration, a mypy configuration, a.coveragerc for coverage, and separate test scripts for Docker and FreeBSD. The coverage badge and the codecov configuration in particular suggest that the sandbox is treated as the part that must not break.

Running a judge and connecting it to a site

The command line for the server shows a small surface, and it is the same surface the configuration file is expected to hold.

$ dmoj --help
usage: dmoj [-h] [-p SERVER_PORT] -c CONFIG [-l LOG_FILE] [--no-watchdog]
            [-a API_PORT] [-A API_HOST] [-s] [-k] [-T TRUSTED_CERTIFICATES]
            [-e ONLY_EXECUTORS | -x EXCLUDE_EXECUTORS] [--no-ansi]
            server_host [judge_name] [judge_key]

The positional arguments are the host to connect to, the judge name and the judge key, and both of the last two override whatever the configuration file says. The configuration file is mandatory, given with the config option, and the log file option names where output goes. The watchdog option turns off the directory watcher, and the API port and API host options open a separate interface for the site to query. The trusted certificates option is how the judge is told which client certificates to accept, which matters when the connection is over TLS.

Two exclusion options are worth noting because they are how you narrow a judge's scope: one keeps only the listed executors and the other skips the listed ones. A judge that can run every runtime in the matrix is a judge that has every runtime's security exposure, so restricting the set to what a site actually offers is a sensible hardening step.

The no-ansi option turns off the escape sequences in the output, which is what you want when the judge runs under a log collector.

Docker images, three tiers and weekly rebuilds

The project maintains Docker images with all the runtimes it supports, in a separate repository called runtimes-docker, and the images are rebuilt and tested every week to pick up the latest runtime versions.

The runtimes are split into three tiers of decreasing support. Tier 1 holds Python 2 and 3, C and C++ with GCC only, Java 8 and Pascal. Tier 3 contains all the runtimes the project itself runs on dmoj.ca. Tier 2 is an in-between mix, and the documentation tells you to read the Dockerfile of each tier for the details. That is a sensible arrangement, because the image you want to run in production is the one that matches what the reference site runs, and the cheapest image to start with is the small one.

The example in the documentation spawns a tier 1 judge image, and it comes with a list of assumptions that are worth reading as a checklist. The relevant environment variables must be set, the network device has to be named enp1s0, problems are placed under the problems directory on the host, and the judge-specific configuration has to live in a judge.yml file in that same place, with the runtime configuration already merged into it automatically.

The container is given the capability to trace, which is what lets the judge implement interactive problems, and the run command takes the bridge address, the judge name and the judge key. The whole sequence is a lot of moving parts for a first attempt, and the sensible order is to run a local judge through dmoj-cli against a single problem before pointing a container at a live site.

AGPL, and a release that took three years

Two things to check before you build on this.

The licence is AGPL-3.0, and that is a deliberate choice for a component that runs inside a service. The AGPL extends the copyleft obligation to network use, so if you run a judge as a service for other people, you are obliged to offer them the corresponding source of your modified judge. If you are running it only for your own students on an internal network, the practical burden is smaller, but the obligation is real and the choice of licence is not accidental. Read the LICENSE file before you decide how to deploy it.

The release history is the second thing. v5.0.0 was published on 2026-06-14, before that v4.1.0 came out on 2023-02-20 and v4.0.1 on 2023-01-28. So the major version moved recently, and there is a long gap between the 4.x line and the 5.0 release, which usually means the 5.0 boundary carries changes worth reading about in the changelog rather than discovering.

The repository itself is not archived, and the last push was on 2026-10-12, so the tree is actively worked on. The reference site is dmoj.ca, the documentation is at docs.dmoj.ca with a section on setting up a judge, and the project runs a Discord community and a Jenkins instance for its FreeBSD build alongside the GitHub Actions workflow for Linux.

Editorial conclusion

judge-server is the right component to reach for if you want to run an online judge of your own and care about interactive or signature graded problems, since those are where a home grown grader usually falls apart. It is a poor fit if you need it to run on Windows, on FreeBSD on anything but x64, or on a distribution other than the latest Debian, because that is where the runtime matrix is thinnest. Before you plan an installation, read the judge setup documentation, check that your kernel supports the sandbox the judge expects, and decide whether the stable package from PyPI is recent enough for your needs or whether you need the live version from the repository.

Frequently asked questions

Official sources

1. DMOJ/judge-server on GitHub
2. License: AGPL-3.0
3. Project website
4. README
5. Releases