Submitty: a course system that grades homework with containers and keeps its migrations on a leash

A course system from a university open source lab

Submitty is an open source course management, assignment submission, exam and grading system from the Rensselaer Center for Open Source Software, the Department of Computer Science at Rensselaer Polytechnic Institute. The project is hosted on GitHub under the Submitty organisation, and the README is short by design: system instructions live at submitty.org, and the key features are described on a features page there.

That framing matters for an evaluator. Submitty is not a library you drop into a course site, it is a system that a department installs and operates, and the documentation is structured for the people who run it rather than for developers browsing the repository.

The development tooling tells you the scale of the operation. There is a GitHub Actions workflow for continuous integration and a separate one for a Vagrant up job, a codecov configuration, a yamllint configuration, a shellcheck ignore file with a run_shellcheck.py script beside it, and a Vagrantfile at the root with a vagrant-workers/ directory. The project supports the hacktoberfest topic tag, which is a hint about how contributions arrive.

The contact route is Zulip, and there are separate documents for how to make a pull request and how to review one, both hosted on the project website rather than in the repository. Support.md, SECURITY.md, AUTHORS.md and CONTRIBUTING.md are in the tree.

The primary language is PHP, with a substantial Python component, and the releases follow a calendar scheme: v26.10.04 on 2026-10-19, v26.10.03 on 2026-10-15 and v26.10.02 on 2026-10-12, with the last push to the repository on 2026-10-20.

The directory layout is the architecture

With a README this short, the tree at the root of the repository is where the design lives, and it divides into a web application and a grading stack.

The site/ directory is the PHP application, and it is where a submitty_logo.png lives under site/public/img/, so the branding assets sit with the front end. The grading side is spread across autograder/, grading/, grading_bin/ and junit_test_runner/, with more_autograding_examples/ for the example configurations and python_submitty_utils/ for shared Python helpers. sample_files/ holds fixtures, and sbin/ and bin/ hold administrative and command line scripts.

The separation of grading/ from autograder/ is the interesting part. One is the code that decides what to run and how to judge it, the other is the machinery that executes and collects results, and junit_test_runner/ is the component that turns those results into a format CI systems already know how to display. Splitting those concerns is what allows the same grader to be driven by a local run, by a course build server, or by a continuous integration job.

migration/ is the third pillar and it is substantial enough to have its own structure, with a migrator and a course/ migrations directory holding dated migration files from 2018 onwards. .setup/ holds the installer and maintenance scripts, and the names in that directory say a lot: CONFIGURE_SUBMITTY.py, update_database.py, reset_system.py, track_git_version.py, generate_repos.py, input_forum_data.py and reupload_old_assignments.py.

A repository that ships a reset script and a re-upload script for old assignments is a repository written by people who have had to recover a production system more than once.

Ruff is configured to leave the old migrations alone

The pyproject.toml at the root is mostly a Ruff configuration, and the first thing it says is the shape of the Python code in the project:

[tool.ruff]
line-length = 100
target-version = "py310"

A line length of 100 and a py310 target tell you this is code written against a specific modern Python baseline, and the file then goes on to exclude a long list of files from formatting.

The list is where the interesting decision is. It covers administrative scripts such as .setup/CONFIGURE_SUBMITTY.py, .setup/bin/code_watcher.py, .setup/bin/reset_system.py, .setup/bin/generate_repos.py and .setup/update_database.py, and a set of command line tools including bin/regrade.py, bin/extract_notes_page.py, bin/make_assignments_txt_file.py and bin/left_right_parse.py. The migration files are excluded too, and the comment explains why: existing migrations are pinned to their historical formatting so this list does not retroactively rewrite them, and new migrations must be ruff-formatted.

That is a policy about history, and it is the right one for a database migration system. A migration file is a record of what a database looked like at a point in time, and reformatting it changes a file that has already run in production for years. Pinning the old ones and requiring the new ones to be formatted means the repository stays consistent going forward without rewriting the past.

The same reasoning appears in the regrade.py script and in the migration directory names: a regrade and a schema change are both operations you cannot redo, and the tooling is arranged so that the risky parts are append only.

Contribution and review are documented as processes, not as good intentions

The contributing section of the README is three lines long and points at two documents on the project website: how to make a pull request and how to review a pull request. That is a deliberate choice, and it is a reasonable one for a project with this many contributors and this much code.

The pattern shows in the repository as well. There is a CONTRIBUTING.md file, a code of conduct, a security policy, a support file and an AUTHORS.md, and the continuous integration is split into two workflows, one for the general CI and one specifically for bringing up the Vagrant environment, which is the way a new contributor gets a working instance of the system to look at.

The linters tell the same story about how the project is policed. Alongside the Ruff configuration there is a yamllint configuration for the workflow and configuration files, and a shellcheck ignore file with a Python script to run it, which is the way the shell scripts in bin/ and sbin/ are kept consistent. A codebase with this many languages needs that kind of per-language tooling, and having it configured at the root rather than per directory suggests it is enforced rather than optional.

The codecov configuration at the root is the other half of the picture. For a project whose whole point is grading, the project itself wants coverage numbers on its own code, and the same test suites that run in CI produce the report.

Licensing is split into the project and its third party components

Submitty is licensed under the BSD 3-Clause licence, which can be viewed in the LICENSE.md file, and the licences for the third party components are in LICENSE-THIRD-PARTY.md.

That second file is the one to look at in a project of this shape. A system that renders a web interface, grades submitted code, talks to a database and runs student programs pulls in a long list of libraries, and the convention of shipping a separate third party licence inventory is how a deployment can satisfy its own obligations without reading every dependency's licence.

The BSD 3-Clause terms themselves are among the most permissive in common use, with a patent grant and a requirement to keep the copyright notice, and they are the same terms the project has used since it was published.

The release numbering is worth noting separately, because it is a schedule rather than a version series. The tags run 26.10.04, 26.10.03 and 26.10.02 within the same month, which is a year and month identifier with a build counter. For an operator that means an upgrade is a dated package rather than a semver decision, and for a deployment the upgrade cadence is a monthly one.

The last push to the repository was on 2026-10-20 and the repository is not archived, so the project is being worked on at the time of this review.

Where Submitty is the wrong tool

The first wrong fit is a single course with a handful of assignments. Submitty is a departmental system with a server side stack, a Vagrant based development environment, a migration subsystem and a grading daemon. If you want students to drop files in a folder and read comments, a lighter submission tool is hours of work rather than months, and the difference is entirely in the operational surface.

The second is a course that grades by hand. The system's value is autograding, and its README lists the grading, autograding and exam features as the core of the product. A course where a human reads every submission is paying for the infrastructure without using the part that makes it worth the infrastructure.

The third is a deployment team that cannot run the whole stack. The grading side needs to execute untrusted student code, which is the hard part of running this system safely, and the repository's own development path goes through a Vagrant environment built for exactly that. An institution that cannot isolate that execution is not a candidate for Submitty regardless of how well the grading works.

The fourth is an instructor who wants their own workflow to survive. Submitty is an open source system that a department configures, and the course data lives in its own database and its own directories. Migrating a course out is work, and the README does not describe a data portability story, so it is worth asking about one before a large deployment.

The fifth is a project looking for a reusable PHP component. What is here is a whole course system, not a component: the site/ directory is an application with its own branding, its own configuration and its own migrations. The one thing to take from the repository is the discipline around migrations and linters, which is visible in the pyproject.toml exclusions.

Editorial conclusion

Submitty fits a course or department that wants students to submit code and have it graded automatically and consistently, with a system that already runs at scale at a large university. It does not fit a single instructor who wants a lighter submission tool, and it does not fit a deployment without the ability to run the server side stack. Before adopting it, read the deployment instructions on submitty.org, because the README itself defers all system instructions to the website and the repository is not a quick start.

Frequently asked questions

Official sources

1. License: BSD-3-Clause
2. Project website
3. README
4. Releases
5. Submitty/Submitty on GitHub