# Autolab is a course management service for autograded programming assignments

> Autolab is a Ruby on Rails course management service built at Carnegie Mellon University for auto-graded programming assignments, with gradebooks, handins, manual grading and a Docker based autograding pipeline. It has been open source since 2014, and its most recent release tag is v3.0.2 from 2024.

**autolab/Autolab** — Course management service that enables auto-graded programming assignments.

- Repository: https://github.com/autolab/Autolab
- Website: http://www.autolabproject.com/
- Stars: 808 · Forks: 234
- Language: Ruby
- License: Apache-2.0
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/autolab-autolab

## Programs evaluating programs, and the scoreboard on top

Autolab is a course management service, initially developed by a team of students at Carnegie Mellon University, that lets instructors offer autograded programming assignments over the web.

The README names two key ideas. The first is autograding, which it defines as programs evaluating other programs. The second is scoreboards. That combination is the shape of the tool: an assignment has a container that runs the student's submission against a test suite, and the results feed a leaderboard that students watch while the course is running.

Around that core sits the list of features an instructor expects from a course management system. Gradebooks, rosters, handins and handouts, lab writeups, code annotation, manual grading, late penalties, grace days, cheat checking, meetings, partners, and bulk emails. That list is the honest measure of the project's scope, and it is a large one for a teaching tool.

The history explains the design. Since 2010 the system has been used at CMU, and the README says each semester around 5,000 CMU students use it across courses in Pittsburgh, Silicon Valley, Qatar and Rwanda. The open sourcing came in 2014, with the stated goal of the same impact at other schools. The licence is Apache-2.0 and the homepage is autolabproject.com.

## A Docker based pipeline does the grading

The v3.0.0 changelog is where the autograding mechanism becomes visible. Improvements to autograding include direct Dockerfile uploading and the option to selectively disable network access, and the same release brought UI-based deployment configuration, a Simple File Manager for managing course files, and import and export of course data in one go.

The Dockerfile at the repository root is the server side, and it is built on the passenger-docker image, phusion/passenger-ruby32:2.6.1, which is where the Ruby version comes from. It installs sqlite3, tzdata and shared-mime-info, then starts Nginx and Passenger by removing the down marker from the nginx service directory and deleting the default site.

The build copies the Gemfile and Gemfile.lock separately before the application code, so that the gem installation layer is cached separately from the source layer, then creates /home/app/webapp, adds the application and precompiles the Rails assets with a dummy secret key:

```dockerfile
RUN mkdir /home/app/webapp
WORKDIR /home/app/webapp

# Add the rails app
ADD . /home/app/webapp
```

The final step starts the baseimage init system, and the container ends up serving the app through Nginx and Passenger rather than through a standalone Rails server.

## The v3.0 line moved configuration into the browser

The changelog for v3.0.0, dated 2024-07-24, lists UI-based deployment configuration, a Simple File Manager, export and import of courses, Tango, and grading improvements. The most consequential line for an administrator is that configuration of authentication, GitHub and SMTP moved to the UI.

That is a significant change for anyone who deployed an earlier version. Server side configuration files for the things an institution is most sensitive about, authentication, the GitHub integration and mail, are now edited through the application itself. The benefit is that a deployment is no longer a set of hand edited files, and the cost is that the first administrator to reach the interface is now configuring who can reach it.

The grading changes in the same release are worth noting separately: an improved Gradebook, a Code Diff Viewer and Starred Problems, plus direct Dockerfile uploading and selective network disabling for the autograding containers.

The tags tell the wider story. v3.0.0 was on 2024-07-25, v3.0.1 on 2024-10-25 and v3.0.2 on 2024-11-18. The repository is not archived and the last push was on 2026-10-21, so work continues on the branch well after the last tag.

## Rails 5 is a separate branch and is no longer supported

There is a short section in the README about Rails 5 support that says more about the project's priorities than most changelogs do. Autolab is now running on Rails 6, and the Rails 5 branch can be found on master-rails-5. The README states that no new features will be backported from master to master-rails-5, and that Rails 5 support has been discontinued.

The release history shows the same pattern in the dependency versions. The v2.12.0 entry records Ruby upgraded to 3.2.2 and Rails upgraded to 6.1.7.6.

So the supported stack is Rails 6 with a specific Ruby version, and the old branch is frozen rather than maintained. For an institution considering adoption, that means the upgrade path runs through a Rails 6 application with a Rails 6 compatible Ruby, not through the newest versions of either.

The docs are built with mkdocs and the mkdocs-material theme, which the README explains. The pipeline is that you install mkdocs, you install the theme package it relies on, and you preview with python3 -m mkdocs serve. Documentation changes merged into master are picked up automatically by a Jenkins job, and a manual deploy with python3 -m mkdocs gh-deploy builds the site from the current branch, places the HTML in the gh-pages branch and lets GitHub deploy it. The mkdocs.yml file is at the repository root.

## The test suite needs its own database and a setup task

The testing section is short and assumes a Rails developer who already knows the conventions.

You add a test database in database.yml, then create and migrate it with a project specific rake task:

```sh
RAILS_ENV=test bundle exec rails autolab:setup_test_env
```

The README emphasises that you must use RAILS_ENV=test bundle exec in front of every rake or rails command, which is a reasonable warning given how easily a test run ends up writing to the development database.

You also create the necessary directories, with mkdir tmp/ as the example given. Running the specs is then bundle exec rails spec. Individual spec files can be run on their own with rake spec SPEC=./spec/<path_to_spec>/<spec_file>.rb, and the README notes you may need to re-run the setup task when re-running tests, because some tests create models in the database.

The repository layout follows a standard Rails application with app, config, db, lib, spec and public directories, and it also carries the tooling a large Ruby project accumulates: a .rubocop.yml with a .rubocop_todo.yml, an .erb-lint.yml, an .overcommit.yml, a .rspec file and a .ruby-version.

## Adoption is a server deployment, and there is a demo site

The README is unusually open about how to get started, and the honest answer is that this is a service you host.

There is a demo site running at nightly.autolabproject.com, and the documentation has a section explaining how to log in and suggestions on what to try. The installation instructions point to the documentation site rather than to a command, which reflects the shape of the project: a Rails application with a database, a background job runner for the autograding, Docker for the grading containers, and a web server in front of it.

The README also says the core developers can be reached at autolab-dev@andrew.cmu.edu, and that on a case by case basis they provide servers for free, especially for an NGO or a small high school classroom. That is a genuine differentiator for the audience the project was released for, and it is also a statement about the shape of the project as a whole: a Rails service with a community around it rather than a hosted product.

The topics list is worth reading for the audience: autograding, autolab, cmu, education, feedback, programming, rails, scoreboard, students, teaching. This is a tool for a course, not a tool for a developer.

## Conclusion

Autolab suits a school or a teaching team that wants to run autograded programming assignments and already has the Rails and Docker infrastructure to host it, since the service covers handins, gradebooks, extensions and the autograding pipeline in one place. It is the wrong tool for a single instructor wanting a hosted autograding service, because running it is a server deployment, and its Rails and Ruby versions are fixed rather than current. Before adopting it, read the installation documentation at docs.autolabproject.com, check that your grading containers and language toolchains are represented, and plan for the Ruby and Rails versions the current release pins.

## FAQ

### What is Autolab and who uses it?

Autolab is a course management service, initially developed by students at Carnegie Mellon University, that enables instructors to offer autograded programming assignments over the web. The README says around 5,000 CMU students use it each semester across courses in Pittsburgh, Silicon Valley, Qatar and Rwanda.

### How do I set up the Autolab test environment?

Add a test database in database.yml, then run RAILS_ENV=test bundle exec rails autolab:setup_test_env to create and migrate it, and create the required directories such as tmp/. The specs are then run with bundle exec rails spec, and the README warns to use RAILS_ENV=test bundle exec in front of every rails or rake command.

### Does Autolab still support Rails 5?

No. The README says Autolab is now running on Rails 6, that the Rails 5 branch is on master-rails-5, that no new features will be backported to it, and that Rails 5 support has been discontinued. The v2.12.0 release record shows Ruby upgraded to 3.2.2 and Rails to 6.1.7.6.

## Sources

- [autolab/Autolab on GitHub](https://github.com/autolab/Autolab)
- [License: Apache-2.0](https://github.com/autolab/Autolab/blob/master/LICENSE)
- [Project website](http://www.autolabproject.com/)
- [README](https://github.com/autolab/Autolab/blob/master/README.md)
- [Releases](https://github.com/autolab/Autolab/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/autolab-autolab