EvalAI: A Self-Hosted Leaderboard and Submission Server for AI Challenges

The comparison problem EvalAI was built to fix

The README opens with a specific complaint: comparing an algorithm against existing approaches is hard because implementations differ slightly, dataset splits are non-standard, and evaluation metrics vary between papers. The stated remedy is a central leaderboard plus a submission interface, so that a reported number and a reproduced number come from the same code path. That framing tells you who the project is for. It is not a library you import to compute a metric. It is a server that a challenge organizer stands up, configures with phases and splits, and hands to participants. The audience is therefore split in two: organizers who need submission handling, scoring, and a leaderboard, and participants who need a way to submit without building their own upload tooling. The README also points at evalai-cli, a separate repository, as the terminal-facing half of that participant experience. If your problem is "I have one model and one test set and I want a number," EvalAI is aimed past you. If your problem is "I have forty teams and I need their predictions scored consistently and ranked publicly," that is the gap the project claims to fill.

Django, SQS, PostgreSQL, and a worker pool

The installation section names the moving parts directly. The default docker-compose invocation starts only db, sqs, and django. Workers are a separate profile, started with docker-compose --profile worker up --build. That split is the architecture in miniature: the web application and the queue exist by default, and the compute that actually scores submissions is opt-in. The README describes what those workers do. For remote evaluation, organizers who need compute beyond what the host provides can add their own cluster of worker nodes; the platform keeps handling hosting, user submissions, and the leaderboard. For evaluation inside environments, participants submit code as Docker images, and during evaluation a worker fetches the image, the test environment, and the model snapshot, then spins up a new container to run the evaluation. That is a container-per-submission model, which explains why the worker profile is separate from the default stack. The README also describes two performance measures: warming up worker nodes at start-up by importing the challenge code and pre-loading the dataset into memory, and splitting the dataset into chunks evaluated on multiple cores. The README claims these reduce evaluation time by an order of magnitude in some cases. That is the project's own claim, not an independently reproduced figure, and no benchmark methodology accompanies it.

Running it locally: three commands and three accounts

The documented local setup is short. Clone the repository with git clone https://github.com/Cloud-CV/EvalAI.git evalai && cd evalai, then run docker-compose up --build. The README warns this may take a while. When the stack is up, the web interface is at http://127.0.0.1:8888. On first boot the platform creates three accounts, all with the password password: admin as superuser, host as host user, and participant as participant user. Those three roles map onto the two audiences described earlier, and the fact that they are seeded automatically is what makes the local instance usable for exploring the organizer and participant flows without any manual account creation. To include workers, use docker-compose --profile worker up --build instead. If the build fails, the README points to an installation troubleshooting page in the Read the Docs documentation rather than listing failure modes inline. One practical consequence of the seeded credentials: any instance reachable from a network with those defaults in place is trivially accessible, so changing them is part of going beyond local experimentation, not an optional hardening step.

Phases, splits, and the public/private leaderboard split

The feature list describes custom evaluation protocols and phases: an arbitrary number of evaluation phases and dataset splits, compatibility with any programming language, and results organized into both public and private leaderboards. Read that as the answer to a familiar competition design. A public leaderboard shows scores on a visible subset during the challenge; a private leaderboard holds back the rest until the challenge closes. Phases let an organizer stage that over time, for example a development phase and a final phase with different data. The claim of any-programming-language compatibility follows from the evaluation model rather than from a language binding: if scoring happens in a worker that runs challenge code or a submitted container, the language of the submitted model is not the platform's concern. The trade-off is that the interesting logic lives in the challenge configuration, which the README does not document in detail. Nothing in the supplied material shows the schema for defining a phase or a split, so you should expect to read the source or the Read the Docs site before you can design a challenge. That is a real gap between the feature list and the getting-started path.

Where EvalAI is the wrong tool

Two constraints stand out. First, the worker profile is not part of the default stack. If you run docker-compose up --build and then wonder why submissions sit unprocessed, the answer is in the README: workers need --profile worker. Anyone evaluating the platform casually will hit this and may conclude the queue is broken. Second, and more consequential for adoption, the repository does not declare a licence. The licence field reads NOASSERTION, which means GitHub could not map the repository's licence files to a known identifier. The README's only legal-adjacent statement is a citation request for people hosting challenges, which is not a licence grant. Before you host a public challenge on your own infrastructure, or accept submissions that include participant code, you need to determine the actual terms from the repository itself. I cannot tell you what those terms are from the material supplied. A third limitation is release cadence. The tagged releases are 1.0 from April 2017 and 1.1 from April 2018. The default branch shows activity far more recently, but there is no tagged release covering it, so anyone deploying from a release is deploying code that predates years of work. Pin a commit if you deploy at all.

How it differs from a hosted benchmark suite

The obvious comparison is a hosted evaluation service: you upload predictions, the operator scores them against a held-out set, and a leaderboard updates. EvalAI can play that role, and eval.ai is the project's own hosted instance. The difference is control over the evaluation path. With a hosted service you accept the operator's metric implementation and cannot change it. EvalAI's remote evaluation feature exists precisely so that an organizer who needs unusual compute or a custom protocol can attach their own worker nodes while the platform continues to handle submissions and the leaderboard. Docker-image evaluation pushes that further: instead of submitting predictions, participants submit an agent, and the worker runs it in a fresh container against a test environment. A hosted prediction-scoring service generally has no equivalent, because running arbitrary participant code is a much larger operational commitment. The cost of that control is that you now operate a Django application, a PostgreSQL database, a queue, and a worker pool. The comparison is not feature-for-feature; it is a trade of operational burden for the ability to define and run the evaluation yourself.

Maintenance cost and what to check before deploying

Everything the README describes runs on open source components: Docker, Django, Node.js, and PostgreSQL. The README frames this as portability, and it does mean there is no proprietary dependency in the critical path. It also means the upgrade surface is the union of those projects. A Django application of this vintage, with a frontend the topics list identifies as AngularJS, carries the usual cost of keeping a multi-service stack patched, and the absence of a release after 1.1 means there is no changelog to read before you move between commits. On licensing, the honest position is that the material does not establish terms. NOASSERTION is a signal to go read the repository's licence files and, if the answer matters commercially, to get a real opinion rather than mine. For a university lab hosting an internal challenge, the practical checks are narrower: confirm the worker profile starts and picks up jobs, confirm the seeded admin, host, and participant accounts are changed or removed, and confirm that the challenge configuration format documented on Read the Docs matches the commit you deployed. Those three checks are specific to this stack and will surface most of the ways a first deployment goes wrong.

Editorial conclusion

Adopt EvalAI if you are a challenge organizer who needs submission handling, public and private leaderboards, and the option to attach your own worker cluster, and if you are willing to run a Django, PostgreSQL, and Docker stack yourself. Do not adopt it if you only need to score a model you already have on disk, or if you need a maintained release cadence: the last tagged version is 1.1 from 2018, so pin a commit rather than a release. Verify three things before committing: the actual licence terms, since the repository does not declare one; that docker-compose --profile worker brings up workers that can reach your challenge code; and whether the default admin, host, and participant credentials created on first boot are acceptable in your environment.

Official sources

1. Cloud-CV/EvalAI on GitHub
2. Issues
3. Project website
4. README
5. Releases