MOABB: A Benchmark Harness for EEG Brain-Computer Interface Pipelines

The reproducibility gap MOABB was built to close

BCI papers report decoding accuracies, but the README states the underlying problem plainly: researchers release datasets and not code, so reproducing a published result is harder than it should be, and performance moves with preprocessing parameters, the toolbox used, and implementation details that go unreported. MOABB's answer is to fix the parts of the experiment that are usually left implicit. Datasets, paradigms, and evaluation protocols become named objects in a Python script, and the pipeline under test is the only thing that varies. The intended audience is the researcher who wants to say a pipeline scored a given value on a named benchmark, and the newcomer who wants to know which algorithm works on which dataset without reading a literature survey first. The project states its own success condition in the README: it will have worked when an abstract cites a MOABB score the way abstracts currently cite private benchmarks.

Paradigms as the contract between datasets and pipelines

The mechanism visible in the README is a three-part composition. A dataset object (BNCI2014_001 in the quickstart) supplies raw recordings and a subject list. A paradigm object (LeftRightImagery with fmin=8 and fmax=35) declares the experimental task and the frequency band, and takes responsibility for turning raw EEG into epochs and labels. An evaluation object (CrossSessionEvaluation) decides how those epochs are split and scored. The pipeline is an ordinary scikit-learn estimator, so make_pipeline(LogVariance(), LDA()) works because it satisfies the same fit and predict interface as any classifier. That is the whole design bet: the paradigm absorbs dataset-specific quirks so the pipeline never sees them. The cost is that a paradigm is an opinionated object. Choosing LeftRightImagery commits you to its band, its channel handling, and its epoch window, and if your claim depends on a preprocessing step the paradigm does not expose, the benchmark number stops describing your method. The evaluation class carries a similar commitment: CrossSessionEvaluation is a specific split, not a default you can ignore when you report results.

Running a first benchmark: install, objects, and the subject list trap

Installation is a single command, pip install moabb. The quickstart then imports the pieces by name: moabb itself for the log level, BNCI2014_001 from moabb.datasets, CrossSessionEvaluation from moabb.evaluations, LeftRightImagery from moabb.paradigms, and LogVariance from moabb.pipelines.features. Pipelines are passed to evaluation.process as a dictionary mapping a display name to a scikit-learn estimator, which is why the results table can be keyed by pipeline name. One line in the quickstart deserves attention: dataset.subject_list = dataset.subject_list[:2]. The README uses it to keep the example short, and it is the single easiest way to produce a number that does not mean what a reader will assume it means. If you trim the subject list for speed and then report the accuracy, you have reported a two-subject result. The documentation site linked from the README lists installation options and troubleshooting, and it also carries a dataset summary page, which is where you should check what a given dataset actually contains before building a claim on it.

Where the harness stops helping

MOABB is an offline evaluation framework, and the README gives no indication that it handles acquisition, streaming, or real-time control. If your work is online BCI, this is the wrong layer. A second boundary is the paradigm abstraction itself. Because the paradigm owns preprocessing, a method whose contribution is a novel artifact-handling or referencing step cannot be expressed as a pipeline without either modifying the paradigm or accepting that the benchmark measures a different system than the one you are proposing. Third, the benchmark's comparability rests on the datasets it ships and the splits it defines. Two groups can both report a MOABB number and still have run different subject lists, different bands, or different evaluation classes, and nothing in the framework prevents that. The README's own framing acknowledges this: it describes the project as open science that may evolve with community need, which is an honest way of saying the protocol is a moving target across releases.

MOABB against Braindecode and raw MNE scripting

The closest alternative in this space is Braindecode, which also builds on MNE and PyTorch but takes a different position on the problem. Braindecode supplies dataset loaders and deep-learning model implementations; you write the training loop, the split, and the metric. MOABB supplies the split and the metric as first-class objects and leaves the model as any scikit-learn estimator. The practical difference shows up in what you get for free. With MOABB you get a results table whose columns are comparable across pipelines because the evaluation object fixed the protocol. With Braindecode you get architectural flexibility and no protocol unless you write one. The second alternative is scripting MNE directly: read raw files, filter, epoch, cross-validate. That gives complete control and zero comparability, which is precisely the situation the README describes as the problem. Pick MOABB when the claim is about relative performance under a shared protocol; pick a lower-level tool when the claim is about a mechanism the protocol would hide.

What you inherit when you depend on it

The repository ships under BSD-3-Clause, which permits commercial and academic reuse with the licence text retained; this is a description of the licence, not legal advice, and anyone embedding MOABB in a product should read the file themselves. The maintenance picture from the supplied material is active: three releases between late August and early September 2026, with v1.7.1 as the most recent, and the default branch is develop rather than a stable line, so installing from source means tracking in-progress work. That has an upgrade consequence worth naming. Because paradigms and evaluation classes define the protocol, a version bump can change what a benchmark number means, and a script that pinned nothing will silently produce a different result after an upgrade. Pin the moabb version in whatever environment produced your reported numbers, and record it alongside the dataset name, the subject list, and the paradigm band. The cost of adopting MOABB is not installation; it is the discipline of reporting those four values together.

Editorial conclusion

Adopt MOABB if you are publishing an EEG decoding pipeline and need a cross-dataset or cross-session number that a reviewer can reproduce from your script. Do not adopt it if you need real-time acquisition, artifact rejection tuning, or source-localisation work; it is an offline evaluation harness built on MNE and scikit-learn, and it will not give you those. Before you quote any number, verify the subject_list you actually ran, the fmin and fmax you passed to the paradigm, and whether the evaluation class you chose matches the split your claim depends on. Those three values are what a reader will try to reproduce.

Official sources

1. License: BSD-3-Clause
2. NeuroTechX/moabb on GitHub
3. Project website
4. README
5. Releases