Mammoth: A PyTorch Harness for Running and Extending Continual Learning Baselines

What Mammoth Is Actually For

Continual learning research has a comparison problem. A new method is usually evaluated against a handful of reimplemented baselines, each with its own data ordering, its own buffer sampling code, and its own idea of what a task boundary is. Mammoth exists to collapse that variance. The README describes it as built to streamline the development and benchmark of continual learning research, and states it includes more than 70 methods and 20 datasets. The repository is also the official codebase for Dark Experience for General Continual Learning, so the DER and DER++ family sits inside it as first-class methods rather than as an external reference implementation.

The intended user is a researcher or engineer who already knows PyTorch and wants either to reproduce a known result or to slot a new training strategy into an existing evaluation loop. It is not aimed at practitioners who want to apply continual learning to a production model. There is no serving component, no inference API, and no packaging story beyond cloning the repository.

How the Framework Is Wired Together

Mammoth is organized around a small number of directories that map to the concepts in a continual learning experiment. New models go in models/, new datasets go in datasets/, and main.py is the single entry point that runs an experiment. Backbones are separated from methods, with a backbone/ directory holding at least a ViT implementation, and utils/ holds shared code. The documentation is split along the same seams: separate reference pages exist for models, datasets, backbone, and utils.

That separation is the design decision that matters. A method in Mammoth is not a monolithic training script; it is a component that consumes a dataset and a backbone chosen at run time. This is why the same command line can swap --model and --dataset independently. It also explains the framework's stated emphasis on being easy to debug: because the training loop is shared, a failure in a new method is more likely to be in the method's own code than in the surrounding plumbing.

The concrete mechanism visible in the README is argument-driven dispatch. main.py parses flags such as --model, --dataset, --buffer_size, --lr, --alpha and --beta, and resolves them into a configured experiment. Hyperparameters can be supplied inline or pulled from a per-model YAML file, which is the framework's answer to the problem of remembering which settings produced which published number.

Running an Experiment: Commands and Config Files

Setup is either pip install -r requirements.txt or, if you use uv, running directly with uv run python main.py. The README notes that PyTorch >= 2.1.0 is required for scaled_dot_product_attention, and gives a fallback: uncomment lines 136-139 under scaled_dot_product_attention in backbone/vit.py if your environment cannot meet that. That fallback is worth reading before you start, because it is the one place where the framework tells you it has a hard version dependency tied to a specific file and line range.

The canonical example in the README runs DER++ on a CIFAR-100 task sequence with a 500-sample buffer:

python main.py --model derpp --dataset seq-cifar100 --alpha 0.5 --beta 0.5 --lr 0.001 --buffer_size 500

To use tuned settings instead of hand-passed values, the README gives:

python main.py --model derpp --dataset seq-cifar100 --model_config best

The --model_config argument looks for a file named <model_name>.yaml inside models/config/. If that file is missing for the method you want, the flag has nothing to resolve, so check the folder before assuming a method ships with a tuned configuration.

Two operational flags are documented under new features. Training captures SIGINT so that Ctrl+C saves the current state to checkpoints/paused/, and this can be turned off with --save_after_interrupt=0. Checkpoint location is controlled by --checkpoint_path, defaulting to the checkpoints/ directory. The --loadcheck option can read arguments saved alongside a checkpoint, so resuming is a matter of running python main.py --loadcheck <checkpoint_name> rather than reconstructing the original command line. There is also a separate uploader, scripts/upload_to_hf.py, with options including --repo-id, --local-dir, --pattern, --remote-dir, --repo-type, --revision, --exclude and --dry-run, for pushing checkpoints and caches to Hugging Face. The newer Task Arithmetic entry in the news section uses --loadcheck with a Hugging Face checkpoint URL and a --fisher_cache pointing at hf://.

Where Mammoth Stops Being the Right Tool

The most concrete limitation is packaging. The README states plainly that Mammoth is not yet available on PyPI, so using it as a library means cloning the repository and running pip install -e . or uv sync. If your project needs a pinned dependency resolved from an index, that is a structural obstacle rather than a missing convenience.

The second limitation is scope of training regime. The update roadmap lists new training modalities such as regression, segmentation and detection as work in progress. As of the material available, the framework's documented examples are classification-style task sequences such as seq-cifar100. If your problem is dense prediction, you are on the roadmap's future work, not on supported ground.

The third is the absence of a results dashboard. The roadmap describes an openly accessible dashboard for visualizing model results both in their respective settings and in a general setting, and adds that this may take some time since compute is not free. Until that exists, the comparative claim of the framework rests on you running the experiments yourself. There are no retrieved releases either, so there is no versioned artifact to pin against; the repository tracks a master branch.

One more caveat deserves stating directly. The README has a section on the reproducibility of Mammoth, but the supplied material does not include its contents. Whether the published numbers are reproduced by the current code is therefore something you must check against the documentation rather than take on faith.

How It Differs from Avalanche and Other CL Libraries

Avalanche is the closest comparison in this space: a continual learning library that also bundles benchmarks and strategies. The difference in approach is one of packaging and intended use. Avalanche is distributed as an installable library with its own abstractions for scenarios and benchmarks. Mammoth is distributed as a research repository you clone, and its abstractions are the folder layout itself: models/, datasets/, backbone/ and utils/, with main.py as the driver. Adding a method to Mammoth means adding a file under models/ and optionally a YAML under models/config/; adding one to a library-style framework usually means conforming to a plugin interface defined by that library.

That makes Mammoth easier to read end to end if you want to see exactly what the training loop does, and harder to consume as a dependency in another codebase. It also means the framework's extension points are conventions rather than enforced contracts. The README points to documentation pages titled build_a_model and build_a_dataset, which is where those conventions are spelled out; the top-level README itself does not define the interface signatures.

A second reference point is the original DER paper implementation, which Mammoth supersedes for that method by folding DER and DER++ into a larger benchmark set. If you only care about DER, the framework's overhead is the price of having 70-plus other methods available for comparison in the same harness.

Maintenance, Upgrades and the MIT Licence

The repository is not archived, and the last push recorded is 2026-05-20, so it is under active development. The README confirms this in the roadmap, stating that all the code is under active development and that new additions will try to preserve the current structure of the repository. That last clause is the upgrade guarantee you actually get: structural stability by intent, not a semantic versioning promise. There are no retrieved releases, so upgrades mean pulling master.

Practical upgrade cost concentrates in two places. First, the PyTorch floor: the scaled_dot_product_attention requirement of >= 2.1.0 ties the framework to a relatively recent torch, and the documented workaround is editing backbone/vit.py. Second, the models/config/*.yaml files: if a method's tuned hyperparameters live in a YAML and that file changes, --model_config best will silently give you different numbers than a previous run. Recording the commit hash alongside your results is the only way to make those runs comparable.

Mammoth is MIT licensed. That permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. It does not grant rights to the datasets the framework loads, and those carry their own terms that you need to check separately. Nothing here is legal advice; read the LICENSE file in the repository for the binding text.

Who Should Clone It

Adopt Mammoth if you are benchmarking a new continual learning method and want a large set of existing baselines sharing one training loop, one dataset loader and one argument surface. The modular layout means your method is a file, not a fork. The --model_config best mechanism is a small but real convenience: it turns a hyperparameter table into a file you can read and diff.

Do not adopt it if you need a pip-installable dependency from PyPI, if your task is regression, segmentation or detection, or if you need a hosted comparison dashboard. Those are explicitly unfinished in the roadmap, and building on them now means building on the roadmap rather than the code.

What to verify first: confirm your torch version satisfies the >= 2.1.0 requirement for scaled_dot_product_attention, or plan to apply the documented edit in backbone/vit.py; check that models/config/ contains a YAML for the specific method you intend to run with --model_config best; and read the documentation's reproducibility section, which the top-level README references but does not reproduce, before treating any baseline number as a target. The framework is a harness for producing comparable numbers, and it only does that if you pin the commit you ran.

Editorial conclusion

Adopt Mammoth if you need to place a new continual learning method against a large set of existing baselines under one harness, and you are willing to work inside its models/ and datasets/ folder conventions. Do not adopt it if you need a supported PyPI package, a regression or detection training regime, or a published results dashboard; the roadmap lists those as unfinished. Before committing, verify the PyTorch version in your environment against the >= 2.1.0 requirement for scaled_dot_product_attention, and confirm that a models/config/<model_name>.yaml file exists for whichever method you intend to run with --model_config best.

Official sources

1. aimagelab/mammoth on GitHub
2. Issues
3. License: MIT
4. README