mvts_transformer: a KDD 2021 time series transformer you can actually run

One transformer backbone serving regression, classification and imputation

This repository is the public code for a KDD 2021 paper, and the project documentation says so plainly: the code corresponds to A Transformer-based Framework for Multivariate Time Series Representation Learning by Zerveas, Jayaraman, Patel and Eickhoff, with an arXiv version also cited. What you get is a research implementation, not a packaged library. There is no setup.py entry point, no pip installable name in the setup notes, and the layout is a plain src/ directory with a single entry script.

The tasks are wired through the same command line tool. Regression and classification are handled by what the project calls ClassiregressionDataset, which requires a labels dataframe in addition to the feature dataframe, while autoregressive tasks such as imputation and transduction only need the feature dataframe. That distinction is enforced in the data layer, and it is the first thing to understand before you add data of your own.

The dataset layer leans on PyTorch Dataset subclasses, and the requirements list pulls in sktime, pandas, scikit-learn and torch. The repository also ships example_data_class.py as a template, which tells you that the intended extension point is a data class rather than a plugin.

One detail worth noting early: for regression the project states that the reported loss is the Mean Square Error, that is, without the Root. If you compare numbers against a paper that reports root mean square error, the gap is in the metric definition, not in the training.

pip install -r requirements.txt, and a failsafe file when sktime drifts

Setup is short, and the setup notes are unusually candid about a specific pain point. Instructions refer to Unix based systems such as Linux and macOS. You clone or enter the project directory, create an output root inside it, and install dependencies into a virtual environment, conda or virtualenv being the suggested route so that conflicting requirements do not collide with your system packages.

The dependency file lists torch, pandas, numpy, scikit-learn, sktime, tensorboard, matplotlib, tqdm, tabulate, protobuf and the xlrd, xlutils and xlwt family for spreadsheet records, plus ipdb for debugging. Installation is a single command, `pip install -r requirements.txt`, run inside that environment.

And the reason failsafe_requirements.txt exists is spelled out: newer versions of packages, with sktime given as the example, sometimes break backward compatibility with previous versions or with other packages in the set. When that happens, the project points you at the pinned file instead, which contains specific versions tested against this codebase.

The environment assumption matters more than it looks. The README says the code was tested with Python 3.7 and 3.8. On a modern interpreter the pip resolution may pull numpy, pandas or scikit-learn builds that no longer support those interpreters, which is the point at which you will want the failsafe file and possibly an older Python.

One output directory per experiment, with checkpoints, metrics and predictions

Experiments do not write into a shared results folder. The project explains that each experiment creates a time-stamped output directory inside the root directory you prepared, and that directory contains model checkpoints, performance metrics per epoch, predictions per sample, the experiment configuration and log files. You create that root yourself, for example as experiments inside the project directory, and pass it with --output_dir.

The entry point is src/main.py, and the full option list is available through the standard help flag, `python src/main.py --help`.

Progress can be followed in the console and in the output.log file, and the project also shows a tensorboard path for a visual view of the same curves, after installing tensorboard with pip install tensorboard:

tensorboard dev upload --name my_exp --logdir path/to/output_dir

Two dataset sources are named. Classification data comes from the multivariate 2018 archive, and regression data from a Zenodo record. It is explicit that the two kinds of dataset belong in separate directories, one for regression and one for classification, and that each dataset then lives in its own subdirectory under those roots. The --data_dir flag in every example command points into that layout.

One warning applies to every command below: the configurations shown are intended for BeijingPM25Quality for regression and SpokenArabicDigits for classification. You are directed to the hyperparameter values in the paper's supplementary appendix for other datasets rather than reusing these.

From scratch versus masked pretraining, the fork that decides your results

The repository offers two genuinely different routes, and choosing badly wastes hours of GPU time. Route one is training from scratch on the target task. Route two is self-supervised pretraining through input masking, then fine-tuning the pretrained weights on the downstream task.

From scratch on a regression task looks like this, and note that the name field interpolates the dataset variable you are told to substitute:

python src/main.py --output_dir path/to/experiments --comment "regression from Scratch" --name $1_fromScratch_Regression --records_file Regression_records.xls --data_dir path/to/Datasets/Regression/$1/ --data_class tsra --pattern TRAIN --val_pattern TEST --epochs 100 --lr 0.001 --optimizer RAdam  --pos_encoding learnable --task regression

The classification equivalent runs for many more epochs and asks for accuracy as the key metric:

python src/main.py --output_dir experiments --comment "classification from Scratch" --name $1_fromScratch --records_file Classification_records.xls --data_dir path/to/Datasets/Classification/$1/ --data_class tsra --pattern TRAIN --val_pattern TEST --epochs 400 --lr 0.001 --optimizer RAdam  --pos_encoding learnable  --task classification  --key_metric accuracy

Pretraining runs as a separate job with input masking as the objective, and it can be reused for any downstream task including imputation and transduction:

python src/main.py --output_dir experiments --comment "pretraining through imputation" --name $1_pretrained --records_file Imputation_records.xls --data_dir /path/to/$1/ --data_class tsra --pattern TRAIN --val_ratio 0.2 --epochs 700 --lr 0.001 --optimizer RAdam --batch_size 32 --pos_encoding learnable --d_model 128

The constraint that bites is stated twice in the setup notes. The network architecture parameters of the pretrained model must match those of the model you fine-tune, and the concrete example given is --d_model 64 for SpokenArabicDigits. A mismatch silently gives you a fine-tune that does not load the checkpoint weights in the shape you expect. The project also notes that for BeijingPM25Quality pretraining the correct batch size is 128, not the 32 used in the generic command.

Adding your own data means writing a class and registering a string

The extension mechanism is direct and small. To train on data that is not in either archive, you add a new data class in datasets/data.py, using the other classes in that file or the template in example_data_class.py as a starting point.

The class contract is described precisely. It sets up one or more pandas DataFrames containing all the data, indexed by example IDs. Autoregressive tasks require a member dataframe called self.feature_df, while regression and classification additionally require a self.labels_df member variable. Those frames are then reached through the PyTorch Dataset subclasses in dataset.py.

Registration is a dictionary entry, and the project shows the shape with existing keys and a placeholder for your own:

data_factory = {'weld': WeldData,
                'tsra': TSRegressionArchive,
                'pmu': PMUData,
                'mydataset': MyNewDataClass}

Once that entry exists, the string becomes a value for --data_class, so the same command line works with your data. The catch is that this only handles the data layer. The records files named in the commands, such as Classification_records.xls, and the specific hyperparameters tuned per dataset in the paper's supplementary section are not generated for you by the factory, so a custom dataset still needs its own record file and its own tuning pass.

For long series the project offers --subsample_factor, described as a way to speed up training on datasets with longer time series without significant performance degradation. That is a training time lever, not a substitute for choosing the right hyperparameters.

A 2021 codebase with no releases and an unmaintained dependency surface

The repository is not archived and the last push was on 2026-04-29, so the code is not frozen. It is also true that the repository has no GitHub releases, there is no versioned package to install, and the environment it was written for is two Python versions behind current practice. The code has been tested with Python 3.7 and 3.8.

That combination produces a specific kind of friction. The dependencies are pinned with lower bounds only in requirements.txt, so a fresh install resolves to whatever is current for each package, and the sktime compatibility problem the project describes is a symptom of that. The failsafe file exists precisely because the open ended list is not reproducible.

The licence is MIT, which is permissive for commercial use and for derivative work, with no copyleft obligation attaching to code you train models with. Nothing in the project offers a support contract, a compatibility promise between Python versions, or a schedule for future releases, so treating this as a dependency in a production pipeline means owning the upgrade yourself.

The honest framing is that this is a faithful artifact of a specific paper. If your goal is to reproduce the paper's numbers, to build on its pretraining idea, or to read a well-structured transformer implementation for multivariate time series, the code is the point. If your goal is a maintained library with published versions and a support community, it is not, and no amount of configuration changes that.

Editorial conclusion

mvts_transformer is the right choice if you want to reproduce or extend the masked pretraining approach from the KDD 2021 paper and are willing to work inside a command line harness that targets Python 3.7 and 3.8. It is the wrong choice if you need a maintained library, a pip installable package, or a dataset loader that already knows your domain. Before you start, install with pip install -r requirements.txt, keep failsafe_requirements.txt in reach when sktime breaks, and confirm that the hyperparameters in the paper's supplementary appendix match the dataset you plan to run, because the example commands are tuned for BeijingPM25Quality and SpokenArabicDigits rather than for general use.

Frequently asked questions

Official sources

1. gzerveas/mvts_transformer on GitHub
2. Issues
3. License: MIT
4. README