transformer: an attention model applied to time series forecasting

A from-scratch implementation rather than a forecasting library

The project is an implementation of the Transformer model, originally from Attention is All You Need, applied to time series and powered by PyTorch. The framing in the documentation is deliberately about the mechanism, listing the three properties of attention based neural networks that the project cares about: linear complexity in the dimension of the feature vector, parallelisation of computing a sequence as opposed to sequential computing, and long term memory, as any input time sequence step can be looked at directly. The third property is the interesting one for forecasting, because a recurrent model compresses history into a hidden state and a transformer can attend directly to a distant point in the sequence. What the repository is not is a library of estimators with a fit and predict interface. The package installs as tst, and the only entry point the documentation points at is the Transformer class, which can be used out of the box. Everything else in the repository is a notebook or a script that the author used to run experiments, which makes it a good teaching codebase and a poor production dependency.

Three edits turn a language model into a time series model

The adaptations section lists three changes, and each one maps to a specific mismatch between the original architecture and a time series input. The embedding layer is replaced by a generic linear layer, which makes sense because there is no token vocabulary to embed, only a vector of numerical features per time step. The original positional encoding is removed, with the note that a regular version, better matching the input sequence day and night patterns, can be used instead, so positional information is available as an option but is not imposed. The third change is the one with the most impact: a window is applied on the attention map to limit backward attention and focus on short term patterns. A window on the attention map is not the same as a window on the input sequence, because it constrains which positions a given position may attend to rather than how much history is fed in. The effect is that the model becomes locally sensitive, which is what a forecaster usually wants, and the trade is that the long term memory property of the original architecture is deliberately given up. If you are reading this to understand why a transformer forecast differs from an LSTM forecast, that window is most of the answer.

The use case is a surrogate model for a building simulation

The dataset and application section describes something more specific than a generic forecasting benchmark, and it explains the shape of the data. The stated use case is modeling a numerical simulator for building consumption prediction. The method was to sample random inputs, covering building characteristics and usage and weather, and collect the simulated outputs, then convert those variables into time series format and feed them to the transformer. That is a metamodel, and it is a good use of a transformer for a specific reason: the input is a high dimensional vector of simulation parameters expanded into time series, and the model has to learn a mapping that is a function of many interacting physical parameters rather than a temporal extrapolation. A metamodel is also a different risk profile from a forecasting model, because the training data is generated rather than observed, so it has no noise, no missing values and no concept drift, and the usual reasons to distrust a forecast do not apply. The drawback is that the surrogate is only as good as the simulator it was trained on, and the simulated outputs are only as good as the simulator. The repository topics list metamodel alongside timeseries and transformer, which is the honest labelling of what this is.

Getting the environment right, and the Python version problem

The installation section is short and carries a warning that matters in practice. All required packages can be found in requirements.txt, and the project expects to be run with python3.7. The requirements file lists torch, jupyterlab, matplotlib, seaborn and tqdm, which tells you the notebooks are the intended interface and that plotting and progress display are part of the workflow. There is also a caveat that pytorch may have to be installed manually if you are not using pip with a Debian distribution, with a pointer to the PyTorch installation page. The documented sequence is a virtual environment on python3.7:

$ apt-get install python3.7
$ pip3 install --upgrade --user pip virtualenv
$ virtualenv -p python3.7 .env
$ . .env/bin/activate
(.env) $ pip install -r requirements.txt

That sequence is the interesting part for a new reader, because python3.7 reached end of life years ago and most current distributions no longer package it. The setup.py metadata is slightly more permissive, declaring python_requires of 3.6 or above and a single install dependency of torch, with the packages list fixed to tst. So the packaging metadata and the documentation disagree about the floor, and the documentation is the one that describes how the author actually ran it. Anyone trying this today should expect to install a modern Python and see whether the code still runs, and the answer to that question is not in the repository.

The dataset is not in the repository and has to be fetched

The data story is the biggest practical obstacle and it is described without softening. The dataset is not included in the repository and must be downloaded manually. It is comprised of two files, dataset.npz, which contains all the input and output values, and labels.json, which is a detailed list of the variables, and the documentation points at issue 2 in the repository for more information on where to get them. So a clone of the repository gives you the model and no data, and the two files are a different shape of artifact from source code. The npz format also means the data loading path is fixed rather than configurable, which is fine for reproducing the work and awkward for applying the model to your own simulation outputs. If you want to use this on a different simulator, the assumption to check is the input structure, since the model was trained on a specific set of variables converted into a specific time series format. The repository does contain a labels.json at the top level, so the variable list is at least visible in the tree even when the data itself is not, which is the piece you need in order to know whether your own variables map onto it.

A training notebook, a cross validation script and a search script

The top level of the repository is a working directory rather than a package layout, and the files tell you what the author was doing. There is training.ipynb, the default notebook the documentation tells you to run under jupyter, and the note that all adjustable parameters are in the second cell. There is also training.py, which appears to be the script form of the same thing, plus cross_validation.py, learning_curve.py, search.py, export_doc.py, a benchmark notebook and a visualization notebook, alongside a src/ directory and a docs/ directory. The one warning in the usage section is specific and easy to get wrong: be careful with the BATCH_SIZE, because it is used to parallelise head and time chunk calculations. That is a coupling between a performance knob and a correctness requirement, and it means the batch size is not a free parameter the way it usually is in a training loop. The documentation build is a Sphinx project, since the docs are built with a make html call from the docs directory.

The class signature is the whole public API

The out of the box usage is one import and one constructor call, and the parameter list is the real documentation for the model:

from tst import Transformer

net = Transformer(d_input, d_model, d_output, q, v, h, N, TIME_CHUNK, pe)

Reading that signature tells you the architecture. d_input and d_output are the feature dimensions of the input and the prediction, d_model is the width of the model, q and v are the dimensions of the query and value projections, h is the number of attention heads, N is the number of encoder layers, TIME_CHUNK is the size of the sequence chunk the model operates on, and pe is the positional encoding option, which is the switch described earlier as the regular version better matching day and night patterns. The generated API documentation lives at the readthedocs site for the project, and the repository has a docs directory with a make html build target. The naming with capitals in TIME_CHUNK and N is not stylised elegance, it is a signal that these are configuration values that come from the notebook rather than defaults in the library, and the practical consequence is that constructing this class correctly requires reading the notebook, not just the signature.

Editorial conclusion

transformer is a good fit for a reader who wants to understand the attention mechanism applied to sequences in a codebase small enough to hold in your head, and for a researcher who needs a metamodel of a numerical simulator and is comfortable building the training loop around it. It is the wrong tool for production forecasting, since the last push was on 2026-04-08 and the last tagged release is 0.4.0 from 2022-09-23, so nothing here has been validated against recent PyTorch, and it is the wrong tool if you want batteries included model selection, because the repository ships a training notebook rather than a library of estimators. Before adopting it, read the attention window section, because the window is the change that most affects results, and check that the training data has been prepared in the expected time series format since the dataset is not included in the repository and has to be downloaded separately. The project is GPL-3.0 licensed and the package installs as tst.

Frequently asked questions

Official sources

1. License: GPL-3.0
2. maxjcohen/transformer on GitHub
3. Project website
4. README
5. Releases