TFB: A Benchmark Harness for Long-Term Time Series Forecasting Methods
[PVLDB 2024 Best Paper Nomination] TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods
At a glance
- What is it?
- TFB is a Shell-driven benchmarking repository that wraps dozens of published forecasting models behind one pipeline and one set of datasets. Its value is comparability, not novelty, and its main constraint is that the harness is only as fair as the hyperparameters you feed it.
- Who is it for?
- Adopt TFB if you need to place a new forecasting model next to a large set of published baselines under one pipeline, and you are willing to read the scripts folder to see which hyperparameters were actually selected per dataset. Do not adopt it if you need anomaly detection or foundation-model evaluation, since the maintainers point those to separate repositories, TAB and TSFM-Bench.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 47 days ago.
- What is it written in?
- Mainly Shell, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The comparison problem TFB was built to answer
Published forecasting results are hard to compare. Two papers can report the same model on the same named dataset and still disagree, because input length, normalization, train/validation split and the number of tuning trials are rarely identical across codebases. TFB's stated goal is comprehensive and fair benchmarking of time series forecasting methods, and the repository is organized around that single claim. It is aimed at researchers who are about to publish a forecasting model and need baselines that were run under one pipeline rather than copied from other papers, and at engineers who want a rough sense of which model families hold up across many datasets before committing to one. The scope is long-term multivariate forecasting primarily; the README notes that PEMS03 and PEMS07 were added in April 2025, bringing the total to 27 multivariate datasets. The paper behind it was nominated for a best paper award at PVLDB 2024, which is the repository's own framing of the work, not a claim about code quality.
What the repository actually contains
The layout is a Python package, ts_benchmark, with a baselines subdirectory holding one folder per model. The README lists entries such as ts_benchmark/baselines/sparsetsf, ts_benchmark/baselines/timekan, ts_benchmark/baselines/xpatch, ts_benchmark/baselines/hdmixer, ts_benchmark/baselines/patchmlp, ts_benchmark/baselines/amplifier, ts_benchmark/baselines/duet and ts_benchmark/baselines/pdf. Each is a reimplementation or vendored copy of a published method, and each is reachable through the same runner, which is the mechanism that makes cross-model comparison possible at all. The README uses a checkbox convention: a checked box means the code is in the repo and results are on the OpenTS-Bench leaderboard, an unchecked box means only the code is present. TimeBridge is currently unchecked, so its numbers are not part of the leaderboard set. That distinction matters when you are assembling a baseline table, because a model can be runnable from this repository without being part of the published comparison.
The scripts folder is the real specification
The README is explicit that the project retested some algorithms and that these results may differ from those in the TFB paper. The hyperparameters ultimately selected for each algorithm on each dataset live in the scripts folder, and the corresponding results are published on the OpenTS leaderboard under the multivariate_forecasting section. This is the part of the repository to read first. A benchmark harness is a set of choices as much as a set of code, and here those choices are externalized into shell scripts rather than buried in Python defaults. The practical consequence is that reproducing a published number means running the script for that model and dataset pair, not invoking the model with your own settings. If you run a model with settings you chose yourself, you are no longer measuring TFB's comparison; you are measuring your own configuration against TFB's numbers, which is a different experiment.
Getting a run started
The repository is Shell-first: the entry points are scripts, and the badges indicate Python 3.8 or later and PyTorch 2.4.1. The README does not inline a full install and run transcript in the material available here, so the reliable path is to read scripts for the model and dataset you want and follow the invocation it contains, since those scripts encode the documented hyperparameters. Two adjacent tools are referenced for specific jobs. The characteristics_extractor directory contains Characteristics_Extractor.py, which computes series characteristics including trend, seasonality, stationarity, shifting, transition and correlation, with separate English and Chinese readme files. That is useful when you want to explain why one model wins on one dataset rather than only reporting that it did. Separately, a March 2025 feature adds support for predicting only a subset of input variables, documented in docs/tutorials/steps_to_predict_only_a_subset_of_input_variables.md. If your deployment forecasts a handful of sensors out of a larger panel, that tutorial is the relevant entry point, not the default multivariate path.
Where TFB stops being the right tool
The benchmark is bounded by what it was designed to measure. It covers long-term multivariate forecasting on a fixed dataset collection; the README points readers to TAB for time series anomaly detection and TSFM-Bench for time series foundation models, which is an admission that TFB does not cover those. If your problem is short-horizon operational forecasting, probabilistic or quantile output, or irregularly sampled series, nothing in the supplied material suggests TFB addresses it. There is a second, subtler failure mode. Because the maintainers retested algorithms and published revised numbers, any comparison you build by mixing TFB leaderboard values with numbers copied from original papers is not a controlled comparison. The repository also carries a large surface area of vendored model code, so a bug in one baseline's adaptation to the shared interface can produce a number that looks like a result. The scripts folder and the leaderboard are the two places where that risk is at least visible.
How this differs from rolling your own harness
The obvious alternative is a hand-rolled evaluation loop: pick three or four models, write a loader, fix a split, and report the numbers. That gives you full control over the interface and no dependency on someone else's vendored code. The difference in approach is that a custom loop is only comparable to itself. TFB's contribution is the shared interface across roughly two dozen model implementations plus a published set of per-dataset hyperparameters, so a new model can be dropped into an existing comparison instead of starting a new one. The trade is real: you inherit the maintainers' dataset choices, their splitting conventions, and their hyperparameter decisions, and you cannot easily argue that a different configuration would have changed the ranking. DUET's release of long-term forecasting results with unified hyperparameters and a fixed input length of 96, referenced in the README, is a useful illustration of how much the input-length convention alone can move reported outcomes.
Maintenance, licence and what to check first
TFB is MIT licensed, which is permissive and places few obligations on how you reuse the code; the vendored baseline implementations may carry their own upstream terms, so check the individual model folders if you plan to redistribute. The repository is not archived and shows recent activity, with the last push dated July 2025 in the supplied metadata, and the news entries run from May 2024 through June 2025. There are no retrieved releases, so expect to track the master branch rather than a tagged version, which means an upgrade is a pull rather than a version bump and can change baseline behaviour underneath your numbers. The cost of adopting TFB is therefore not installation but pinning: record the commit you ran, the script you used, and the dataset version, because the README already documents one round of retesting that moved results away from the paper. Start with characteristics_extractor if you need to justify a result, and with scripts if you need to reproduce one.
Editorial conclusion
Adopt TFB if you need to place a new forecasting model next to a large set of published baselines under one pipeline, and you are willing to read the scripts folder to see which hyperparameters were actually selected per dataset. Do not adopt it if you need anomaly detection or foundation-model evaluation, since the maintainers point those to separate repositories, TAB and TSFM-Bench. Before trusting any comparison, verify whether the configuration you intend to run matches a script that is present in scripts, and check the OpenTS leaderboard entry for that model, because the README states that some results were retested and may differ from the paper.
Community notes