Toto forecasts observability metrics with a decoder-only transformer over alternating attention

A decoder-only transformer built for the shape of monitoring data

Toto is described as a foundation model for multivariate time series forecasting with a focus on observability metrics, and the architecture decisions in the README are all aimed at one problem: forecasting many variables at once, from a context of variable length, over a horizon you choose.

The decoder-only architecture is the frame. It supports variable prediction horizons and context lengths, which means the same weights serve a five-minute-ahead capacity forecast and a week-ahead one without a separate model per horizon. For observability, where a service has dozens of metrics on different schedules and the question is asked at many different ranges, that flexibility is the point rather than a side benefit.

The attention is the specific trick. Toto 2.0 uses a u-muP-scaled transformer with alternating time and variate attention, and the multi-variate support feature is described as efficiently processing multiple variables using that alternating attention. The alternation is what lets the model reason across variables without the cost of full attention over the flattened variate and time axes at once, and it is why the model handles what the README calls high-dimensional support, time series with a large number of variables, efficiently.

The u-muP part is the scaling discipline. u-muP is the unit-scaled maximal update parameterisation, an approach to making a model's behaviour independent of its width so a small model trained one way transfers to a larger one. Toto 2.0 applies it to the whole family, and the repository carries a separate package, dd-unit-scaling, whose only job is to make u-muP behave correctly when the code is compiled with torch.compile and run under FSDP2. That is a fairly deep piece of infrastructure for a forecasting model, and it is the reason a single codebase produces checkpoints from 4 million to 2.5 billion parameters with the same training recipe.

The primary language is Jupyter Notebook, the licence is Apache-2.0, the default branch is main, and the last push was on 2026-10-19. The recent releases are toto-models v1.0.0, Toto 2.0.0 and dd-unit-scaling v0.1.0, all dated 2026-06-04.

Quantile output rather than a point forecast

Most forecasting APIs return a number and, if they are careful, a confidence interval around it. Toto returns a set of quantiles, and the shape is visible in the quick start.

The model call takes a dictionary with target, target_mask and series_ids, plus a horizon and the two decoding flags, and it returns quantiles of shape (9, batch, n_variates, horizon) with quantile levels [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]. So nine quantile levels per variate per time step, which is a full predictive distribution sampled at deciles rather than a point and a band.

That distinction is the one that matters for observability. A capacity forecast that says the metric will be at 80 percent of current value is not actionable for an alerting system, while a distribution whose 0.9 quantile tells you where the metric is likely to exceed a threshold is. The median sits at 0.5, so a point forecast is available as a byproduct, but the interesting output is the tail.

The mechanism is described as a quantile head, and the feature list calls the result probabilistic predictions, generating both point forecasts and uncertainty estimates via that head.

The mechanism differs between generations and the README is explicit about it. Toto 2.0 uses a quantile head. Toto 1.0 used a Student-T mixture model for the same purpose. So the legacy generation is not simply a smaller or older version of the same design, it is a different probabilistic model underneath, and that is worth remembering when comparing output shapes between the two.

The series_ids argument is also worth a moment, because it is the one part of the interface that is not about the data. It carries an identifier per series in the batch, so a batch can contain series from different sources and the model can tell them apart, which is what lets you batch metrics from unrelated services into one call rather than making one call per service.

The decode_block_size flag is where accuracy and speed are traded

The README gives inference tips for forecast() and the most important one is decode_block_size, which selects the decoding strategy.

None means a single forward pass. The description says it is faster and better for short-term accuracy, and that it was used for all leaderboard results. The alternative is a block decode, with 768 given as the example, which the README says gives better long-term stability for horizons of roughly 1000 or more. The default in the quick start and in the notebooks is 768, which means the shipped examples and the published numbers are not using the same setting, and that is stated plainly rather than left for you to discover.

The other flag is has_missing_values, set to False in the example. When your context has no gaps, that enables Flash Attention kernels for a meaningful speedup. The instruction is to leave it as True, the default, if target_mask contains any False entries. In other words the flag is telling the model whether the missing-value handling path is needed, and setting it wrong on data with gaps produces silently degraded forecasts rather than an error.

Together these two flags are the whole performance surface of an inference call. One chooses between throughput and long-horizon stability, the other chooses between a faster kernel and correct handling of gaps.

The quick start in full:

import torch
from toto2 import Toto2Model

model = Toto2Model.from_pretrained("Datadog/Toto-2.0-22m")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device).eval()

# (batch, n_variates, time_steps)
target = torch.randn(1, 1, 512, device=device)
target_mask = torch.ones_like(target, dtype=torch.bool)
series_ids = torch.zeros(1, 1, dtype=torch.long, device=device)

# Returns quantiles of shape (9, batch, n_variates, horizon)
# Quantile levels: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
quantiles = model.forecast(
    {"target": target, "target_mask": target_mask, "series_ids": series_ids},
    horizon=96,
    decode_block_size=768,
    has_missing_values=False,
)

Note the horizon of 96, which is the default in the example, and the four-dimensional output. The model is moved to a device explicitly and put in eval mode, which is the standard PyTorch pattern and worth copying rather than omitting, because a model left in train mode will produce different numbers from the ones the README reports.

Five model sizes, and the gap where fine-tuning lives

The checkpoint table lists five sizes: Toto-2.0-4m, Toto-2.0-22m, Toto-2.0-313m, Toto-2.0-1B and Toto-2.0-2.5B, with the parameter counts named. That range is the practical decision: a 4 million parameter model and a 2.5 billion parameter model are different products, and the u-muP scaling is what makes the family coherent across that range.

The capability gap is documented just as carefully. There is a note that fine-tuning and exogenous variable support are planned for a future 2.0 release but not yet available, and that if you need these features today you should use Toto 1.0. The legacy section repeats it in the other direction, saying Toto 1.0 is still the right choice if you need fine-tuning or exogenous variable support, which are planned for 2.0 but not yet available.

So the two generations split by capability rather than by quality. Toto 2.0 has the newer architecture and the quantile head and the model family, and it cannot be fine-tuned. Toto 1.0 can, and it also has exogenous covariate support, added in February 2026 along with the fine-tuning support. The fine-tuning support came with a training script, configs and a tutorial notebook.

The exogenous covariate question is the interesting one for observability, because the things that predict a metric are often not other metrics. A deployment, a config change, a scheduled job, a marketing campaign. A model that can be told about those events and learn from them is a different thing from a model that only sees its own history, and the README is candid that this is not yet in the current generation.

The update log reads in reverse chronological order: Toto 2.0 released in April 2026 with five model sizes, fine-tuning support added to Toto 1.0 in February 2026, and exogenous covariate support added to Toto 1.0 in February 2026. There is also a Toto 1.0 paper at arxiv 2505.14766 and a Toto 2.0 technical report at arxiv 2605.20119, with model weights on Hugging Face.

BOOM: the benchmark built from real observability data

The repository is not only a model. It also hosts the code for evaluating time series models on BOOM, the Benchmark of Observability Metrics, described as a large-scale forecasting dataset composed of real-world observability data.

The evaluation section points at two notebooks, one for GIFT-Eval and one for BOOM, and the GIFT-Eval notebook lives in the Salesforce AI Research gift-eval repository, which tells you the model is evaluated through someone else's harness rather than through a bespoke script.

The reason a dedicated observability benchmark exists is the one the README implies when it distinguishes BOOM from the multi-domain benchmarks. GIFT-Eval is described as multi-domain, which means it covers forecasting across several kinds of time series at once. BOOM is built from the same kind of data that a monitoring system produces, so a model that scores well on GIFT-Eval is not thereby good at the forecasting question an on-call engineer actually asks.

The description says the benchmark is composed of real-world observability data rather than synthetic series. Synthetic benchmarks are convenient because the generating process is known, but a metric produced by a service with weekly seasonality, deploys, traffic spikes and a monitoring pipeline that itself drops points is a different object from a sinusoid plus noise. A model trained on the latter and evaluated on the former will disappoint, which is the whole reason for building a benchmark from the data you actually have.

The BOOM dataset card is on Hugging Face alongside the Toto 1.0 model card, and the README links the Toto 2.0 BOOM evaluation notebook at boom/notebooks/toto.ipynb with a separate BOOM README in the boom/ directory. The top-level entries confirm the layout: boom/, dd_unit_scaling/, toto/, toto2/ and toto_models/ as the three generations and packages, plus mypy.ini, pytest.ini, requirements.txt and a LICENSE-3rdparty.csv for the third-party licence accounting that a model release needs.

Where Toto is the wrong tool

Three limits, and the first is the one that decides the architecture question.

The first is that zero-shot forecasting and fine-tuning are different products, and the current release offers only one of them. The headline capability is forecasting without fine-tuning on your specific time series, which is genuinely useful when you have many services and cannot train per service. It is much weaker when you have one important service with a long, clean history and a known future, because then the right answer is to fit that history, and the README tells you to fall back to the older generation to do it. Choosing a foundation model for its architecture while running the previous release for the feature you need is a workable arrangement but an awkward one, and it means the model you evaluate is not always the model you deploy.

The second is the hardware requirement. The listed requirements are Python 3.12 or newer, PyTorch 2.5 or newer, and a CUDA-capable device, with Ampere generation or newer recommended for optimal performance. The installation is one pip install of toto-models, but the weights are on Hugging Face and range up to 2.5 billion parameters, so the memory cost of the large checkpoints is a real consideration for a monitoring system running on every node. The 4m and 22m checkpoints exist precisely for that reason, and choosing between them is a latency and memory decision that the README does not quantify for you.

The third is that this is a research release from a vendor, and the packaging reflects it. The pyproject.toml names the package toto-ts at version 0.2.0 with dynamic dependencies read from requirements.txt, and the requirements file pins the whole stack to exact versions including torch at 2.7.0, transformers at 4.52.1, numpy at 1.26.4 and jupyter at 1.1.1. Exact pins across three dozen packages are what a reproducible research environment looks like and what a long-lived service dependency graph usually does not. There is a LICENSE-3rdparty.csv and a NOTICE file, which is the careful work of tracking what a release contains, and it is also a reminder that the release carries obligations beyond the Apache-2.0 grant on the code itself.

The alternative approach is a conventional statistical forecasting stack on your existing monitoring pipeline, where a seasonal naive baseline or an exponential smoothing model per metric is a few lines of code and needs no GPU, no weights download and no version pinning. The difference in approach is that a foundation model transfers across many services without per-service training, which is a saving when you have hundreds of them, while a per-metric classical model is often as accurate on a well-behaved metric and far easier to reason about when it is wrong. For a large fleet of heterogeneous services the model wins on coverage; for a handful of critical metrics with a clean seasonal structure, the classical approach is still competitive and much cheaper to operate.

Editorial conclusion

Adopt Toto 2.0 if you want a zero-shot forecast for observability metrics without training on your own history, because the model family covers five sizes from 4 million to 2.5 billion parameters and returns a full quantile range rather than a single number. Do not adopt it if you need fine-tuning or exogenous covariates today, because the README is explicit that both are planned for a future 2.0 release and that Toto 1.0 is the right choice until they ship. Verify first that your Python is 3.12 or newer and your PyTorch is 2.5 or newer, that you have a CUDA-capable device with Ampere generation or newer for reasonable performance, that the model size you pick fits your latency and memory budget given the decoding strategy you choose, and that you understand the decode_block_size trade-off, since the default of 768 favours long-term stability over the single forward pass used for the published leaderboard numbers.

Frequently asked questions

Official sources

1. DataDog/toto on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases