weatherbench2: the evaluation framework behind global weather model scores

What WeatherBench 2 is actually for

WeatherBench 2 is described as a framework for evaluating and comparing data-driven and traditional numerical weather forecasting models. It was built as an evolving tool for the entire community, and the project welcomes feedback as GitHub issues or contributions.

The four components in the README describe the whole design. First, publicly available, cloud-optimized ground truth and baseline datasets, with a complete list on the data guide page. Second, open-source evaluation code, with a quick-start for the basic functionality and API docs for more detail. Third, a website displaying up-to-date scores of many of the state-of-the-art data-driven and physical approaches. Fourth, a paper describing the rationale behind the evaluation setup, the arXiv entry 2308.15560.

The reason this matters is that weather model evaluation has a reproducibility problem. A model can be described as better than another in a blog post, but if the ground truth, the interpolation method, the metrics and the verification region differ between the two evaluations, the comparison means nothing. WeatherBench 2 is an attempt to fix the ground truth and the metric definitions so the comparison is meaningful, and the leaderboard is a consequence of that rather than a separate product.

The audience is therefore research teams publishing model results, and secondarily anyone who wants to check a claim about a forecast model against a standard reference.

The scalability decision behind Xarray-Beam and Dataflow

The scalability note in the README is specific: since high-resolution forecast files can be large, the WeatherBench 2 code was written with scalability in mind. The command-line scripts are based on Xarray-Beam, and there is a separate guide for running the scripts on GCP using Dataflow.

That is the central architectural decision, and it explains the rest of the dependency list. The base requirements in setup.py include apache_beam at version 2.31.0 or later, xarray-beam, xarray, zarr, numpy, pandas, scipy, scikit-learn, cftime, jax with the cpu extra, and a gcsfs based extra set for GCP. Cloud storage is reached through fsspec style filesystems, and zarr provides the chunked array format the datasets are stored in.

The practical consequence is that the same evaluation code runs at two scales. On a laptop, you evaluate a low resolution forecast against a local or cloud-hosted zarr dataset. On Dataflow, the same script processes terabytes of high resolution output with the parallelisation handled by the Beam runner rather than by anything in the framework itself.

The jax cpu dependency is a separate thread. JAX is used for the machine learning models in the comparison set, and the framework has to run inference for data-driven baselines, so the model stack sits inside the same evaluation harness rather than being pre-computed and shipped as data. That keeps the comparison honest, because a baseline that is slow to run is scored on the same footing as one that is fast, rather than being given a fixed time budget in advance.

Installing the package and its optional extras

The package is published as weatherbench2 and installed from PyPI. The setup.py file declares the base dependency set and a set of extras, so a first install is best done with the extras you actually need rather than the full set.

The base requirements look like this:

base_requires = [
    'apache_beam>=2.31.0',
    'cftime>=1.6.2',
    'jax[cpu]',
    'numpy>=2.1.3',
    'pandas>=2.2.3',
    'scipy',
    'scikit-learn',
    'xarray>=2024.11.0',
    'xarray-beam',
    'zarr',
]

The extras are the interesting part. There is a tests extra with absl-py, pytest and pyink, plus a pinned numcodecs below 0.16.0 that the source comments identify as a workaround for a zarr-python issue. There is a docs extra with myst-nb, myst-parser, sphinx and the sphinx_rtd_theme. There is a gcp extra adding apache_beam with the gcp extra and gcsfs, which is what you need for the Dataflow path rather than a local runner. And there is a viz extra containing matplotlib, so plotting is not part of the core install.

That separation matters more than it looks. A user evaluating a forecast on a laptop never needs the gcp extra, and a user who only wants the metrics does not need the model stack for inference.

The repository also carries a pyproject.toml whose only configuration is for pyink, the formatter, with line-length 80, pyink-indentation 2 and pyink-use-majority-quotes set to true, and a conftest.py at the root. The package itself lives in the weatherbench2/ directory, with scripts/ holding the command line entry points and docs/ the Sphinx source.

Running a first evaluation from the quick-start

The documented entry point for a first run is the quick-start page, which the README points at for exploring the basic functionality, with the API docs for more detail. There is also a Colab notebook linked directly from the README, at docs/source/evaluation.ipynb, which is the fastest way to see the framework operate on a small sample without setting up a local environment.

That notebook is a sensible first step because the framework has a lot of moving parts: a ground truth dataset, a baseline dataset, a forecast to evaluate, metrics computed over a verification region and lead times, and a choice of interpolation between the forecast grid and the target grid. The quick-start is where the order of operations is explained, and the API reference is where the object model is.

The command line scripts are the other route, and they are the ones built for scale. Rather than driving evaluation from a Python session, you run a script that opens the forecast and target as xarray datasets, applies the metric, and writes the result out. Because the scripts are Beam based, the same invocation can be pointed at a local runner or at Dataflow with the gcp extra installed.

The metrics themselves are not described in detail in the README, and that is deliberate rather than an omission. The paper is stated to describe the rationale behind the evaluation setup, and the specification of which metric is computed on which variable, over which region and lead times, and against which baseline, is the thing that makes two papers comparable. The framework is the shared implementation of that specification, so the documentation lives with the methodology rather than in a list of function signatures.

WeatherBench-X is the codebase the project now points at

The most important line in the README is a banner at the very top: WeatherBench-X has been released, it provides an updated version of the evaluation code base, and users are encouraged to switch to the new codebase. The data guide is explicitly still up to date.

That distinction is useful. The datasets and the data guide remain the shared reference, while the evaluation code is where the successor lives. A team starting a new evaluation today should read that notice before reading anything else, because the code in this repository is the earlier generation.

The release history here is short and old. The tagged releases are v0.2.0 from 2023-12-04 and v0.1.0 from 2023-08-31, while setup.py declares version 0.2.1, so the packaged version has moved past the last tag. The repository is not archived and the last push was on 2026-10-20, so the tree is still being touched even though the release line has not moved in years.

There is also a submission path. The README mentions a guide for those who want their model to be part of WeatherBench, which is how the leaderboard gets populated. If you are building a model and want a comparable published score, that guide is the entry point, and it is worth reading before you spend a week building the evaluation pipeline yourself.

Licence, citation and who this is for

The licence is Apache 2.0, and the README includes the full Apache 2.0 notice with a 2023 Google LLC copyright. The README also states plainly that this is not an official Google product, which is the standard disclaimer for a Google Research repository and worth reading as a signal about support commitments rather than about the code.

A benchmark carries an unusual obligation: if you use it, you cite it. The README provides the BibTeX entry, with the title WeatherBench 2: A benchmark for the next generation of data-driven global weather models, the arXiv eprint 2308.15560, the physics.ao-ph primary class, and a long author list from Stephan Rasp and Stephan Hoyer through to Fei Sha.

The point of the citation is not politeness. The framework's value is that everyone uses the same definitions, and that shared property only holds if the papers that report scores say which definitions they used. A score computed with a different metric is not comparable, and the citation is what lets a reader check that.

The audience is narrower than the generic phrase machine learning benchmark suggests. This is global, that is whole atmosphere, weather model evaluation. It is not a general time series library, not a downscaling tool, and not a local nowcasting benchmark. A team working on a regional model, or on radar nowcasting, or on a single site, will find the evaluation design aimed at a different problem than theirs.

Where WeatherBench 2 is the wrong tool

The clearest mismatch is the scale of the problem. The framework is built for global forecasts evaluated on a global grid against global ground truth. If your task is a regional model, a local nowcast, or a single site, the machinery of a global verification setup, the interpolation to a common grid and the cloud-optimized dataset layout will cost you more than it gives you.

The second mismatch is that you may not need a framework at all. If you want a root mean square error between your forecast and a reference series, xarray and xarray with a couple of lines of arithmetic will do it, and it will be far easier to reason about. WeatherBench 2 earns its complexity when the evaluation has to be reproducible by someone else, when you are comparing against published baselines, or when the data volume makes a distributed runner necessary.

The third is the successor problem. The project itself tells you to move to WeatherBench-X for evaluation code. Choosing this repository means choosing the older generation deliberately, and the honest reason to do so is compatibility with an existing codebase or with a published baseline set.

The alternative worth understanding is not a different Python package, it is a different evaluation strategy. If your goal is a defensible comparison, running your model through the current WeatherBench-X codebase and contributing the result to the shared leaderboard gives you a score that others can trust, and it costs you less work than reproducing the pipeline here. WeatherBench 2 remains the reference for the data and the rationale, and the newer codebase is where the evaluation should happen.

Editorial conclusion

weatherbench2 is worth adopting if you need a score for a global forecast model that someone else can reproduce, because the ground truth, the baseline data and the evaluation code are public and the same code path scales to high resolution forecasts on Dataflow. It is the wrong tool for a regional or nowcasting problem, for anyone who only needs a single RMSE against their own reference, and for a new project, since the project itself recommends switching to WeatherBench-X. Before building on it, read the WeatherBench-X notice at the top of the README, pick a version of the ground truth from the data guide, and reproduce one of the published baseline scores with the command line scripts before you trust your own evaluation numbers.

Frequently asked questions

Official sources

1. google-research/weatherbench2 on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases