EMS Annual Meeting Abstracts
Vol. 23, EMS2026-469, 2026, updated on 22 Jun 2026
https://doi.org/10.5194/ems2026-469
EMS Annual Meeting 2026
© Author(s) 2026. This work is distributed under
the Creative Commons Attribution 4.0 License.
Oral | Monday, 07 Sep, 11:45–12:00 (CEST)| Room Quest
Leveraging scale dependent accuracy measures for a fair comparison between AI and NWP weather prediction models
Britta Seegebrecht1,2, Sabrina Wahl1, Stefanie Hollborn1, George Craig2, Erik Pavel3, Wael Almikaeel3, Sebastian Buschow3, Martin G. Schultz3, Christian Lessig4, Ilaria Luise4, Anas Al-Iahham5, and Juergen Gall5
Britta Seegebrecht et al.
  • 1German weather service (DWD), Offenbach am Main, Germany (britta.seegebrecht@dwd.de)
  • 2Ludwigs-Maximilians-University Munich, Germany
  • 3Jülich Supercomputing Centre (JSC), Forschungszentrum Jülich, Jülich, Germany
  • 4European Centre for Medium-Range Weather Forecasts (ECMWF), Bonn, Germany
  • 5Lamarr Institute for Machine Learning and Artificial Intelligence, Rheinische Friedrich-Wilhelms- Universität Bonn, Germany

Data-driven weather prediction models based on artificial intelligence (AI) have rapidly advanced in recent years and are frequently reported to outperform traditional physics-based numerical weather prediction (NWP) models for selected verification scores. However, optimization with respect to a specific loss function can adversely affect other metrics, potentially leading to unrealistic forecast characteristics, such as overly smooth spatial structures when mean-squared or mean-absolute error–based loss functions are used.

In Bonavita & Geer, 2026 an orthogonal decomposition of the Mean Squared Error (MSE) into Information Error and Noise Error is introduced to unravel different strategies for minimizing this commonly used accuracy metric. These insights allow for a more meaningful interpretation of the MSE as accuracy measure.

Additionally, a scale dependent analysis of model performance can help to reveal systematic differences between AI and NWP models specifically on smaller scales such as the effective resolution.

The combination of both approaches – the decomposition of the scale dependent MSE into Information and Noise Error – has been derived and applied to different AI and NWP models.

First results show the potential of this method to disentangle different effects allowing for a fairer, more comprehensive comparison between AI and NWP weather prediction models.

The analysis is partly based on forecasts from the Weather Prediction Model Intercomparison Project (WP MIP), which provides a collection of NWP and AI-model forecasts from multiple national weather services and research institutions.

The work is conducted within the RAINA project, which aims to develop a foundation model for the atmosphere with a particular focus on reliable, high-resolution forecasts of extreme wind and precipitation events.

Bonavita, M. & Geer, A.J. (2026) Forecast verification using information and noise. Quarterly Journal of the Royal Meteorological Society, e70109. Available from: https://doi.org/10.1002/qj.70109

How to cite: Seegebrecht, B., Wahl, S., Hollborn, S., Craig, G., Pavel, E., Almikaeel, W., Buschow, S., Schultz, M. G., Lessig, C., Luise, I., Al-Iahham, A., and Gall, J.: Leveraging scale dependent accuracy measures for a fair comparison between AI and NWP weather prediction models, EMS Annual Meeting 2026, Utrecht, Netherlands, 6–11 Sep 2026, EMS2026-469, https://doi.org/10.5194/ems2026-469, 2026.