Uncertainty quantification for data-driven weather prediction: From probabilistic forecasts to fair model comparisons

Artificial intelligence (AI)-based data-driven weather prediction (AIWP) models have experienced rapid progress over the last years. They achieve impressive results and demonstrate substantial improvements over state-of-the-art physics-based numerical weather prediction (NWP) models across a range of variables and evaluation metrics. However, most efforts in data-driven weather forecasting have been limited to deterministic, point-valued predictions, making it impossible to quantify forecast uncertainties, which is crucial in research and for optimal decision making in applications.

I will present recent work on uncertainty quantification (UQ) methods in the context of data-driven weather prediction. The post-hoc use of UQ methods enables the generation of skillful probabilistic weather forecasts from a state-of-the-art deterministic AIWP model [1]. Further, by subjecting the deterministic backbone of physics-based and data-driven models post hoc to the same UQ technique, and computing the in-sample mean continuous ranked probability score of the resulting forecast, we propose a new measure that enables fair and meaningful comparisons of single-valued output from AIWP and NWP models, called potential continuous ranked probability score [2].

References

[1] Bülte, C., Horat, N., Quinting, J. and Lerch, S. (2025). Uncertainty quantification for data-driven weather models. Artificial Intelligence for the Earth System, in press. DOI:10.1175/AIES-D-24-0049.1

[2] Gneiting, T., Biegert, T., Kraus, K., Walz, E.-M., Jordan, A. I., and Lerch, S. (2025). Probabilistic measures afford fair comparisons of AIWP and NWP model output. Preprint, arXiv:2506.03744. DOI:10.48550/arXiv.2506.03744