Valid Prediction Intervals for Weather Forecasting with Conformal Prediction

In recent years, machine learning has emerged as a promising alternative to numerical weather prediction models, offering the potential for cost-effective and accurate forecasts. However, a significant limitation of current machine learning methods for weather forecasting is the lack of principled and efficient uncertainty quantification—a key element given the complexity of the Earth's climate system and the challenges in modeling its processes and feedback mechanisms. Inadequate uncertainty quantification and reporting undermines trust in and the practical use of current weather forecasting methods (Eyring et al., 2024).

Uncertainty quantification methods for weather forecasting typically use prediction intervals and can be categorized into Bayesian and frequentist approaches. Bayesian methods, while theoretically appealing, often involve restrictive assumptions and do not scale well to the complexity of spatio-temporal data. Frequentist approaches, such as ensemble-based methods, are widely used in weather forecasting and include techniques like perturbing initial states with noise (Bi et al., 2023; Scher et al., 2021), varying neural network parameters (Graubner et al., 2022), or training generative models (Price et al., 2023). However, most frequentist methods provide only asymptotically valid prediction intervals, which may not suffice in all weather forecasting applications.

Conformal prediction (CP) is a promising uncertainty quantification framework that delivers valid and efficient prediction intervals for any learning algorithm, without requiring assumptions about the underlying data distribution (Vovk et al., 2005). Despite its growing popularity in the machine learning and statistics communities, traditional CP methods are not tailored to spatio-temporal data in weather forecasting. This is due to challenges arising from spatial and temporal dependencies—such as spatial autocorrelation and temporal dynamics—that violate the exchangeability assumption underlying standard CP methods. Several recent studies attempted to address these challenges by introducing new CP algorithms specifically designed for various types of non-exchangeability (Oliveira et al., 2024). However, these adaptations face several limitations, including high computational complexity, asymptotic guarantees, and/or the need for recalibration of prediction intervals.

In this presentation, we will evaluate CP methods in the context of weather forecasting and discuss several limitations. In addition, we will highlight recent advances and discuss potential future directions that could address challenges underlying the use of CP in weather forecasting.

References:

Eyring, V., et al. Pushing the Frontiers in Climate Modelling and Analysis with Machine Learning. Nature Climate Change, 2024.

Leutbecher, M., et al. Ensemble Forecasting. JCP, 2008.

Bi, K., et al. Accurate Medium-range Global Weather Forecasting with 3D Neural Networks. Nature, 2023.

Scher, S., et al. Ensemble Methods for Neural Network-based Weather Forecasts. JAMES, 2021.

Graubner, A., et al. Calibration of Large Neural Weather Models. NeurIPS, 2022.

Price, I., et al. Probabilistic Weather Forecasting with Machine Learning. Nature, 2025.

Vovk, V., et al. Algorithmic learning in a random world. New York: Springer, 2005.

Oliveira, R.I., et al. Split Conformal Prediction and Non-exchangeable Data. JMLR, 2024.