From Forecast Skill to Forecast Value: Do AI Weather Forecasts Deliver Real-World Economic Benefits?

Recent years have witnessed rapid advances in data-driven weather forecasting, with an ever-increasing number of AI-based models reporting skill comparable to or exceeding that of physical models. Comparing AI and physical forecasting systems, however, remains challenging: these models often exhibit a different set of strengths and weaknesses, making their real-world value strongly dependent on the specific application. Yet, most existing comparisons of AI and physical models focus exclusively on meteorological skill, largely overlooking the question of forecast value in real-world decision-making.

 

In this talk, we tackle this question by proposing an application-dependent framework to evaluate the real-world value of AI weather forecasts. The framework is based on the classical concept of relative economic value, which we extend in several novel ways to better reflect realistic use cases. Besides allowing for varying cost–loss ratios to represent different protection and forecast costs, we introduce flexible penalty functions to account for compounding losses from sequential forecast misses as well as declining user trust due to repeated false alarms.

 

We apply the framework to a number of case studies, comprising cities exposed to high economic losses from weather-related natural hazards. We show that forecast value in these contexts depends not only on forecast and prevention costs, but also on the choice of penalty function and on whether compound losses from repeated misses or false alarms are considered. We thus advocate for evaluating real-world value alongside meteorological skill when developing and comparing forecasting models, to ensure that improvements in predictive accuracy translate into meaningful societal and economic benefits.