ExtremeWeatherBench 1.0: A Flexible Evaluation Framework for Extreme Weather Events

Properly evaluating AI and NWP models before deployment will help to ensure that the final models are trustworthy. Currently, most evaluation is done at a global scale, such as with WeatherBench, rather than focusing on high-impact events. While this global evaluation is important, it can obscure the results of how a model performs on high-impact events. For example, a heat wave may be poorly forecast by one model but the model may look promising overall when examining global Root Mean Squared Error. Only by examining specific case studies do we get the bigger picture of how the model performs on phenomena that impact humanity around the world.

We introduce Extreme Weather Bench (EWB), a new community driven benchmarking suite with almost 300 case studies of high-impact weather events across the globe. EWB facilitates model validation and verification on a variety of high-impact hazards that matter to people around the globe. EWB provides a standard set of case studies (spanning multiple spatial and temporal scales and different parts of the weather spectrum), observational data, impact-based metrics, and open-source code for users to evaluate their models. The case studies include tropical cyclones, atmospheric rivers, convective weather outbreaks, heat waves and major freeze events. To facilitate ease-of-use, EWB is distributed as a pure Python package, and integrates with either local data or data saved on the cloud.

EWB will help to drive the science forward for all weather models, enabling true comparisons across models and enabling people to evaluate their models on specific high-impact phenomena while diving deeply into case studies. EWB is a free open-source community-driven system and will be adding additional phenomena, test cases and metrics in collaboration with the worldwide weather and forecast verification community.