Can AI Weather Models Extrapolate Extremes? Evaluating Generalisation to Unseen Precipitation Events

Recent advances in data-driven weather and climate modelling have challenged the traditional dominance of physics-based numerical prediction systems. Machine-learning-based models, such as AIFS, have demonstrated competitive or superior performance for a range of standard forecast metrics. However, these evaluations primarily focus on large-scale variables under typical conditions, while the representation of local, high-impact extreme events remains insufficiently explored.

This study investigates the ability of data-driven models to represent extreme precipitation and to generalise beyond the conditions encountered during training. A graph neural network model, trained using the Anemoi framework, is developed based on output from a convection-permitting (4 km) regional climate simulation with the ALARO model over Western Europe. Model performance is quantified using precipitation return levels derived from annual maxima across multiple accumulation durations, ranging from hourly to multi-day timescales. This experimental design enables a systematic comparison between data-driven and physics-based representations of extremes.

To explicitly assess generalisation, a second model is trained on a modified dataset in which the upper tail of the precipitation distribution is selectively masked. By comparing return levels from both data-driven models with those from the reference simulation, we evaluate the extent to which such models can reproduce extremes that are absent from the training data.

This framework provides a controlled setting to assess the robustness of AI-based models for high-impact precipitation events and their ability to generalise to unseen conditions. This is particularly relevant for extreme precipitation, which is inherently underrepresented in training data and is expected to evolve under climate change, potentially leading to conditions outside the historical training distribution.