Forecast Error Diagnostics in Neural Weather Models Using Gridpoint Relaxation

In physics-based numerical weather prediction models, underlying physical laws and numerical discretisation schemes offer some interpretability of the simulated processes and point to potential sources of errors. In contrast, data-driven deep-learning (DL) models lack explicit physics-based interpretation of their predictions. However, the low computational cost and auto-differentiable implementation allow for fast and extensive diagnostics of the origins of forecast errors using well-established explainable AI methods and traditional diagnostics tools. Here, we combine saliency maps and gridpoint relaxation to perform a multivariate regional analysis of the sources of forecast errors in global DL weather forecasting.

We developed ConvCastNet, a DL global weather prediction model based on depth-wise separable convolutional neural networks. The model predicts 6 atmospheric variables at 13 pressure levels and includes 2 additional single-level prognostic and 7 constant input fields. The forecast is computed on a 3-degree lon-lat grid and uses 12-hour autoregressive time stepping to roll out the forecast. ConvCastNet achieves significant success in predicting geopotential at the 500 hPa pressure level, with 8.5 days of useful forecast (based on an anomaly correlation coefficient greater than 0.6), which makes it a suitable tool for diagnostics of the origin of the forecast error.

We use ConvCastNet to systematically nudge subdomains of the forecast fields for 1) planetary boundary layer, 2) stratosphere, and 3) tropics towards a "true" weather state (reanalysis) and monitor the forecast skill improvements beyond selected subdomains. Our results show that an 8-day mid-latitude weather forecast improves significantly with relaxation in the stratosphere, whereas relaxation in the tropics has no significant impact on mid-latitude. This highlights the importance of accurately representing the stratosphere for medium-range weather prediction and the limited impact of tropical variability on mid-latitude forecasts.

Additionally, we investigate the relationship between model error sensitivity to initial conditions and relaxation experiments. By utilising the model's auto-differentiability, we analyse saliency maps, i.e. the gradients of the forecast errors with respect to input fields, to identify overlapping regions of large error sensitivity and high impact of relaxation to the truth. We find the model sensitivity largely consistent with physics-based expectations, with local errors being sensitive to the upstream dynamics and varying sensitivity to different variables and pressure levels. We believe that this combined approach provides valuable heuristics for diagnosing neural model errors and guiding targeted model improvements.