Observation-based verification of AI weather prediction models: What can we expect?

Current state-of-the-art artificial-intelligence weather prediction (AI-WP) systems are trained on a large archive of atmospheric reanalysis data. The training objective is to replicate the analysis at a future time step using the previous time steps. Loss functions guide the model to minimize the prediction error on known data. An analysis-based verification of forecasts derived from unseen data will reveal the strength and weaknesses of the AI-WP model in reproducing the statistical and dynamical characteristics of the underlying reanalysis.

In contrast, the development and fine-tuning of traditional physics-based numerical weather prediction (NWP) systems relies on verification against observations, with the aim of reducing discrepancies relative to various observational systems. This fundamental difference raises the question of what to expect when applying observation-based verification to AI-WP models that are trained on reanalysis rather than directly on observations.

Reanalysis datasets have well-known errors with respect to observations which are documented in literature. Consequently, observation-based verification of AI-WP systems will inherently reflect the observational error characteristics of the reanalysis. Deviations from this expectation are particularly informative: a larger error than that of the reanalysis may indicate deficiencies in emulation, whereas a smaller error raises the question of whether, and from where, additional information beyond the reanalysis has been obtained.

To address these questions, we apply the multiple correlation decomposition based on partial correlations introduced by Glowienka-Hense et al. (2020). This method decomposes the explained variance of two different datasets with respect to the same observations into a component of information contained in both datasets (shared explained variance) and the respective added values, i.e., information present in one dataset but not in the other. This decomposition enables quantification of the information transferred from the reanalysis into the forecasts and reveals potential deficiencies, or improvements relative the reanalysis, in the training process. Furthermore, it facilitates comparison of different forecasting systems in terms of there shared and unique information content. The method is demonstrated using 2m-temperature station observations and global deterministic AI-WP and NWP forecasts.

Glowienka-Hense et al. (2020): Comparing forecast systems with multiple correlation decomposition based on partial correlation, ASCMO, 6, 103–113, https://doi.org/10.5194/ascmo-6-103-2020