Towards next generation machine learning-based Earth system models that exploit a wide range of datasets

The training of machine learning models for weather and climate on multiple datasets, including local high-resolution reanalyses and level-1 observations, is one of the frontiers of the field. It promises to allow for models that are no longer constrained by the capabilities of equation-based models, as is currently still largely the case when training on global reanalyses. For example, level-1 observations contain the feedback from arbitrary scale processes and hence do not suffer from the closure problem of equation-based models. Training on observations might hence lead to machine learning-based Earth system models with reduced systematic biases, in particular for long-term climate projections. Local reanalyses are only available for a small set of regions, mainly over Europe and North America. Appropriate training might allow one to generalize the detailed process information in these to other regions or even globally. 

In this talk, we present results on the effective training with a combination of global and local reanalysis as well as level-1 observations. We consider different pre-training protocols to learn the correlations between datasets, which is critical to obtain a benefit through their combination. We use a forecasting task as baseline and study the effectiveness of different variants of masked-token modeling and more sophisticated approaches that exploit the latent space of the machine learning models. We also study different fine-tuning strategies to extract a best state estimate from multiple datasets and to generalize regional datasets globally. For this, we build on the extensive results on fine-tuning of large language models that have been developed in the last years. Our results aim to determine general principles which combination of datasets is beneficial. We also perform a detailed analysis of the physical consistency and physical process representation in the model output. Through this, we believe our work provides an important stepping stone for the next generation of machine learning-based models for weather and climate.