Separation of climate models and observations based on daily output using two machine learning classifiers

Climate models are primary tools to investigate processes in the climate system, to project future changes, and to inform decision makers. The latest generation of models provides increasingly complex and realistic representations of the real climate system while there is also growing awareness that not all models produce equally plausible or independent simulations. Therefore, many recent studies have investigated how models differ from observed climate and how model dependence affects model output similarity, typically drawing on climatological averages over several decades.

Here, we show that temperature maps from individual days from climate models from the CMIP6 archive can be robustly identified as “observation” or “model” even after removing the global mean. An important exception is a prototype high-resolution simulation from the ICON model family that can not be so  unambiguously classified into one category. These results highlight that persistent differences between observed and simulated climate emerge at very short time scales already, but very high resolution modelling efforts may be able to overcome some of these shortcomings.

We use two different machine learning classifiers: (1) logistic regression, which allows easy insights into the learned coefficients but has the limitation of being a linear method and (2) a convolutional neural network (CNN) which represents rather the other end of the complexity spectrum, allowing to learn nonlinear spatial relations between features but lacking the easy interpretability logistic regression allows. For CMIP6 both methods perform comparably, while the CNN is also able to recognize about 75% of samples from ICON as coming from a model, while linear regression does not have any skill for this case.

Overall, we demonstrate that the use of machine learning classifiers, once trained, can overcome the need for multiple decades of data to investigate a given model. This opens up novel avenues to test model performance on much shorter times scales.