Which model do you choose? Effects and drivers of climate model selection

A model is always a simplification of reality, in particular for systems as complex as the climate. Therefore, building models requires choices, including simplifications and assumptions. However, many such choices remain epistemically underdetermined, meaning that no scientific reason per se dictates a choice, be it between two model resolutions or a one- and two-moment cloud microphysics scheme. In other words, there is generally uncertainty around what option is best, but in setting up a model, one has to commit to one option. Moreover, there are consecutive model versions. Newer versions are generally supposed to lead to improvements in the representation of processes and/or the similarity of historical simulations and observed climate. For example, newly added modules or higher resolutions permitting convection may be considered a step change in model development. However, these changes do not always improve results. At least when combining metrics and considering match to observations, physical basis and usability, there is no clearly superior climate model. Thus, when choosing an already-built model or model version as a basis of a scientific study, this choice itself is also epistemically underdetermined.

Seeing that choices in setting up and choosing a model are epistemically underdetermined, yet need to be made, what is their effect on conclusions, and how are such choices made? To address these questions we conducted two separate analyses.

First, we address the effect of choices in model construction with a variance analysis of CMIP models, comparing variances due to model choice, model version choice, and forced climate response over a diverse range of output variables. We find that for many variables, much of the inter-generational ensemble variance origins from variance between different versions of one and the same model. Variance from historical climate change between 1979 and 2005, as represented in AMIP simulations of 29 models, is negligible in almost all 36 investigated global annual mean variables.

Second, we investigate drivers of model selection. With a bibliometric analysis of more than 7000 papers we show a strong correlation between the model used in a study and the study’s first author's institution. In other words, institutions display model attachment, with a median attachment of over 60 % to their favorite model. This shows that model selection is largely driven by context rather than by epistemic considerations.

That model choice is influenced by contextual factors but matters for study conclusions motivates further exploration of model variants, for example using perturbed parameter ensembles to explore the space of possible models. The institutional attachment shows how model space is currently sampled unequally: each institute largely samples only part of the model space. When single institutes concentrate research power, other parts of the model space remain undersampled. We give examples of ways to address these issues: the climate-scientific community could acknowledge contextual factors and study their effects, reconsider choices where pragmatically possible, and strengthen efforts to identify where results may generalise beyond the specific model or model version and the contextual factors in effect.