Understanding the distribution of multi-model ensembles

Ensembles of model experiments have become the standard tool both in studies of climate change and in studies of prediction on many different time-scales. When analyzing such ensembles the mean of the ensemble is often interpreted as the best estimate and the spread of the ensemble as an estimate of the uncertainty.  Naively we might argue that the error of the ensemble mean would approach zero as the size of the ensemble increases. However, this argument is based on the assumption that the ensemble is centered around the observations - the truth-plus-error interpretation. A competing assumption - the indistinguishable interpretation -- holds that the observations and the models are all drawn from the same distribution.

The rationale for the truth centered interpretation is that it is the situation that would be expected after calibration of statistical models. However, for multi-model ensembles of climate models there is an increasing amount of evidence pointing towards the indistinguishable interpretation. But why should the indistinguishable interpretation hold for an ensemble that basically is a representation of our incomplete knowledge of the climate system?

Here we analyze CMIP5 ensembles focusing on three measures that separate the two interpretations: the error of the ensemble mean relative to the error of individual models, the decay of the ensemble mean error for increasing ensemble size, and the correlations of the model errors. To get more freedom in our analysis we use a simple statistical model where observations and models are drawn from distributions with different variances and which include a bias. The two interpretations can be found as limits of this more comprehensive model  for which analytical results can be found using the simplifying properties of high dimensional space (the blessing of dimensionality).

We find that the indistinguishable interpretation becomes an increasingly better assumption when the errors are based on smaller and smaller temporal and spatial scales. Building on this, we present a simple conceptual mechanism for the indistinguishable interpretation based on the assumption that the climate models are calibrated or tuned on large scale features such as, e.g., annual means or global averages.