Assessment of European, North American and African pollen-climate calibration models in h-block cross-validations

Numerous quantitative calibration methods have been developed for preparation of quantitative paleoclimate reconstructions from microfossil proxies. Here we evaluate the performance of seven calibration approaches, using h-block cross-validations (Telford & Birks 2009) of calibration models fitted to (sub-)continental scale datasets of modern pollen assemblages and climate. We use a total of seven calibration methods falling into three methodological families: three classical unimodal methods including weighted averaging (WA), weighted averaging-partial least squares (WAPLS), and maximum likelihood regression curves (MLRC); three machine-learning methods based on regression-tree ensembles including random forest (RF), extremely randomized trees (ERT), and the boosted regression tree (BRT); and the modern-analogue technique (MAT) based on matching fossil assemblages with modern pollen samples.

Using ecologically grounded, regionally selected climate variables, we prepared h-block cross-validations in four regions: Northern Europe (July and January temperature), Southern Europe (January temperature), eastern North America (July temperature and annual water balance), and Africa (annual water balance). The cross-validations were run with a range of h values from 0 to 1500 km, to assess the performance of the models with a gradually diminishing pool of modern analogues (Salonen et al. 2019). The cross-validation performance was evaluated using the root-mean-square error of prediction (RMSEP), maximum bias, and the coefficient of determination (R²).

In our results (Fig. 1) we find the machine-learning methods (BRT, ERT, and RF) to be the three best-performing (lowest RMSEP) approaches at moderate h values in all cases except in Africa (Fig. 1F) where WA performs best, perhaps due to the robustness of the parametric modelling approach of WA with the more spatially clustered modern pollen data in Africa. A distinct pattern is observed with MLRC, with relatively high RMSEP values but often clearly the lowest maximum bias (Fig. 1C,E,F). In general, the between-method differences are considerably greater in maximum bias than in RMSEP. In all cases here, the maximum bias figures represent bias towards the modern gradient mean at either gradient end. Hence the maximum bias can be highly relevant in cases where paleo-reconstructions must be prepared from environments similar to the modern gradient end.

Our work highlights the practicality of the variable-radius h-block cross-validation approach. While the merits of h-block cross-validation are well argued, the method involves the challenge of selecting a suitable h, representing a balance between removing pseudoreplicate samples and an excessive loss of modern analogues (Trachsel & Telford 2016). By running cross-validations at a range of h, we find that after an initial loss of performance with increasing h, the performance tends to plateau at h of about 200–1000 km. This allows the identification of calibration methods that perform robustly at a range moderate h values.

Figure 1. Performance of calibration methods with different datasets. Methods are ranked based on increasing RMSEP in h-block cross-validations, using an h of 200 to 600 km depending on dataset.

References

Telford RJ & Birks HJB (2009) Quat. Sci. Rev. 28:1309–1316. https://doi.org/10.1016/j.quascirev.2008.12.020

Trachsel M & Telford RJ (2016) Clim. Past 12:1215–1223. https://doi.org/10.5194/cp-12-1215-2016

Salonen JS et al. (2019) Sci. Rep. 9:15805. https://doi.org/10.1038/s41598-019-52293-4