A call for prediction-domain adaptive validation for assessing the performance of spatial prediction models

Spatial predictive mapping is widely used in geoscience to generate spatially explicit maps from limited field observations and holds particular significance for soil mapping. In this approach, point-based observations are linked to spatially continuous predictor variables, and (due to expected nonlinearity) machine learning algorithms are often employed to learn their relationships and produce spatial predictions. A key challenge, however, is assessing the quality and reliability of the resulting maps.

While there is consensus that map accuracy is ideally assessed using an independent probability sample from the prediction area, such data are often unavailable. Consequently, practitioners commonly rely on splitting the available observations into training and testing sets or on repeated data partitioning via cross-validation. The resulting performance statistics are used to obtain a proxy for the final map accuracy, where cross-validation additionally supports model tuning and selection. In recent years, a considerable debate has emerged regarding how data should be partitioned into training and test sets during cross-validation. Studies have shown that estimated performance metrics can differ substantially depending on the chosen data-splitting strategy, for example, whether observations are split randomly or according to spatial structures, such as in spatial cross-validation approaches that partition data by spatial units (e.g., spatial block cross-validation or leave-region-out schemes). While some researchers argue that random cross-validation is inappropriate because it yields overly optimistic performance estimates, others contend that spatial cross-validation can be overly pessimistic and therefore advocate for random validation instead.

We argue that both spatial and random validation approaches can provide appropriate proxies for map accuracy, but their suitability depends on how well they align with the specific prediction context. Many spatial prediction tasks involve a combination of interpolation and extrapolation in geographic space, feature space, or both, and the chosen cross-validation strategy should explicitly account for this. To address this, we propose a new category of cross-validation methods, termed prediction-domain adaptive validation. Methods in this category flexibly adapt data partitioning to reflect the underlying prediction task, ensuring that validation data resemble the intended prediction scenario. To illustrate the potential of these new methods, we reproduce a simulation study, compare different validation methods, and discuss their purpose.

We show that random cross-validation methods are suitable when training samples are randomly distributed across the prediction area, whereas spatial cross-validation is better suited for extrapolation-dominated scenarios. In practice, however, most applications fall between these two extremes. In such cases, prediction-domain adaptive cross-validation can provide more reliable estimates of map accuracy, as it explicitly adapts to the underlying prediction situation. We believe that the proposed prediction-domain adaptive validation approach helps consolidate the ongoing discussion on various strategies by providing a balanced approach that yields more suitable estimates of map accuracy during cross-validation. This, in turn, supports model tuning and enhances the quality of the resulting maps, such as those generated during digital soil mapping.