Consistency in Model Performance as a Criterion for Trustworthy Hydrological Modelling

Various models are available to hydrologists, including models of different structures, spatial and temporal discretisation, or multiple parameter sets of a single model. But the "trustworthiness" of these models is called into doubt when they reproduce runoff equally well in the calibration period (equifinality), but diverge in their simulation outputs outside this period. A common way to account for modelling uncertainty is to use so-called ensembles that combine several model members. However, it has been debated/discussed that models that do not provide “the right answers for the right reasons” and, consequently, yield poor performance in a prediction or forecasting mode, should be omitted from such ensembles. Various evaluation protocols aimed at detecting such models have emerged over the years, however, this remains an open research question, and more research is needed especially in the context of shifting hydrologic regimes in a changing climate.

 

Adopting the consistency in model performance in reproducing runoff as an additional criterion to select among multiple models emerges as a plausible way to identify the most “trustworthy” ones. We propose an approach that relies on detailed analyses of model performance across subperiods of increasing length contained within the calibration period. A good performance in both short and longer subperiods is crucial as the former can be quite extreme (e.g., extremely dry or wet), while the latter “expose” a model to various hydroclimatic conditions. To analyse the consistency in model performance, an efficiency measure (e.g., the Kling-Gupta coefficient, KGE) can be computed in each subperiod, and each model can be ranked in each subperiod according to the measure. Models yielding the most consistent and the highest performance can then be selected either (1) as a certain percentage of models with the highest rank averaged across all subperiods, or (2) by imposing a rank threshold that has to be reached in every subperiod. We here further propose to additionally evaluate the selected subset of consistent and high-performing models over an independent period using various other performance indicators (e.g., Nash-Sutcliffe coefficient or volumetric efficiency) as well as model ability to reproduce hydrological signatures (e.g., mean, high and low flows, or runoff dynamics). The evaluation performance of the selected models can then be compared to the best (reference) model obtained from the calibration over the full calibration period with the selected efficiency measure (here KGE) as the objective function.

 

To showcase the advantages of the proposed approach, it is here applied to two different models (3DNet-Catch and GR4J) each with 20,000 randomly sampled parameter sets in three unimpaired catchments. In addition to the promising results, the proposed approach is characterised by its ease-of-use and flexibility, i.e., it can be implemented with any ensemble of models (e.g., randomly selected parameter sets of a single model, or different models created e.g., from a modular framework), or with any other aspect of model performance.