Diagnostic efficiency - a diagnostic approach for model evaluation

A better understanding of what is causing the performance of hydrological models to be “poor” or “good” is crucial for a diagnostically meaningful evaluation approach. However, current evaluation efforts are mostly based on aggregated efficiency measures such as Kling-Gupta Efficiency (KGE) and Nash-Sutcliffe Efficiency (NSE). These aggregated measures allow to distinguish between “poor” and “good” model performance only. Especially in case of “poor” model performance it is important to identify the errors which may have caused such unsatisfying simulations. These errors may have their origin in the model parameters, the model structure, and/or the input data. In order to provide insight into the origin of the error, we define three types of errors which may be related to the source of error: constant error (e.g. caused by consistent precipitation overestimation), dynamic error (e.g. caused by deficient vertical redistribution) and timing error (e.g. caused by precipitation or infiltration routine). Based on these types of errors, we propose the novel Diagnostic Efficiency (DE) measure, which accounts for the three error types by representing them in three individual metric components. The disaggregation of DE into its three metric components can be used for visualization in a 2-D space using a diagnostic polar plot. A major advantage of this visualization technique is that regions of error terms can be clearly distinguished from each other. In order to prove our concept, we first systematically generated errors by mimicking the three error types (i.e. simulations are calculated by manipulating observations). Secondly, by computing DE and the related diagnostic polar plots for the mimicked errors, we could supply evidence of the concept. Moreover, we tested our approach for a real case example. For this we used the CAMELS dataset. In particular, we compared streamflow simulations of a single catchment realized with different parameter sets to the observed streamflow. For this real case example the diagnostic polar plot suggests, that dynamic errors explain the model performance to a large extent. With the proposed evaluation approach, we aim to provide a diagnostic tool for model developers and model users. Particularly, the diagnostic polar plot enables hydrological interpretation of the proposed performance measure.