Assessing skills of the ULYSSES global multi-model hydrological seasonal prediction system

It is a well-known fact that multi-model forecast systems provide greater reliability over single-model systems, as hydrological models have a solid contribution to forecast uncertainty [1]. Yet many prevalent skill scores for verification of forecasting systems are calculated relative to a benchmark skill. Benchmark skills across hydrological models could vary largely because the benchmark simulation is biased. This bias is not accounted for in the benchmark skill. For example, hydrological models with a high auto-correlation in their state variables tend to have increased skill but do not necessarily have the lowest error when compared against observations [2]. Thus, the correct interpretation of skill can only be conducted if the models are compared against the ground truth, i.e. observations.

With this outlook, we assess the performance of ULYSSES [3] - the first seamless global multi-model hydrological seasonal prediction system. Using four state-of-the-art hydrological models (Jules, HTESSEL, mHM, and PCR-GLOBWB), the production chain utilizes identical land surface datasets (e.g. DEM, soil properties) and forecast inputs for all HMs, and the same river routing scheme (i.e., the multi-scale Routing Model; mRM). The system initializes based on the ERA5-Land dataset, and the seasonal forecasts are driven by a 51-member ensemble generated by the ECMWF seasonal forecasting system 5.

The skill assessment includes the verification of seasonal streamflow forecast at 2400+ GRDC gauges distributed globally during the period from 1993 to 2019, at a monthly time scale. The set of skill scores considered includes metrics concerning monthly observations (Kling-Gupta efficiency skill score, i.e., KGESS, KGE components, Equitable Threat Score for droughts, relative bias), skills with reference to benchmark run (CRPSS) and skills on forecast characteristic (forecast extremity, spread). On average, the system was found to have the skill (monthly KGESS) at most for two months. At the lead of 1 month, mHM exhibits KGESS of 0.56, HTESSEL has KGESS of 0.5, Jules 0.48, and PCR-GLOBWB 0.46. A KGESS value of one corresponds to a perfect forecast. Evaluating the median KGE r component (or Pearson's correlation), mHM (0.68), PCR-GLOBWB (0.59), HTESSEL (0.59) and Jules (0.57). The percentage of gauges with positive KGESS is distributed evenly with PCR-GLOBWB (84.9 %), HTESSEL (84.2 %), Jules (82.4 %) and mHM (80.5 %). Model performances over median skill and gauges with positive skill indicates models to have contrasting performance at high and low skill gauges. Besides, the spatial distribution of KGESS shows marked seasonal changes in the skill of the hydrological models. All of this provides insights on the strengths and weaknesses of the models for further improvement of the system.

In the future, the skill assessment would be expanded to additionally compare forecasted fluxes and state variables (e.g., terrestrial water storage anomalies, soil moisture) against other observations such as GRACE, SMOS, etc. All ULYSSES outputs will be made available in the Copernicus Climate Data Store [4] and will be open access. We aim to engage institutions and researchers around the world that are willing to evaluate the forecasts model performance to improve the system in the future.

[1] https://doi.org/10.1175/BAMS-D-17-0274.1

[2] https://doi.org/10.1175/JHM-D-18-0040.1

[3] https://www.ufz.de/ulysses

[4] https://cds.climate.copernicus.eu