Comparing different versions of the continuous ranked probability score to account for forecast or observation uncertainty

Recent studies have shown that probabilistic forecasts are superior to
deterministic forecasts in terms of quality, reliability, and representing the
uncertainty of future states. One of the most well-known and widely used
tools for assessing the performance of (probabilistic) forecast systems is the
continuous ranked probability score (CRPS). This metric is employed to
evaluate the forecasting system when only forecast uncertainty is
considered. In addition to multiple sources of uncertainty in a forecasting
system (such as initial conditions, model structure and parameters, and
boundary conditions), the uncertainty can also originate from observations
(e.g., streamflow). However, this uncertainty, which has rarely been
explored in previous research, should also be regarded in evaluating the
forecasting system. A version of the CPRS is redefined and analyzed to
overcome this important flaw, considering the observation's uncertainty. To
estimate the uncertainty associated with streamflow observations, the
Bayesian Rating curve method (BaRatin) is utilized. This study focuses on
comparing the different versions of the CRPS in considering the
uncertainties of forecasts and observations. Three types of streamflow
forecasting systems are used in this study: deterministic forecasts, raw
ensemble forecasts (applying meteorological ensemble forecasts as inputs to
the hydrological model), and post-processed ensemble forecasts (postprocessing
of hydrological model outputs using weighted ensemble dressing
method). The assessment is performed for short-term forecasts (lead times of
1 to 5 days) for the Au Saumon watershed in southern central Quebec,
Canada. It is found that considering observation uncertainty has a significant
effect on the values of CRPS compared to when only forecast uncertainty is
considered. In addition, CRPS changes in probabilistic forecasts are more
than deterministic ones. Our results also point out that using the modified
version of the CRPS can help end-users better understand and evaluate their
forecasting system.