How to use the fractions skill score for ensemble forecast verification

This study evaluates and demonstrates how to apply the Fractions Skill Score (FSS) for probabilistic forecast verification. The FSS is a spatial verification score originally designed for deterministic forecast verification. It is a neighborhood method frequently used to verify intermittent forecast fields, such as precipitation, that suffer from double penalty errors. Although the FSS was not designed for probabilistic verification, it is frequently used for verifying ensemble forecasts. However, as we show, systematic differences can occur depending on how an ensemble-based FSS is computed. 

Our study compares and evaluates four potential approaches for computing an FSS for ensemble forecasts. We study the dependence of these four approaches on different parameters, such as ensemble size, neighborhood size, or the forecast event frequency of occurrence. The dependence on ensemble size is examined using various subsamples of a large ensemble. Our comparison shows that the behavior of the FSS with ensemble size can vary greatly depending on the approach used to compute the score. Our experiments explore unique convective-scale 1000-member ensemble forecasts and precipitation over Germany for a high-impact summer weather period. For verification, we use random independent members as synthetic observation to minimize the effect of systematic errors, such as biases between the model and observation. In addition, we introduce a probabilistic believable scale and study its dependence on ensemble size. This evaluation highlights that a suitable ensemble size depends on the forecast event frequency. Our study can guide researchers who want to apply an FSS for ensemble forecast verification, as our findings provide insights on how to compute and interpret ensemble-based FSS results correctly.