Quantifying double-penalty effects and mitigating their impact on forecast verification

Location errors in precipitation forecasts are ubiquitous in high-resolution weather forecasts due to the misplacement of convective cells but also of mesoscale or synoptic-scale features such as convergence lines or low-pressure systems. However, those kinds of forecast errors never appear in isolation and are usually mixed with intensity errors and more generally with systematic model biases. Therefore, correctly disentangling the contribution of location errors in verification scores is challenging, but also essential to advance forecast quality due to a couple of reasons. Firstly, location errors will incur a double penalty in traditional point-by-point verification (one false alarm event and one missed event). As a result, traditional metrics have the undesirable property of penalizing more a forecast with a correct feature in the wrong location than a forecast that misses the feature. In second place, once a forecast has a location error, traditional metrics are insensitive to the magnitude of the displacement. Hence, traditional metrics are not good at detecting improvements in the size of location errors (i.e. they lack discrimination). Both problems imply that intrinsically better forecasts do not necessarily get better scores.

 

Here we showcase some novel ways to tackle those two issues from different perspectives. Regarding the first issue, we present a new decomposition of the Mean Squared Error into three positive definite terms, one of which is linked to the amount of double penalty. Then we show how this allows screening forecasts for their levels of double penalty.

Regarding the second issue, recent advances in spatial verification techniques have enabled estimating location errors of global precipitation forecasts by approximating the Wasserstein distance between unbiased fields. This technique constructs a transport plan to move all the precipitation water in the forecasts to the correct locations according to observations, from which a mean location error is computed. We have extended this technique to be able to work with biased fields and at the same time curtail the location errors to prevent large displacement errors that result from biases in regions far away. This enables us to detect improvements at global or regional scale in ECMWF forecasts.

Finally, we have also shown that there is a relationship between the travel plan employed to compute the mean location error and the Mean Absolute Error. This allows us to go back to the first issue again and assess what proportion of MAE is contributed by location errors of a certain scale.