Estimating location errors in precipitation forecasts with the Wasserstein and Attribution distances

Due to the strongly fluctuating nature of precipitation both in space and time, there is a need for high-resolution forecasts in order to provide accurate information for user applications. However, as we transition to high-resolution forecasts, the predictability limits imposed by the physics of convective motions render traditional verification techniques less effective for measuring forecast quality. While high-resolution models might be able to realistically simulate convective motions and their associated precipitation, the exact location of the updrafts and the surface precipitation cannot be determined precisely. This poses a problem with classical point-to-point verification techniques such as root mean squared error (RMSE), because any displacement of the precipitation in the forecasts will result in a double penalty. There are three specific problems with RMSE when there are location errors: a) forecasts that have less variability than observations score better than misplaced but realistic forecasts in those cases, favouring unrealistic solutions; b) low-resolution forecasts can score better than more realistic high-resolution ones; and c) forecasts where an observed feature is misplaced but nearby receive the same score than forecasts that are misplaced and farther away.

Measuring the location error, i.e. the distance between precipitation spots in forecast and observation fields is an intuitive way to address the third issue. However, to measure the displacements, one needs to have an assignment between features in the forecasts and the observations. The Wasserstein distance, defined as the minimum displacement over all possible assignments, is a theoretical way forward. However, computing it is prohibitively expensive. Fortunately, there has been growing interest among the machine learning community in utilizing Wasserstein distances to circumvent too literal comparisons. As a result, new algorithms have been developed that can approximate Wasserstein distances and scale linearly with respect to the number of points to be assigned. In this presentation, we demonstrate the practical application of two fast approximate algorithms, namely the Flowtree and the Attribution distance methods, for measuring location errors. Both methods are very flexible, allowing the computation of location errors on gridded or unstructured datasets, even on the spheric geometry of the Earth. We showcase the utilization of these novel verification metrics in specific use cases with ECMWF forecasts to highlight their strengths and weaknesses.