Many studies in climate change research rely on error metrics to evaluate the performance of climate models. However, the majority of these studies use only one or two metrics, which can limit the insights obtained from the analysis. This is because each metric evaluates only a specific aspect of the model-data relationship, and important information may be missed if not considered. To gain a more comprehensive understanding of model performance, it is necessary to use multiple error metrics. Doing so can reveal model strengths and weaknesses and provide insights for improving the model. Nevertheless, the choice of metrics should be based on the study's specific objectives and research questions, as different metrics may be more relevant or meaningful in different contexts. This study presents the Bergen Metric, a composite error metric that evaluates climate models' overall performance based on the p-norm framework. This approach utilizes a non-parametric clustering technique to reduce the number of error metrics without losing any relevant information. The research emphasizes the importance of using multiple error metrics to gain a thorough understanding of the model behavior. This study has evaluated 89 regional climate simulations of precipitation and temperature over Europe using 38 different error metrics for eight sub-regions in Europe, providing useful information about the metrics' performance in different regions. Furthermore, the study highlights the possibility of observing conflicting behavior among error metrics while examining a single model, underscoring the need for using multiple error metrics tailored to specific use cases. Overall, the Bergen Metric framework provides a useful tool to assess climate model performance and simplify the interpretation of results from multiple error metrics.

How to cite: Samantaray, A., Mooney, P., and Vivacqua, C.: A Framework to Evaluate the Climate Models, EMS Annual Meeting 2023, Bratislava, Slovakia, 4–8 Sep 2023, EMS2023-222, https://doi.org/10.5194/ems2023-222, 2023.

Due to the strongly fluctuating nature of precipitation both in space and time, there is a need for high-resolution forecasts in order to provide accurate information for user applications. However, as we transition to high-resolution forecasts, the predictability limits imposed by the physics of convective motions render traditional verification techniques less effective for measuring forecast quality. While high-resolution models might be able to realistically simulate convective motions and their associated precipitation, the exact location of the updrafts and the surface precipitation cannot be determined precisely. This poses a problem with classical point-to-point verification techniques such as root mean squared error (RMSE), because any displacement of the precipitation in the forecasts will result in a double penalty. There are three specific problems with RMSE when there are location errors: a) forecasts that have less variability than observations score better than misplaced but realistic forecasts in those cases, favouring unrealistic solutions; b) low-resolution forecasts can score better than more realistic high-resolution ones; and c) forecasts where an observed feature is misplaced but nearby receive the same score than forecasts that are misplaced and farther away.

Measuring the location error, i.e. the distance between precipitation spots in forecast and observation fields is an intuitive way to address the third issue. However, to measure the displacements, one needs to have an assignment between features in the forecasts and the observations. The Wasserstein distance, defined as the minimum displacement over all possible assignments, is a theoretical way forward. However, computing it is prohibitively expensive. Fortunately, there has been growing interest among the machine learning community in utilizing Wasserstein distances to circumvent too literal comparisons. As a result, new algorithms have been developed that can approximate Wasserstein distances and scale linearly with respect to the number of points to be assigned. In this presentation, we demonstrate the practical application of two fast approximate algorithms, namely the Flowtree and the Attribution distance methods, for measuring location errors. Both methods are very flexible, allowing the computation of location errors on gridded or unstructured datasets, even on the spheric geometry of the Earth. We showcase the utilization of these novel verification metrics in specific use cases with ECMWF forecasts to highlight their strengths and weaknesses.

How to cite: Lledó, L., Skok, G., and Haiden, T.: Estimating location errors in precipitation forecasts with the Wasserstein and Attribution distances, EMS Annual Meeting 2023, Bratislava, Slovakia, 4–8 Sep 2023, EMS2023-602, https://doi.org/10.5194/ems2023-602, 2023.

Within the framework of Destination Earth (https://digital-strategy.ec.europa.eu/en/policies/destination-earth) we are running global weather forecasts at the km-scale. A major goal is to improve forecasts of wind for renewable energy purposes, as well as precipitation in extreme weather events. Initialisation of the high-resolution forecasts requires analysis that can be constrained by high-resolution observations.

Our experiments use a relatively denser set of clear-sky radiances from geostationary satellites. In assimilating the clear-sky observations, we simultanously vary the spatial & temporal sampling. Short-range forecasts compared to independent humidity sensitive satellite instruments such as IASI and CrIS show significant improvements as well as all-sky instruments such as SSMIS. Comparing observation of wind measurements such as AEOLUS reveals a better agreement under certain conditions together with improvements in comparisons to conventional wind measurements.

Furthermore, we diagnose the effect on simulated satellite images for GOES and SEVIRI based on the study by Lopez & Matricardi (2022). Such a diagnostic tool allows for verification at the km-scale, as well as at very high temporal resolution down to 10 min. Comparing infrared, visible, water vapor observations with ifs simulated cloud fields we find better performance in the high-resolution simulations as briefly discussed by Lledo et al. (2022) using the fractional-skill score on clouds. In future experiments, we aim to compare forecasts at the km-scale based on assimilation experiments with higher temporal and spatial data density compared to the current operational setting of the IFS.

• L. Lledó, T. Haiden, J. Schröttle, and R. Forbes, Scale-dependent verification of precipitation and cloudiness at ECMWF.ECMWF Newsletter Number 174 (2022)
• P. Lopez and M. Matricardi, Validation of IFS+RTTOV/MFASIS0.64-μm reflectances against observations from GOES-16, GOES-17, MSG-4 and Himawari-8. ECMWF Technical Memoranda (2022)

How to cite: Schröttle, J., Lledó, L., Lupu, C., Burrows, C., and Holm, E.: On the benefits of assimilating a denser set of geostationary clear-sky radiance, EMS Annual Meeting 2023, Bratislava, Slovakia, 4–8 Sep 2023, EMS2023-82, https://doi.org/10.5194/ems2023-82, 2023.