High-resolution model Verification Evaluation (HiVE). Part 1: Using neighbourhood techniques for the assessment of ocean model forecast skill

HiVE is a CMEMS funded collaboration between the atmospheric Numerical Weather Prediction (NWP) verification and the ocean community within the Met Office, aimed at demonstrating the use of spatial verification methods originally developed for the evaluation of high-resolution NWP forecasts, with CMEMS ocean model forecast products. Spatial verification methods provide more scale appropriate ways to better assess forecast characteristics and accuracy of km-scale forecasts, where the detail looks realistic but may not be in the right place at the right time. As a result, it can be the case that coarser resolution forecasts verify better (e.g. lower root-mean-square-error) than the higher resolution forecast. In this instance the smoothness of the coarser resolution forecast is rewarded, though the higher-resolution forecast may be better. The project utilised open source code library known as Model Evaluation Toolkit (MET) developed at the US National Center for Atmospheric Research. 

 

This project saw, for the first time, the application of spatial verification methods to sub-10 km resolution ocean model forecasts. The project consisted of two parts. Part 1 describes an assessment of the forecast skill for SST of CMEMS model configurations at observing locations using an approach called HiRA (High Resolution Assessment). Part 2 is described in the companion poster to this one.  

 

HiRA is a single-observation-forecast-neighbourhood-type method which makes use of commonly used ensemble verification metrics such as the Brier Score (BS) and the Continuous Ranked Probability Score (CRPS). In this instance all model grid points within a predefined neighbourhood of the observing location are considered equi-probable outcomes (or pseudo-ensemble members) at the observing location. The technique allows for an inter-comparison of models with different grid resolutions as well as between deterministic and probabilistic forecasts in an equitable and consistent way. In this work it has been applied to the CMEMS products delivered from the AMM7 (~7km) and AMM15 (~1.5km) model configurations for the European North West Shelf that are provided by the Met Office. 

 

It has been found that when neighbourhoods of equivalent extent are compared for both configurations it is possible to show improved forecast skill for SST for the higher resolution AMM15 both on- and off-shelf, which has been difficult to demonstrate previously using traditional metrics. Forecast skill generally degrades with increasing lead time for both configurations, with the off-shelf results for the higher resolution model showing increasing benefits over the coarser configuration.