Verifying thunderstorm forecast polygons using a novel machine learning approach

The presence of cumulonimbus (CB) and thunderstorm areas always has a strong influence on aviation. Observations, and especially forecasts, of CB areas are important in air traffic flow management (ATM). Eumetnet provides a unique product, the Cross-Border Convection Forecast (CBCF), which supports planning of air traffic flows across Europe for Eurocontrol and other local air traffic flow management systems. CBCF is issued from May to October on the European domain for the current and following five three-hourly periods between 06 and 21 UTC. Forecasters in each state simultaneously produce forecast (FCST) polygons based on a risk matrix that reveals the probability and convective mode (isolated ISOL, clustered CLST, or widespread WSPR). In addition to CBCF, for internal air traffic management purposes, the meteorological watch office in Croatia issues a similar internal ATM forecast (ATMF) covering the Croatian domain, valid for all days (h24) in three-hourly intervals. After some efforts to subjectively verify ATMF, we developed a novel verification approach with an emphasis on the convective mode.
For the purposes of this analysis, to define observed CB (OBS) polygons we rely solely on 1-min lightning detection data. To obtain OBS polygons in an unbiased and objective manner, we employ HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise), a popular unsupervised machine learning algorithm often used in image segmentation. Since this algorithm also relies on a set of optional, but purpose-dependent thresholds, we will demonstrate the sensitivity of final verification scores on these parameters.
On the one hand, we demonstrate verification results in a traditional parameter space bounded by so-called precision and recall. On the other hand, we report verification of FCST polygons also by the Dice similarity coefficient which, unlike precision and recall, can be viewed as a combined quality score combining areas of both OBS and FCST polygons. 
Preliminary verification scores indicate progressively better forecasting as convective mode organization increases from ISOL to WSPR. In other words, this particular method of verification does not seem plausible when attempting to verify rather small ISOL polygons.
We presented a method for verifying forecasted CB areas, with particular emphasis on the treatment of observed organisational convection mode and verification scores. This approach could also be applied to the verification of other novel Eumetnet products, such as CBCF and eGAFOR.