Evaluation of ESTOFEX convective outlooks from 2007 to 2021. Part 2: climatology and reliability of threat level polygons

The European Storm Forecast Experiment (ESTOFEX) is a team of volunteer forecasters that have been providing experimental convective outlooks for Europe since 2002. Probabilistic storm forecasts issued by ESTOFEX address threats posed by severe convective storms, i.e. lightning, large hail, severe wind gusts, tornadoes and excessive precipitation. ESTOFEX also serves as a platform for exchange of knowledge about forecasting severe convective storms with a goal of improving their understanding among both members of ESTOFEX and others. While not official, ESTOFEX products have been widely used by national meteorological services, severe storm communities and the public. ESTOFEX forecasters have regularly contributed to the ESSL Testbeds and are using an ingredients based forecasting methodology to forecast severe storms. Consistently improving severe storm reporting in the European Severe Weather Database (ESWD) and availability of ground-based lightning detection measurements over the last decade enabled the verification of a large number of ESTOFEX forecasts. Thus, in this work we evaluate 4019 convective outlooks issued by ESTOFEX forecasters since 2007. Our goals are to detect spatiotemporal patterns in convective outlooks and test the reliability of issued threat level polygons, i.e. for a low and high probability of lightning, and an increasing probabilities of severe weather: level 1, level 2 and level 3. We performed the verification by applying a number of methods, including contingency table statistics, receiver operating characteristic curves, practically perfect hindcasts and by calculating spatial coverage of detected lightning (ATDnet network) and local storm reports (ESWD) within issued polygons. Results indicate that products issued by ESTOFEX over the last 15 years, when combined together, are consistent with convective climatologies based on reanalyses and lightning detection data. However, we note that forecasters tend to issue outlooks relatively more often for severe weather outbreaks across western and central Europe. We found that while 95% of the issued lightning probability areas fulfilled the required criterion of coverage, this was only true for 40% of the severe weather probability areas. One reason is that while lightning observations are relatively homogeneous across the forecast domain, the same cannot be said about severe weather observations. These are lacking in regions such as southeastern or eastern Europe, while forecasters calibrated themselves to the higher observed coverage in western and central Europe. The reliability of ESTOFEX forecasts increased over the time, but we found underestimation of lightning probabilities over southern Europe and an overestimation of lightning probabilities over British Isles and Scandinavia.