Protocol for an end-to-end evaluation of operational warning systems

The contribution addresses, from a conceptual point of view, the complex issue of evaluating the performance of warning systems that are operating over large areas to cope with the risk posed by extreme weather events. In the protocol, the performance of the systems is evaluated, at each step in the warning production process, considering the “warning value chain” schematization developed in the HIWeather project of the World Meteorological Organization (http://hiweather.net/Lists/130.html). In a perfect warning chain, the warning received by the end user would contain precise and accurate information that perfectly met their need, contributed by each of the many players in the chain; in real warning chains, information, and hence value, are always lost as well as gained at each link in the chain (Golding 2022, https://link.springer.com/book/10.1007/978-3-030-98989-7).

The protocol is structured as a three-part evaluation process: 1) description of the system; 2) assessment of criticalities during high impact events; 3) routine assessment of daily operations. For each part, the protocol prescribes a set of must-do. The description of the warning system must be based on the schematic subdivision of the warning value chain, i.e., six main capabilities and outputs and five information exchanges elements. An important focus on the evaluation of an operational warning system must be devoted to high impact events. For such cases, the evaluation must include: essential information on the event; information on how each element of the warning value chain has been working during the event; synthetic assessment on the performance of the warning system. Finally, the routine assessment must include: identification of the system’s operational elements; identification of the areas covered by the system; identification of period for which to conduct the assessment and sources of data to be used; identification of appropriate and computable (considering the available data) performance indicators for the different elements of the warning value chain; analysis of relevant data for the chosen time period in the identified areas; evaluation of the performance of the different elements of the waring value chain; final judgment on the overall performance of the system.

This study is being carried out within the Horizon Europe project “The HuT: The Human-Tech Nexus - Building a Safe Haven to cope with Climate Extremes” (https://thehut-nexus.eu/). The protocol has been developed considering two cases studies, and will be further put to test during the remaining part of the project. Through this action, detailed information from many different warning systems will be collected and used for a comparative study between warning systems operating, in different areas of the world, for different weather and climate related risks.