Software-driven structured expert judgment: modern tools to efficiently synthesize scientific knowledge for uncertainty quantification in volcanic hazard assessment

When physical and data sciences are not sufficient to support models and/or decisions, expert judgment is a recognized approach to quantify the uncertainties around specific issues. Among different expert judgment methods, Structured Expert Judgment (SEJ) employs a formalized, documented procedure for obtaining probabilistic belief statements from a group of experts about unknown quantities or parameters. This provides an attractive approach for performing assessments at volcanoes characterized by large knowledge gaps by integrating diverse kinds of information. Uncertainties are likely to be large in these cases, and SEJ can quantify these uncertainties to provide scientists and decision-makers with indications of the reliability of the assessments. The final goal of this approach is to obtain the group’s synthesized uncertainty distribution (representing a new “virtual expert” often called Decision Maker - DM) around specific items (or “target questions”), that result from combining elicited judgments of all the experts.

More specifically, performance-based expert elicitation relies on validating expert probability assessments through an impartial empirical trial. Thus, in a performance-based elicitation, experts are tasked with providing their estimations of probability distributions for a set of known quantities, often referred to as “seed items,” which, jointly, serve as calibration benchmarks for expert performance. Experts’ responses provide the basis for performance scoring using the Classical Model algorithm (Cooke, 1991), determined empirically on the individual expert’s attainment, jointly in terms of two separate metrics: “statistical accuracy” and “informativeness”, which are evaluated on the set of seed items overall.

Traditionally, the management of a performance-based expert elicitation is time consuming and involves the collection of tens of questionnaires, often manually copied from hard-copies of the responses or email attachments. The performance-calibration algorithms and the production of standard outputs, including statistical samples of the DM responses, are necessary steps every time an elicitation is conducted, but relied on different scripts and independent pieces of software.

In this study we present the latest version of the recently released software ELICIPY (de’Michieli Vitturi et al. 2024), which allows organizing and managing performance-based expert elicitation sessions in a partially automated way, resulting in a significantly enhanced efficiency. This new version includes, among other improvements, an “agreement index” to quantify the level of agreement among experts, a continuous version of the Classical Model to compute expert weights, the possibility to import weights from an external file, new plot options, an interactive dashboard for results exploration.

ELICIPY enables the smooth conduction of elicitations with a relatively large number of questions and/or experts. Moreover, the management of multiple elicitation sessions, with questionnaire modifications and response updates are made easier. Some of these improvements are demonstrated on the case study of a hazard and risk assessment for the active Kolumbo submarine Volcano (Aegean Sea, Greece), which was made of 20 Seed Items and 64 Target Items in total, the latter structured into 17 subject matter groups (Bevilacqua et al., 2025).