Ontological ensemble modelling to account for different kinds of uncertainties

Ensemble modeling combines different models or their parametrizations into a single model. Conventional ensemble methods merge individual forecast distributions into one (e.g., the average). We introduce the Ontological Ensemble (OE) model, which preserves all individual forecast distributions and quantifies their dispersion, thereby capturing the epistemic uncertainty. This integrity acknowledges different kinds of uncertainties, keeps them separated, and provides a complete description of our knowledge and its limitations. Unlike conventional ensemble methods, the OE quantifies the reliability of a forecast and enables a more meaningful model validation. Specifically, the OE allows exposing representational errors of a system, the so-called ontological error [Marzocchi & Jordan 2014], by testing if observations (i.e., the “true” unknown distribution of the underlying process) fall outside the OE distribution. To construct this new type of ensemble, our approach is twofold:

• In a first step, we create a weighted average ensemble (in terms of an average forecast). We employ multi-variate logistic regression to obtain model weights that maximize the forecasting skill of the ensemble [Herrmann & Marzocchi 2023]. Retrospective testing on 15 years of Operational Earthquake Forecasting data in Italy [Marzocchi et al. 2014] demonstrated a significant improvement over the best candidate forecast model in terms of cumulative information gain per earthquake (cumIGPE).
• In a second step, we create the actual OE forecast by modeling a forecast distribution with the Beta distribution and the weighted dispersion of the candidate forecasts (using the weights and weighted average determined in step 1).

Our ensemble software framework is flexible and extensible. In step 1 for instance, we implemented sequence-specific ensembling as a more advanced ensemble strategy to acknowledge the spatiotemporal variability of seismicity and forecasts. It extends Herrmann & Marzocchi 2023 by not only fitting the logistic regression to the whole region (i.e., all spatiotemporal bins), but separately to sequences (i.e., only the affected spatiotemporal bins) and excluding those from the regional fit. This separation also better exploits the candidate forecast models: it acknowledges those that perform well during sequences (aftershocks) and those that perform well generally (background seismicity). Compared to the previous (purely regional) ensemble, it improved the cumIGPE over the best forecast model by 56%. Additionally, it leads to a more honest uncertainty quantification in the OE. We have also operationalized our framework for near real-time applications.

Validating this new type of forecast model requires new testing routines, which we plan to develop for the Collaboratory for the Study of Earthquake Predictability (CSEP, cseptesting.org); it will involve implementing the OE in pyCSEP [Savran et al. 2022] and/or floatCSEP [Iturrieta et al.]. Future plans also include exploring more ensemble configurations and strategies to further improve forecast skill and uncertainty quantification.

References

Herrmann & Marzocchi (2023). Maximizing the forecasting skill of an ensemble model. doi: 10.1093/gji/ggad020

Iturrieta et al. (in preparation). Modernizing CSEP Earthquake Forecasting Experiments: The Floating Testing Center.

Marzocchi & Jordan (2014). Testing for ontological errors in probabilistic forecasting models of natural systems. doi: 10.1073/pnas.1410183111

Marzocchi et al. (2014). The establishment of an operational earthquake forecasting system in Italy. doi: 10.1785/0220130219

Savran et al. (2022). pyCSEP: A Python Toolkit for Earthquake Forecast Developers. doi: 10.1785/0220220033