The Effect of Data Limitations on Earthquake Forecasting Model Selection

In our recent study, we have developed an ETAS-based (Epidemic-Type Aftershock Sequence; Ogata, 1988) time-dependent earthquake forecasting model for Europe. Aside from inverting a basic set of parameters describing aftershock behaviour on a highly heterogeneous dataset, we have proposed several model variants, focusing on implementing the knowledge about spatial variations in the background rate inferred by ESHM20 already during the inversion of ETAS parameters, fixing the term dictating the productivity law to specific values to balance the more productive triggering by high-magnitude events (productivity law) with their much rarer occurrence (GR law), and using the b-positive method for the estimation of the b-value.

When testing the model variants, we apply the commonly used approach of performing retrospective tests on each model to check for self-consistency over long time periods and pseudo-prospective tests for comparison of models on one-day forecasting periods during seven years. While such pseudo-prospective tests reveal that some models indeed outperform others, for other model pairs, no significant performance difference was detected.

Here, we investigate in more detail the conditions under which performance differences of two competing models can be detected with statistical significance. Using synthetic tests, we investigate the effects of a catalog’s size and the magnitude range it covers on the significance of model performance difference. This will provide insight into whether recording many small events can, in this sense, replace having a large enough dataset of higher-magnitude events. Furthermore, due to the underrepresentation (or absence) of high-magnitude earthquakes in both training and testing data, both the models and tests are prone to overfitting to small events, potentially resulting in forecasts that underestimate both productivity of sequences with a high-magnitude main event and probabilities that a larger earthquake will follow such an event. We focus on defining metrics that highlight these properties as they are often of interest when applying time-dependent forecasting models to issuing operational earthquake forecasts.