Comparing probabilistic and dichotomous forecasts of hazardous events on a consistent basis: Toy statistical models applied to tornado forecasting

In 1963, Fred Sanders wrote “it is urged the probability be acknowledged as the proper internal language of forecasters.” As such, we can consider all forecasts to be based on probabilities. Nevertheless, as Sanders discussed, issuance of probabilistic forecasts to users is not completely clear. In many cases, users may prefer a categorical forecast and, as such, the internal language of probabilities must be translated into an external language of categorical forecasts. Although there are simple ways to threshold probabilities of a dichotomous (yes/no) event into a yes/no forecast, doing so in a way that is consistent between the two expressions is not always easy. This becomes even more complex when not all users have the same decision threshold and, as a result, may have different preferences for where the optimal threshold is set.

A typical example of this problem is the area of tornado forecasting, especially for short range (<1 hour) forecasts of events, such as is the case for tornado warnings in the United States. Here, I develop and explore some simple (“toy”) models of probability distributions for the forecast and occurrence of tornadoes, as well as the user decision problems. Since each of the three models have one (occurrence given a forecast) or two (forecast probability and distribution of user decisions) parameters, it is relatively easy to construct probabilistic models that can be thresholded to mimic tornado warning performance over time in the United States. In addition, the apparent global benefits (or losses) can be estimated for different probabilistic forecast distributions related to different user decisions compared to dichotomous forecasts.

Although this methodology could not directly produce tornado warnings in an operational setting, it provides some limits on how a future system based on probabilities could be constrained. The process allows for the development of thresholded forecasts that are consistent with the underlying probabilistic information.