Confidence intervals for the reliability diagram

The reliability diagram is often used to assess the reliability of a set of probabilistic forecasts. It plots the observed relative frequency of an event against its predicted probability. Reliability is evaluated by assessing the distance of the plotting positions from the diagonal which designates 'perfect reliability'. The reliability diagram is easy to construct and to understand. However, there is a caveat: it is not immediately clear which distance from the diagonal would still be considered 'reliable'. Due to finite sample size, even a perfectly reliable forecasting system could result in a reliability diagram where not all points are on the diagonal. Various authors have proposed visual guidance that allows a forecaster to assess whether the observed relative frequencies fall within the variations that can be expected even when a forecasting system is perfectly reliable. These include 'consistency bars', a modified version of the reliability diagram which is plotted on probability paper and  a 'standardized reliability diagram' based on the Normal transform of the Poisson binomial distribution. The present contribution provides another method to visualize the expected deviation from the diagonal: confidence intervals based on the Poisson binomial distribution. The application is demonstrated in various case studies.