A flexible approach for evaluating the value of probabilistic forecasts for different decision types and risk averse decision-makers

Forecasts have the potential to improve decision-making but have not been widely evaluated because current forecast value methods have critical limitations. The ubiquitous Relative Economic Value (REV) is limited to binary decisions, cost-loss economic model, and risk neutral decision-makers. Expected Utility Theory can flexibly model more real-world decisions, but its application in forecasting has been limited and the findings are difficult to compare. To enable a systematic comparison of these methods a new metric, Relative Utility Value (RUV), is developed based on Expected Utility Theory. It has the same interpretation as REV but is more flexible and able to handle a wider range of real-world decisions because all aspects of the decision-context are user-defined. Also, when specific assumptions are imposed it is shown that REV and RUV are equivalent. We demonstrate the key differences and similarities between the methods with a case study using probabilistic subseasonal streamflow forecasts in a catchment in the Southern Murray-Darling Basin of Australia. This showed that for most decision-makers the ensemble forecasts were more valuable than a reference climatology for all lead-times (max 30 days), decision types (binary, multi-categorical, and continuous-flow), and levels of risk aversion. Risk aversion had a mixed impact across the different decision-types and the key driver was found to be the specific decision thresholds relative to the damage function. The generality of RUV makes it applicable to any domain where forecast information is used for making decisions, and the flexibility enables forecast assessment tailored to specific decisions and decision-makers. It complements forecast verification and enables assessment of forecast systems through the lens of customer impact.