NP5.1

Postprocessing ensemble weather predictions to correct systematic errors has become a standard practice in research and operations. However, only few recent studies have focused on ensemble postprocessing of wind gust forecasts, despite its importance for severe weather warnings, e.g. in European winter storms. First, we provide a comprehensive review and systematic comparison of several statistical and machine learning methods for probabilistic wind gust forecasting via ensemble postprocessing, then we assess the performance of selected methods within winter storms. The methods can be divided in three groups: State of the art postprocessing techniques from statistics (ensemble model output statistics (EMOS), member-by-member postprocessing, isotonic distributional regression), established machine learning methods (gradient-boosting extended EMOS, quantile regression forests) and neural network-based approaches (distributional regression network, Bernstein quantile network, histogram estimation network). The different approaches are systematically compared using six years of data from a high-resolution, convection-permitting ensemble prediction system run operationally at the German weather service, and hourly observations at 175 surface weather stations in Germany. While all postprocessing methods yield calibrated forecasts and are able to correct the systematic errors of the raw ensemble predictions, incorporating information from additional meteorological predictor variables beyond wind gusts as well as estimating locally adaptive neural networks leads to significant improvements in forecast skill. Assessing the performance of EMOS and neural network-based postprocessing for selected winter storms, we find that the networks better adapt to the extreme conditions than the statistical benchmark and thus yield a superior predictive performance. However, results suggest that the performance can still be further improved, e.g. via regime-dependent postprocessing.

How to cite: Schulz, B. and Lerch, S.: Machine learning for postprocessing ensemble forecasts of wind gusts with a focus on European winter storms, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-869, https://doi.org/10.5194/egusphere-egu22-869, 2022.

Statistical post-processing of ensemble forecasts has become a common practice in research to correct biases and errors in calibration. While many of the developments have been focused on univariate methods that calibrate the marginal distributions, practical applications often require accurate modeling of spatial, temporal, and inter-variable dependencies. Copula-based multivariate post-processing methods, such as ensemble copula coupling, have been proposed to address this issue and proceed by reordering univariately post-processed ensembles with copula functions to retain the dependence structure. We propose a novel multivariate post-processing method based on generative machine learning where post-processed multivariate ensemble forecasts are generated from random noise, conditional on the inputs of raw ensemble forecasts. Moving beyond the two-step strategy of separately modeling marginal distributions and multivariate dependence structure, the generative modelling approach allows for directly obtaining multivariate probabilistic forecasts as output. The flexibility of the generative model also enables us to incorporate additional predictors straightforwardly and to generate an arbitrary number of post-processed ensemble members. In a case study on the surface temperature and wind speed forecasts from the European Centre of Medium-Range Weather Forecasts at weather stations in Germany, our generative model that incorporates additional weather predictors substantially improves upon the multivariate spatial forecasts from copula-based approaches. And the model shows competitive performance even with state-of-the-art neural network-based post-processing models applied for the marginal distributions.

How to cite: Chen, J., Lerch, S., and Janke, T.: Generative machine learning methods for multivariate ensemble post-processing, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-921, https://doi.org/10.5194/egusphere-egu22-921, 2022.

Efficient wind farm operation requires reliable probabilistic forecasts of power ramps. These are sudden fluctuations in power production which, if unanticipated, can lead to significant imbalances in the electrical grid.  The power produced by a turbine strongly depends on the wind speed at hub-height, making it is useful to base these forecasts on calibrated wind speed scenarios generated by statistically postprocessing numerical weather predictions (NWPs). Since the probability of a ramp event depends jointly on the wind speed distributions forecasted at multiple future times, postprocessing methods must not only calibrate the marginal forecasts for each lead time, but also estimate temporal dependencies among their errors.

We use new multivariate Gaussian regression (MGR) models to postprocess all next-day hourly 100m wind speeds near offshore wind farms in one step. The postprocessed forecast is a multivariate Gaussian distribution with mean vector μ — containing the 24 forecasted hourly mean wind speeds — and Σ — the 24 × 24 covariance matrix containing uncertainties of the individual forecasts as well as their temporal error correlations.  Joint distributions are estimated conditionally by flexibly linking the components of μ and parameters specifying Σ to predictors derived from an ECMWF ensemble using generalized additive models for each distributional parameter.

The joint distribution — predicted uniquely for each ECMWF initialization — can simulate postprocessed wind speed ensembles with any number of members. Subsequently, the forecasted ensembles are transformed into power space using an idealized turbine power curve and probabilities computed for different ramp events. Ramp forecasts from MGR outperform those obtained using reference methods which postprocess wind speed forecasts in two-steps: (i) first calibrating the marginal distributions with nonhomogeneous Gaussian regression before (ii) constructing temporal error dependencies using either the order statistics of the NWP ensemble (ensemble copula coupling, ECC) or those of raw observations (Schaake Shuffle).

How to cite: Muschinski, T., Lang, M. N., Mayr, G. J., Messner, J. W., Simon, T., and Zeileis, A.: Probabilistic power ramp forecasts using multivariate Gaussian regression, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-2176, https://doi.org/10.5194/egusphere-egu22-2176, 2022.

Nowadays, most weather forecasting centers produce ensemble forecasts.  Ensemble forecasts provide information about probability distribution of the weather variables. They give a more complete description of the atmosphere than a unique run of the meteorological model. However, they may suffer from bias and under/over dispersion errors that need to be corrected. These distribution errors may depend on weather regimes. In this paper, we propose various extensions of the Gaussian mixture model and its associated inference tools for ensemble data sets.  The proposed models are then used to identify clusters which correspond to different types of distribution errors. Finally, a standard calibration method known as Non homogeneous Gaussian Regression (NGR)  is applied cluster by cluster in order to correct ensemble forecast distributions. It is shown that the proposed methodology is effective, interpretable and easy to use.  The clustering algorithms are illustrated on simulated and real data. The calibration method is applied to real data of temperature and wind medium range forecast for 3 stations in France. 

How to cite: Jouan, G., Cuzol, A., Monbet, V., and Monnier, G.: Gaussian mixture models for clustering and calibration of ensemble weather forecasts, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-2311, https://doi.org/10.5194/egusphere-egu22-2311, 2022.

Probabilistic predictions of precipitation call for rather sophisticated postprocessing approaches due to its low predictability, high spatio-temporal variability and highly positive skewness. Moreover, the large number of zeros makes the generation of physically realistic postprocessed forecast scenarios using standard approaches like ensemble copula coupling (ECC) rather difficult. In addition to classical statistical approaches, recently, machine learning based methods gained increasing popularity in the field of postprocessing of probabilistic weather forecasts.

In this study, we compare conditional generative adversarial network (cGAN) based postprocessing of daily precipitation with a quantile regression based approach. In principle, an appropriately trained cGAN model should be able to generate postprocessed forecast scenarios that improve forecast skill and cannot be distinguished from observed data in terms of spatial structure. While we use ECC to generate physically realistic forecast scenarios from quantile regression, cGAN does not need any additional ECC steps. For training and verification, we use COSMO-E ensemble forecasts with a grid resolution of about 2 km over Switzerland and the corresponding CombiPrecip observations, which are a gridded blend of radar and gauge observations. Preliminary results suggest that it is possible to generate realistic looking forecast scenarios using cGAN, but up to now, we have not been able to increase forecast skill. On the other hand, quantile regression seems to increase forecast skill at the expense of relying on an additional ECC step to generate forecast scenarios.

How to cite: Hemri, S., Bhend, J., Spirig, C., Nerini, D., Moret, L., Furrer, R., and Liniger, M. A.: Postprocessing of gridded precipitation forecasts using conditional generative adversarial networks and quantile regression, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-5609, https://doi.org/10.5194/egusphere-egu22-5609, 2022.

New postprocessing methods are sometimes introduced without proper comparison to other available techniques, and therefore the institutions responsible for the operational implementation of weather forecasts may struggle deciding the best choice for their particular usecase. With the goal of helping the weather community to make such decisions, the benchmark of different postprocessing methods on predefined datasets is an important topic and is a key deliverable of the current EUMETNET postprocessing module. This benchmark is also a collaborative effort from several meteorological institutions, members of EUMETNET, and academia to define common pratices and shape standards.

In this presentation, we will highlight the different aspects of the benchmark: (1) its current status and organization and (2) its objectives for the next 2 years. We will also detail the challenges ahead for this exercise, and the foreseen datasets and infrastructures needed to tackle them.

How to cite: Demaeyer, J.: News about the EUMETNET statistical postprocessing benchmark, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-5797, https://doi.org/10.5194/egusphere-egu22-5797, 2022.

A lot of Numerical Weather Prediction (NWP) models and their associated Model Output Statistics (MOS) are available. Expert aggregation has a bunch of advantages to deal with all these models, like being online, adaptive to model changes and having theoretical guarantees. With a new expert aggregation algorithm - FSBOA - a combination of BOA (Wintenberger 2017) and FS (Herbster and Warmuth 1998), and the use of a sliding window, we improved the temperature prediction on average without loosing too much reactivity of the expert weights. We also tested several aggregation strategies in order to improve the prediction of  extrem temperature events like cold and heat waves. To do so, we added some biased experts of the Météo-France 35-member ensemble forecast (PEARP) to the set of models. We also tried out the SMH (Mourtada et al. 2017) algorithm which fits the sleeping experts framework.

How to cite: Pfitzner, L., Mestre, O., Wintenberger, O., and Adjakossa, E.: Temperature prediction with expert agregation, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-7407, https://doi.org/10.5194/egusphere-egu22-7407, 2022.

Predicting subseasonal to seasonal weather and climate yields numerous benefits for economic and environmental decision-making.
Forecasters verify the forecast quality of models by initializing large sets of retrospective forecasts to predict past variations and phenomena in hindcast studies.

Quantifying prediction skill for multi-dimensional geospatial model output is computationally expensive and a difficult coding challenge. The large datasets require parallel and out-of-memory computing to be analyzed efficiently. Further, aligning the many forecast initializations with differing observational products is a straight-forward, but exhausting and error-prone exercise for researchers.

To simplify and standardize forecast verification across scales from hourly weather to decadal climate forecasts, we built climpred: a python package for computationally efficient and methodologically consistent verification of ensemble prediction models. We rely on the python software ecosystem developed by the open pangeo geoscience community. We leverage NetCDF metadata using xarray and out-of-core computation parallelized with dask to scale analyses from a laptop to supercomputer.

With climpred, researchers can assess forecast quality from a large set of metrics (including cprs, rps, rank_histogram, reliability, contingency, bias, rmse, acc, ...) in just a few lines of code:

hind = xr.open_dataset('initialized.nc')
obs = xr.open_dataset('observations.nc')
he = climpred.HindcastEnsemble(hind).add_observations(obs)
# he = he.remove_bias(how='basic_quantile',
#                                       train_test_split='unfair', 
#                                       alignment='same_verif')
he.verify(metric='rmse',
                comparison='e2o',
                alignment='same_verif',
                dim='init',
                reference=['persistence', 'climatology'])

This simplified and standardized process frees up resources to tackle the large process-based unknowns in predictability research. Here, we perform a live and interactive multi-model comparison removing bias with different methodologies from NMME project hindcasts and compare against persistence and climatology reference forecasts.

Documentation: https://climpred.readthedocs.io

Repository: https://github.com/pangeo-data/climpred

Reference paper: Brady, Riley X. and Aaron Spring (Mar. 2021). “Climpred: Verification of Weather and Climate Forecasts”. en. Journal of Open Source Software 6.59, p. 2781. https://joss.theoj.org/papers/10.21105/joss.02781

How to cite: Spring, A.: climpred: weather and climate forecast verification in python, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-8200, https://doi.org/10.5194/egusphere-egu22-8200, 2022.

The UK Met Office is developing an open-source probability-based post-processing system called IMPROVER to exploit convection permitting, hourly cycling ensemble forecasts. The system is tasked with blending these forecasts with both deterministic nowcast data, and coarser resolution global ensemble model data, to produce seamless probabilistic forecasts from the very short to medium range.

A majority of the post-processing within IMPROVER is performed on gridded forecasts, with site-specific forecasts extracted as a final step, helping to ensure consistency. IMPROVER delivers a wide range of probabilistic products to both operational meteorologists and as input to automated forecast production. and this presentation will detail some of the work that has been undertaken in the past year to prepare, with a focus on the use of statistical post-processing.

Statistical post-processing plays two complimentary roles within IMPROVER; ensuring forecasts better reflect reality, and in so doing, bringing different models into better alignment, which improves the seamlessness of model transitions. For a selection of diagnostics, the gridded forecasts from different source models are calibrated independently using ensemble model output statistics (EMOS). Results of experiments looking at the calibration of gridded forecasts will be discussed briefly.

More recently calibration of site forecasts has been introduced as a final step for temperature and wind speed forecasts. Results of experiments using EMOS to perform calibration in a variety of different ways will be presented, including justifications and trade-offs made in choosing a final approach.

• This will include some discussion of the remaking of weather symbol products as period, rather than instantaneous, forecasts and the implications for their verification.

How to cite: Moseley, S., Rust, F., Evans, G., Ayliffe, B., Hurst, K., Howard, K., Wright, B., and Jackson, S.: IMPROVER : A probabilistic, multi-model post-processing system for meteorological forecasts, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-8706, https://doi.org/10.5194/egusphere-egu22-8706, 2022.

Over time scales between 10 days and 10-20 years – the macroweather regime – atmospheric fields, including the temperature, respect statistical scale symmetries, such as power-law correlations, that imply the existence of a huge memory in the system that can be exploited for long-term forecasts. The Stochastic Seasonal to Interannual Prediction System (StocSIPS) is a stochastic model that exploits these symmetries to perform long-term forecasts. It models the temperature as the high-frequency limit of the fractional energy balance equation (fractional Gaussian noise) which governs radiative equilibrium processes when the relevant equilibrium relaxation processes are power law, rather than exponential.

The multivariate version of the model (m-StocSIPS), exploits the space-time statistics of the temperature field to produce realistic global simulations, including realistic teleconnection networks and El Niño events and indices. One of the implications of this model is the lack of Granger-causality: the optimal predictor at gridpoint i is obtained from the past of the timeseries i and cannot be improved using past temperatures from any other location j. This allows to treat predictions for long-memory processes as “past value” problems rather than the conventional initial value approach that uses the current state of the atmosphere to produce ensemble forecasts.

To improve the stochastic predictions, a zero-lag independent (non-stochastic) predictor is needed. Here we use the Canadian Seasonal to Interannual prediction System (CanSIPS), as a deterministic co-predictor. CanSIPS is a long-term multi-model ensemble (MME) system using two climate models developed by the Canadian Centre for Climate Modelling and Analysis (CCCma). The optimal linear combination of CanSIPS and StocSIPS (CanStoc) was based on minimizing the square error of the final predictor in the common hindcast period 1981-2010 using different out-of-sample validations. Global time series and regional maps at 2.5ºx2.5º resolution show that the skill of CanStoc is better than that of each individual model for most of the regions when non-overlapping training and verification periods are used.

How to cite: Del Rio Amador, L. and Lovejoy, S.: Causality in long-term predictions, past-value problems and a stochastic-deterministic hybrid, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-10869, https://doi.org/10.5194/egusphere-egu22-10869, 2022.

As a consequence of the growing importance of probabilistic predictions in various application fields due to a necessary functional risk management and strategy, there is an increasing demand for appropriate probabilistic model evaluation. Besides proper scoring rules, which can evaluate not only the expectation but the entire predictive distribution, graphical assessment methods are particularly advantageous to diagnose possible model misspecifications.

Probabilistic forecasts are often based on distributional regression models, whereby the computation of predictive distributions, probabilities, and quantiles is generally dependent on the software (package) being used. Therefore, routines to graphically evaluate probabilistic models are not always available and if so then only for specific types of models and distributions provided by the corresponding package. An easy to use unified infrastructure to graphical assess and compare different probabilistic model types does not yet exist. Trying to fill that gap, we present a common conceptual framework accompanied by a flexible and object-oriented software implementation in the R package topmodels (https://topmodels.R-Forge.R-project.org/).  

The package includes visualizations for PIT (probability integral transform) histograms, Q-Q (quantile-quantile) plots of (randomized) quantile residuals, rootograms, reliability diagrams, and worm plots. All displays can be rendered in base R as well as in ggplot2 and provide different options for, e.g., computing confidence intervals, scaling or setting graphical parameters. Using examples of post-processing precipitation ensemble forecasts, we further discuss how all theses types of graphics can be compared to each other and which types of displays are particularly useful for bringing out which types of model deficiencies.

How to cite: Lang, M. N., Stauffer, R., and Zeileis, A.: Graphical Model Assessment of Probabilistic Forecasts, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-11689, https://doi.org/10.5194/egusphere-egu22-11689, 2022.

In 2020, 36.6 % of the total electricity demand of the world was covered by renewable sources, whereas in the EU (UK included) this share reached 49.3 %. A substantial part of green energy is produced by wind farms, where accurate short range power predictions are required for successful integration of wind energy into the electrical grid. Accurate predictions of the produced electricity require accurate forecasts of the corresponding weather quantity, where the state-of-the-art method is the probabilistic approach based on ensemble forecasts. However, ensemble forecasts are often uncalibrated and might also be biased, thus require some form of post-processing to improve their predictive performance.

To calibrate (hub height) wind speed ensemble forecasts we propose a novel flexible machine learning approach, which results either in a truncated normal or a log-normal predictive distribution (Baran and Baran, 2021). In a case study based on 100m wind speed forecasts of the operational AROME-EPS of the Hungarian Meteorological Service, the forecast skill of this method is compared with the predictive performance of three different ensemble model output statistics approaches and the raw ensemble predictions. We show that compared with the raw ensemble, post-processing always improves the calibration of probabilistic and accuracy of point forecasts, and from the five competing methods the novel machine learning based approaches result in the best overall performance. 

Reference

Baran, S., Baran, Á., Calibration of wind speed ensemble forecasts for power generation. Idöjárás 125 (2021), 609-624.

How to cite: Baran, S. and Baran, Á.: Calibration of wind speed ensemble forecasts for power generation, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-13118, https://doi.org/10.5194/egusphere-egu22-13118, 2022.

To account for forecast uncertainty in numerical weather prediction (NWP) models it has become common practice to employ ensemble prediction systems generating probabilistic forecast ensembles by multiple runs of the NWP model, each time with variations in the details of the numerical model and/or initial and boundary conditions. However, forecast ensembles typically exhibit biases and dispersion errors as they are not able to fully represent uncertainty in NWP models. Therefore, statistical postprocessing models are employed to correct ensembles for biases and dispersion errors in conjunction with recently observed forecast errors.

For incorporating dependencies in space, this work proposes a spatially adaptive extension of the state-of-the-art Ensemble Model Output Statistics (EMOS) model. The new approach, named Markovian EMOS (MEMOS), introduces a Markovian dependence structure on the model parameters by employing Gaussian Markov random fields. For fitting the MEMOS model in a Bayesian fashion the recently developed Integrated Nested Laplace Approximation (INLA) approach is utilized, allowing for fast and accurate approximation of the posterior distributions of the parameters. To obtain physically coherent forecasts the basic MEMOS model is provided with an additional spatial dependence structure induced by the Ensemble Copula Coupling (ECC) approach, which makes explicit use of the rank order structure of the raw ensemble.

The method is applied to temperature forecasts of the European Centre for Medium-Range Weather Forecasts (ECMWF) over Europe, where it exhibits comparable or improved performance over univariate EMOS variants.

How to cite: Möller, A., Thorarinsdottir, T., Lenkoski, A., and Gneiting, T.: Spatially adaptive Bayesian estimation for Probabilistic Temperature Forecasts, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-13388, https://doi.org/10.5194/egusphere-egu22-13388, 2022.

We consider the problem of post-processing forecasts for multiple lead times simultaneously. In particular, we focus on post-processing wind speed forecasts for consecutive lead times (0 - 48h ahead) from the deterministic HARMONIE-AROME NWP model. Given the strong temporal dependency between forecasts at consecutive lead times, it is essential to model the problem as a multivariate statistical post-processing problem in order to take this temporal correlation into account.

A standard procedure in multivariate statistical post-processing is to produce multiple probabilistic forecasts independently for each lead time and introduce the dependency between them at a later stage using an empirical copula. For our specific problem, a successful example of this approach is to use EMOS to fit truncated normal marginal distributions at each lead time and then model the joint distribution by drawing samples from these distributions and reconstructing the temporal dependencies using the Schaake Shuffle.

Our aim is to explore alternative methods that can model and exploit temporal dependencies more explicitly with the goal of improving forecast performance and moving away from sample based distribution modelling. We develop two new methods that produce multivariate truncated normal probabilistic forecasts for all lead times simultaneously, by combining elements from time series analysis and artificial neural networks.

In our first method, we exploit the autoregressive dependencies in the residuals of the NWP wind speed forecasts to deduce an explicit multivariate model. By using a neural network to determine the parameters of this model, we arrive at our first method, which we coin ARMOSnet.

In our second method, we apply Long Short-Term Memory networks, which rank among the state-of-the-art tools for the forecasting of time series. We adapt the LSTM architecture to output a multivariate density that models the temporal dependencies between the consecutive lead times.

We compare our two methods to EMOS combined with the Schaake Shuffle for post-processing wind speed forecasts from the HARMONIE-AROME NWP model. Our new methods both outperform the EMOS-Schaake Shuffle approach in terms of the logarithmic, energy, and variogram scores. Among the two new methods, ARMOSnet exhibits the best performance.

How to cite: Tolomei, D., Dirksen, S., Whan, K., and Schmeits, M.: Multivariate post-processing of temporal dependencies with autoregressive and LSTM neural networks, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-13413, https://doi.org/10.5194/egusphere-egu22-13413, 2022.