Session NP5.2

[Programme]

NP5.2 | Advances in statistical post-processing, blending, and verification of deterministic and probabilistic forecasts

Advances in statistical post-processing, blending, and verification of deterministic and probabilistic forecasts

Convener: Maxime TaillardatECSECS | Co-conveners: Stéphane Vannitsem, Jochen Broecker, Sebastian LerchECSECS, Julie Bessac

Orals

| Fri, 19 Apr, 10:45–12:30 (CEST)

Room K2

Posters on site

| Attendance Thu, 18 Apr, 10:45–12:30 (CEST) | Display Thu, 18 Apr, 08:30–12:30

Hall X3

Posters virtual

| Attendance Thu, 18 Apr, 14:00–15:45 (CEST) | Display Thu, 18 Apr, 08:30–18:00

vHall X3

Statistical post-processing techniques for weather, climate, and hydrological forecasts are powerful approaches to compensate for effects of errors in model structure or initial conditions, and to calibrate inaccurately dispersed ensembles. These techniques are now an integral part of many forecasting suites and are used in many end-user applications such as wind energy production or flood warning systems. Many of these techniques are flourishing in the statistical, meteorological, climatological, hydrological, and engineering communities. The methods range in complexity from simple bias correction up to very sophisticated machine learning and/or distribution-adjusting techniques that take into account correlations among the prognostic variables.

At the same time, a lot of efforts are put in combining multiple forecasting sources in order to get reliable and seamless forecasts on time ranges from minutes to weeks. Such blending techniques are currently developed in many meteorological centers. These forecasting systems are indispensable for societal decision making, for instance to help better prepare for adverse weather. Thus, there is a need for objective statistical framework for "forecast verification'', i.e. qualitative and quantitative assessment of forecast performance.

In this session, we invite presentations dealing with both theoretical developments in statistical post-processing and evaluation of their performances in different practical applications oriented toward environmental predictions, and new developments dealing with the problem of combining or blending different types of forecasts in order to improve reliability from very short to long time scales.

Orals: Fri, 19 Apr | Room K2

Chairpersons: Jochen Broecker, Sebastian Lerch

10:45–10:50

5-minute convener introduction

Verification and diagnostics

10:50–11:00

EGU24-11096

On-site presentation

The Environmental Modeling Center Verification System (EVS): Real-time verification of Unified Forecast System (UFS) models

Jason Levit, Geoffrey Manikin, Alicia Bentley, Logan Dawson, and Tara Jensen

Currently in the final stages of development, the Environmental Modeling Center Verification System (EVS) is a real-time software system that, once implemented on NCEP supercomputers, will provide verification statistics and graphics for NCEP operational forecast systems. Using the Model Evaluation Toolkit (METplus) suite of verification software, the EVS will use both real-time forecast output and environmental observations to create information on the performance of all NCEP environmental models and products, which are either derived or directly created from the Unified Forecast System (UFS) suite of models. The EVS will generate hundreds of metrics for the suite, which when viewed on EMC webpages will provide a comprehensive and up-to-date overview of the performance of NCEP forecast systems for the general public, model developers, researchers, and decision makers. As EMC works with the international modeling community to develop modeling systems that are based on the Unified Forecast System (UFS), EMC is now evolving towards using the EVS for a single, unified verification software system for real-time verification analysis, evaluation of new and upgraded systems proposed for NWS operations, and as a capability to determine systematic errors and biases that illustrate areas for potential model improvements. This project, led by EMC’s Verification, Post-Processing, and Product Generation Branch (VPPPGB) aims to unify EMC’s verification strategy under one maintainable and supportable software system, and to use performance metrics that have been vetted and peer-reviewed by the UFS community via the Developmental Testbed Center’s 2021 UFS Metrics Workshop.

How to cite: Levit, J., Manikin, G., Bentley, A., Dawson, L., and Jensen, T.: The Environmental Modeling Center Verification System (EVS): Real-time verification of Unified Forecast System (UFS) models, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-11096, https://doi.org/10.5194/egusphere-egu24-11096, 2024.

11:00–11:10

EGU24-8807

ECS

On-site presentation

The fractions skill score for ensemble forecast verification

Ludwig Wolfgruber, Tobias Necker, Lukas Kugler, Martin Weissmann, Manfred Dorninger, and Stefano Serafin

This work explores how the Fractions Skill Score (FSS), originally developed for deterministic forecasts of binary events, can be used for probabilistic forecast verification. By comparing a selection of four ensemble-based methods to compute the FSS, we highlight their distinct behaviour with ensemble size, neighbourhood size, and frequency of occurrence of the forecast event. Our study emphasizes that only a specific variant of the FSS, which we refer to as "probabilistic FSS", demonstrates reasonable behaviour with ensemble size. We reveal that the probabilistic FSS depends on ensemble size in a similar way as the Brier Skill Score, despite performing a neighbourhood-based instead of a grid-point-based forecast evaluation. We derive a formula that describes the expected behaviour of the probabilistic FSS with changes in ensemble size. Finally, utilizing a unique dataset of high-resolution 1000-member ensemble precipitation forecasts for Germany, we explore the impact of ensemble and neighbourhood size on the predictive skill by studying various subsamples of the full ensemble.

How to cite: Wolfgruber, L., Necker, T., Kugler, L., Weissmann, M., Dorninger, M., and Serafin, S.: The fractions skill score for ensemble forecast verification, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-8807, https://doi.org/10.5194/egusphere-egu24-8807, 2024.

Discussion Post a comment

11:10–11:20

EGU24-2145

ECS

Highlight

On-site presentation

Quantification of the practical predictability of thunderstorm occurrence using machine learning

Kianusch Vahid Yousefnia, Tobias Bölle, Isabella Zöbisch, and Thomas Gerz

While the statistical post-processing of numerical weather prediction (NWP) data constitutes a powerful ingredient of many forecasting suites of severe weather, post-processing for thunderstorm occurrence becomes ever more difficult as the lead time of the NWP forecast increases. In terms of identifying thunderstorm occurrence as a function of lead time, this increased difficulty is reflected in a decay of skill for which even sophisticated machine learning (ML) models cannot fully compensate. In this work, we propose how the time scale of skill decay of supervised ML models can be studied as a function of the spatiotemporal label resolution used for training. If the label is constructed from lightning observations, label resolution is modified by varying the time and radius thresholds by which strokes of lightning are associated with NWP data. We exemplify our method using SALAMA, a feedforward neural network model which we have developed for identifying the probability of thunderstorm occurrence in NWP data. The model has been trained on convection-resolving ensemble forecasts over Central Europe and lightning observations. We show for SALAMA that the time scale for skillful thunderstorm predictions increases linearly with label resolution, which underlines the practical ability of our method to quantify the predictability of thunderstorm occurrence.

How to cite: Vahid Yousefnia, K., Bölle, T., Zöbisch, I., and Gerz, T.: Quantification of the practical predictability of thunderstorm occurrence using machine learning, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-2145, https://doi.org/10.5194/egusphere-egu24-2145, 2024.

Discussion Post a comment

Post-processing and blending

11:20–11:30

EGU24-4357

On-site presentation

Autoregressive extensions of EMOS with application to surface temperature ensemble postprocessing

Annette Möller, David Jobst, and Jürgen Groß

Two extensions of the autoregressive EMOS (AR-EMOS) which are based on the idea of smooth EMOS (SEMOS) model are proposed: The de-seasonalized EMOS (DAR-SEMOS) approach models time series behavior in the mean and variance of the predictive distribution separately, the standardized AR-SEMOS (SAR-SEMOS) method attempts to incorporate both effects jointly by fitting a time series model to the standardized forecast errors. The proposed modifications both allow to incorporate seasonal and trend effects as well as autoregressive behavior into the mean and variance parameter of the predictive distribution. Due to this explicit modelling of seasonal and trend behavior a rolling training period is not required anymore, and a longer (static) training period can be utilized for model fitting. The extended models can postprocess ensemble forecasts with arbitrary forecast horizons. In a case study for 2m surface temperature the extensions DAR- and SAR-SEMOS yield substantial improvements over AR-EMOS and SEMOS, for all considered forecast horizons and at the majority of observations stations. Overall, the SAR-SEMOS model yields the most noticeable improvements. At the same time its seamless approach of jointly modelling the time series behavior in the mean and variance parameter makes it appealing for practical and possibly operational use.

How to cite: Möller, A., Jobst, D., and Groß, J.: Autoregressive extensions of EMOS with application to surface temperature ensemble postprocessing, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-4357, https://doi.org/10.5194/egusphere-egu24-4357, 2024.

11:30–11:40

EGU24-2861

ECS

On-site presentation

D-Vine GAM Copula based Quantile Regression with Application to Ensemble Postprocessing

David Jobst, Annette Möller, and Jürgen Groß

Temporal, spatial or spatio-temporal probabilistic models are frequently used for weather forecasting. The D-vine (drawable vine) copula based quantile regression (DVQR) is a powerful tool for this application field, as it incorporates important predictor variables from a large set by a data-driven sequential forward selection procedure and is able to model complex nonlinear relationships among them. However, the current DVQR does not always explicitly and economically allow to account for additional covariate effects, e.g. temporal or spatio-temporal information. Consequently, we propose an extension of the current DVQR, where we parametrize the bivariate copulas in the D-vine copula through Kendall's Tau which can be linked to additional covariates. The parametrization of the correlation parameter allows generalized additive models (GAMs) and spline smoothing to detect potentially hidden covariate effects. The new method is called GAM-DVQR, and its performance is illustrated in a case study for the postprocessing of 2m surface temperature forecasts. We investigate a constant as well as a time-dependent Kendall's Tau. The GAM-DVQR models are compared to the benchmark method gradient-boosted Ensemble Model Output Statistics (EMOS-GB). The results indicate that the GAM-DVQR models are able to identify time-dependent correlations as well as relevant predictor variables and significantly outperform the state-of-the-art method EMOS-GB. Furthermore, the introduced parameterization allows using a static training period for GAM-DVQR, yielding a more sustainable model estimation in comparison to DVQR using a sliding training window.

How to cite: Jobst, D., Möller, A., and Groß, J.: D-Vine GAM Copula based Quantile Regression with Application to Ensemble Postprocessing, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-2861, https://doi.org/10.5194/egusphere-egu24-2861, 2024.

11:40–11:50

EGU24-4404

On-site presentation

Machine learning-based parametric post-processing of solar irradiance ensemble forecasts

Sándor Baran and Ágnes Baran

By the end of 2022, the renewable energy share of the global electricity capacity reached 40.3% and the new installations were dominated by solar energy, showing a global increase of 21.7%. Due to the high volatility of photovoltaic energy sources, their successful integration into the electrical grid requires accurate short-term power forecasts. These forecasts are obtained from the predictions of solar irradiance, where the most advanced method is the probabilistic approach based on ensemble forecasts. However, ensemble forecasts are often underdispersive and subject to systematic bias. Hence, they require some form of statistical post-processing, where parametric models provide full predictive distributions of the weather variables at hand.

We propose a general two-step machine learning-based approach to calibrating ensemble weather forecasts, where, in the first step, improved point forecasts are generated, which then together with various ensemble statistics serve as input features of the neural network estimating the parameters of the predictive distribution [1]. In a case study based on global horizontal irradiance forecasts of the operational ensemble prediction system of the Hungarian Meteorological Service, the predictive performance of this novel method is compared with the forecast skill of the raw ensemble and the state-of-the-art ensemble model output statistics approaches [2]. We show that at least up to 48h, statistical post-processing substantially improves the predictive performance of the raw ensemble for all forecast horizons considered; the maximal gain e.g. in terms of the mean continuous ranked probability score is above 20%. Furthermore, the proposed two-step machine learning-based approach outperforms in skill its competitors.

References

1. Baran, Á, Baran, S., A two-step machine-learning approach to statistical post-processing of weather forecasts for power generation. Q. J. R. Meteorol. Soc. (2023), doi:10.1002/qj.4635.

2. Schulz, B., El Ayari, M., Lerch, S., Baran, S., Post-processing numerical weather prediction ensembles for probabilistic solar irradiance forecasting. Sol. Energy 220 (2021), 1016-1031.

*Research is supported by the Hungarian National Research, Development and Innovation Office under grant no. K142849

How to cite: Baran, S. and Baran, Á.: Machine learning-based parametric post-processing of solar irradiance ensemble forecasts, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-4404, https://doi.org/10.5194/egusphere-egu24-4404, 2024.

11:50–12:00

EGU24-2794

ECS

On-site presentation

Improving Distributional Regression Forests for Post-processing Extreme Wind Gust Forecasts

Bastien François, Harun Kivril, Maurice Schmeits, and Kirien Whan

Extreme events, such as wind gusts or extreme precipitation, can generate huge impacts on our society. Accurate predictions of such events are thus vital for taking preventive measures. In spite of continued scientific progress in weather forecasting, ensemble forecasts exhibit biases and under-dispersion and have to be calibrated using observations before being used, e.g., in hydrological or renewable energy applications. Several post-processing techniques have therefore been developed and applied over the last decades in order to improve forecast quality. Many existing post-processing methods are parametric, i.e. they assume that the predictive distribution belongs to a class of known probability distributions. Parameters of the assumed distribution are then modeled as functions of predictors obtained from numerical weather prediction models, for example using nonhomogeneous regression or more advanced tree-based methods. One of the main limitations of such methods is that a suitable family of probability distributions has to be selected to describe the distribution of the target variable. This implies that intermediate and high values are modeled with the same parametric distribution, which can lead to suboptimal results for extremes. We propose to adapt an existing distributional tree-based technique (Distributional Regression Forests) used for ensemble post-processing to overcome this limitation by allowing the method to choose different statistical distributions to model intermediate and extreme values. The proposed method is applied to forecasts of 6-hourly maximum wind gusts from 2018 to 2022 over the Netherlands using the ECMWF-IFS ensemble data. Results are compared against several state-of-the-art parametric and non-parametric post-processing methods. In comparison with these alternatives, the proposed algorithm reasonably corrects intermediate values and presents the largest skill improvements for wind gust extremes depending on lead times, stations and thresholds. However, it remains difficult to beat the raw forecasts of extremes. Therefore, it encourages further research on adding more flexibility to parametric methods for the post-processing of extreme weather forecasts.

How to cite: François, B., Kivril, H., Schmeits, M., and Whan, K.: Improving Distributional Regression Forests for Post-processing Extreme Wind Gust Forecasts , EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-2794, https://doi.org/10.5194/egusphere-egu24-2794, 2024.

12:00–12:10

EGU24-11394

ECS

On-site presentation

Attention-based postprocessing of ensemble weather forecasts for renewable energy applications by leveraging inter-ensemble relationships of multiple predictors

Aaron Van Poecke, Ruoke Meng, Jonathan Demaeyer, Joris Van den Bergh, Geert Smet, Piet Termonia, Peter Hellinckx, and Hossein Tabari

Indirect models for renewable energy forecasting rely heavily on accurate weather predictions. Operational weather forecasting today is mainly based on numerical weather prediction models, often employing ensembles to estimate the day-to-day forecast uncertainty. To correct for errors due to simplifications in these models, inaccurate initial conditions, and representativeness problems, statistical postprocessing becomes necessary for these ensemble forecasts. Current postprocessing techniques often disregard possible inter-ensemble relationships by correcting each member separately, or employ a distributional approach that requires extra multivariate methods to restore spatio-temporal and inter-variable correlations. In this work, we tackle these shortcomings with an innovative, attention-based member-by-member approach which postprocesses each member individually while simultaneously integrating information from other ensemble members. Variables required for renewable energy forecasting are postprocessed at the station level by regressing ensemble forecasts of multiple predictors, including the forecasted variable itself, against observational data. The training data utilized is sourced from the EUPPBench dataset, which contains ensemble predictions from the integrated forecasting system of the ECMWF and corresponding observations. Transformer modules built around Self-Attention are employed to capture dependencies between different predictors, such as temperature and total cloud cover, next to significant relationships between the ensemble members themselves. Additionally, our model postprocesses the forecasts for all lead times simultaneously, taking into account the correlation between the postprocessed variable and forecasts generated at earlier and later lead times. This results in postprocessing techniques that can be employed in downstream applications for conversion to renewable energy forecasts.

How to cite: Van Poecke, A., Meng, R., Demaeyer, J., Van den Bergh, J., Smet, G., Termonia, P., Hellinckx, P., and Tabari, H.: Attention-based postprocessing of ensemble weather forecasts for renewable energy applications by leveraging inter-ensemble relationships of multiple predictors, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-11394, https://doi.org/10.5194/egusphere-egu24-11394, 2024.

12:10–12:20

EGU24-8137

On-site presentation

Temperature Forecasting with Markov Expert Aggregations

Olivier Wintenberger, Leo Pfitzner, and Olivier Mestre

12:20–12:30

EGU24-9326

ECS

On-site presentation

Leveraging deterministic weather forecasts for in-situ probabilistic predictions via deep learning

David Landry, Anastase Charantonis, and Claire Monteleoni

We propose a neural network approach to produce probabilistic weather forecasts from a deterministic numerical weather prediction. The developed method is applicable to any gridded forecast including the recent machine learning weather prediction model outputs. To postprocess multiple lead times using a single model, we introduce a lead time embedding that encodes the shift in biases as the forecast progresses. We apply our approach to operational outputs from the Global Deterministic Prediction System up to ten-day lead times. The model is trained to predict METAR in-situ surface temperature observations in Canada and the United States. The resulting forecasts have a mean CRPS below 2.5 K at 10 days lead time while maintaining a spread-error ratio of approximately 0.9, suggesting appropriate calibration. For extreme temperatures, the model’s biases are comparable to that of the underlying deterministic forecast. Our approach increases the utility of a deterministic forecast by adding information about the uncertainty, without incurring the cost of simulating multiple trajectories. It requires no information regarding forecast spread and can be used to generate probabilistic predictions from any deterministic forecast.

How to cite: Landry, D., Charantonis, A., and Monteleoni, C.: Leveraging deterministic weather forecasts for in-situ probabilistic predictions via deep learning, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-9326, https://doi.org/10.5194/egusphere-egu24-9326, 2024.

Posters on site: Thu, 18 Apr, 10:45–12:30 | Hall X3

Display time: Thu, 18 Apr 08:30–Thu, 18 Apr 12:30

Chairpersons: Maxime Taillardat, Jochen Broecker

X3.36

EGU24-2361

ECS

Precipitation forecast post-processing: blending deterministic NWPs with machine learning

Luca Monaco, Roberto Cremonini, and Francesco Laio

Direct model output forecasts by Numerical Weather Prediction models (NWPs) present some limitations caused by errors mostly due to sensitivity to initial conditions, sensitivity to boundary conditions and deficiencies in parametrization schemes (i.e. orography).
These sources of error are unavoidable, and atmosphere chaotic dynamics makes prediction errors to spread rapidly in time in the course of the forecast, inducing both systematic and random errors.
Nonetheless, in the last 50 years NWPs had a significant decrease in the impact of these source of errors, even in the long-term forecast, thanks for instance to an ever-increasing computational capability, but still their relevance is not neglectable.
Moreover, different NWPs present specific different pros and cons which are findable empirically. For instance, in the case of precipitation forecast in the north-west Italy, low spatial resolution models (e.g. ECMWF-IFS) tend to be more reliable in terms of space and time in predicting the average precipitation, while high resolution models (e.g. COSMO-2I) tend to forecasts the maximum precipitation better. Research purposes apart, actual limitations must be seen in an operational context, where weather forecasts’ skillfulness and associated uncertainty are information of the utmost importance to the forecaster and in general to the user of a certain forecasts system.
In order to tackle the limitations of NWPs and the need of an uncertainty-quantified meteorological forecast, we propose a machine learning based multimodel post-processing technique for precipitation forecast. We focus on precipitation since it is the most important variable in the issue of spatially localized weather alert notice by the Italian Civil Protection’ system and at the same time it is one of the most challenging variables to forecast.

We use different Convolutional Neural Networks (CNNs) to obtain both deterministic and probabilistic forecast grids over 24h up to 48h focusing in the North-West Italy, using different high and low resolution deterministic NWPs as input and using high resolution rain-gauge corrected radar observations as ground truth for the training. We use constrainted linear regressions as a mean of deterministic benchmark, and ECMWF-EPS as a mean of probabilistic benchmark. The test phase show decent improvements in terms of RMSE for every season.

How to cite: Monaco, L., Cremonini, R., and Laio, F.: Precipitation forecast post-processing: blending deterministic NWPs with machine learning, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-2361, https://doi.org/10.5194/egusphere-egu24-2361, 2024.

X3.37

EGU24-2005

Enhancing member-by-member post-processing with neural networks

Sebastian Lerch, Jakob Freytag, Thomas Muschinski, and Sam Allen

Using post-processing methods to correct systematic errors of ensemble forecasts has become standard practice in research and operations. During recent years, a new focal point of research interest has been the use of modern machine learning methods to allow for more flexible post-processing methods that incorporate additional input predictors. In particular, neural network (NN) models have been shown superior predictive performance in various case studies [1-3].

In contrast, the member-by-member (MBM) post-processing approach [4] adjusts each ensemble member individually using a relatively simple statistical model. This has the advantage that the post-processed ensemble forecasts are not only calibrated, but physically consistent over time, space and different weather variables. Therefore, multivariate dependencies are preserved even if MBM is applied separately for each component. The drawback is that MBM has no straightforward way of incorporating additional input variables (beyond ensemble predictions of the target variable) and therefore typically fails to perform as well as NN-based post-processing approaches [3].

To address this shortcoming, we propose a novel NN-enhanced MBM post-processing approach (“MBM-NN”), which combines the basic idea of MBM with a neural network for incorporating additional predictors to leverage advantages of both approaches. In case studies on probabilistic wind gust forecasting over Germany and on the EUPPBench dataset [5], we demonstrate that the MBM-NN model achieves significant improvements over the standard MBM approach, and reaches comparable performance to state-of-the-art NN-based post-processing models, while retaining multivariate dependencies.

References

[1] Rasp, S. and Lerch, S. (2018). Neural networks for post-processing ensemble weather forecasts. Monthly Weather Review, 146, 3885-3900

[2] Vannitsem, S., Bremnes, J.B., Demaeyer, J., Evans, G.R., Flowerdew, J., Hemri, S., Lerch, S., Roberts, N., et al. (2021). Statistical Postprocessing for Weather Forecasts - Review, Challenges and Avenues in a Big Data World. Bulletin of the American Meteorological Society, 102, E681-E699

[3] Schulz, B. and Lerch, S. (2022). Machine learning methods for postprocessing ensemble forecasts of wind gusts: A systematic comparison. Monthly Weather Review, 150, 235-257

[4] Van Schaeybroeck, B. and Vannitsem, S. (2015). Ensemble post‐processing using member-by-member approaches: theoretical aspects. Quarterly Journal of the Royal Meteorological Society 141, 807-818

[5] Demaeyer, J., Bhend, J., Lerch, S., Primo, C., Van Schaeybroeck, B., Atencia, A. Ben Bouallègue, Z., Chen, J., Dabernig, M., Evans, G., Faganeli Pucer, J., Hooper, B., Horat, N., et al. (2023). The EUPPBench postprocessing benchmark dataset v1.0. Earth System Science Data, 15, 2635-2653

How to cite: Lerch, S., Freytag, J., Muschinski, T., and Allen, S.: Enhancing member-by-member post-processing with neural networks, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-2005, https://doi.org/10.5194/egusphere-egu24-2005, 2024.

X3.38

EGU24-6959

ECS

Can neural networks outperform quantile mapping for post-processing seasonal weather forecast variables over the Alpine region?

Sameer Balaji Uttarwar, Sebastian Lerch, Diego Avesani, and Bruno Majone

The possibility to use seasonal weather forecasts is of paramount importance in hydrological and socio-economical applications. However, current seasonal weather forecasts from global numerical weather prediction (NWP) models inherit systematic biases resulting from inaccurate representation and parameterization of local to global scale environmental processes. Therefore, the hydrological community frequently uses the quantile mapping (QM) statistical postprocessing for bias correction and downscaling of the meteorological inputs (i.e., daily precipitation and temperature) to hydrological models. The QM often assumes a linear and static relationship between quantiles of observed and simulated data over time. These limitations can be relaxed by employing a Neural Network (NN) based postprocessing method. In this context, the objective of this study is to compare the accuracy of QM and NN statistical postprocessing of ensemble seasonal weather forecasts over the Trentino-South Tyrol region (north-eastern Italian Alps), characterised by complex topography.

The study uses the latest fifth-generation seasonal weather forecast system (SEAS5) total precipitation and 2m-temperature dataset produced by European Centre for Medium-Range Weather Forecast (ECMWF), available at a horizontal grid resolution of 0.125° x 0.125° with 25 ensemble members in a re-forecast period from 1981 to 2016. The respective reference dataset is a high-resolution gridded observation (250 m x 250 m) over the region of interest. The QM method derives a functional relationship between the variable of interest and the corresponding predictor, whereas the NN based methods can be used with a set of predictors to learn the linear and non-linear relationships in a data-driven way.

The analysis is divided into training (1981 – 2010, 30 years) and testing (2011 – 2016, 6 years) period to compare the cumulative ranked probability scores (CRPS) of both the statistical postprocessing methods. The statistical postprocessing is implemented univariately on the daily dataset (2m temperature and precipitation) over a month for each lead time. The raw forecasts and postprocessed forecasts are compared with particular focus on the effects of the forecast lead time and location, as well as diurnal and seasonal cycles in forecast performance. The postprocessed forecasts revealed a significant improvements compared to the raw forecasts.

How to cite: Uttarwar, S. B., Lerch, S., Avesani, D., and Majone, B.: Can neural networks outperform quantile mapping for post-processing seasonal weather forecast variables over the Alpine region?, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-6959, https://doi.org/10.5194/egusphere-egu24-6959, 2024.

X3.39

EGU24-7878

The past, current and future activities on statistical postprocessing in the context of the European Meteorological Network (EUMETNET)

Stéphane Vannitsem and Jonathan Demaeyer

X3.40

EGU24-10797

ECS

Highlight

Improving probabilistic wind speed forecasts with deep learning

Katharina Klein, Daniel Tolomei, Sjoerd Dirksen, Kirien Whan, and Maurice Schmeits

This project aims at developing post-processing models for deriving probabilistic weather forecasts from NWP forecast data using deep learning techniques.

The first part of the project involves improving probabilistic wind speed forecasts for consecutive lead times using an autoregressive model. Post-processing multiple lead times simultaneously is challenging because of the inherent temporal dependencies. Classical approaches often involve processing lead times individually and subsequently employing empirical copula methods to handle such dependencies. Building on previous work, we instead consider the ARMOS model which incorporates temporal dependencies through the autoregressive property of forecast errors and can be used to obtain an explicit multivariate probability distribution for the weather variable in question. As such, it is a generalization of the widely used Ensemble Model Output Statistics (EMOS) used for estimating marginal distributions.

For the purpose of this project, the model is applied to deterministic forecasts from the Harmonie-Arome model of KNMI, yielding a multivariate parametric forecast distribution for hourly wind speeds up to 48 hours ahead. We model the marginal conditional distributions as truncated normal distributions, and the model parameters are estimated both linearly and as the output of a neural network with convolutional and optional LSTM layers which can detect spatial patterns and temporal dependencies, respectively. We compare the resulting models to a variant of EMOS adapted to deterministic forecasts that is paired with a copula method.

The ARMOS model has so far shown good performance in modeling temporal dependencies explicitly without the need to use a copula method. Moreover, the network models outperform the classical approach of estimating the distribution parameters linearly. We will provide an update on these results as well as an outlook on planned future work.

How to cite: Klein, K., Tolomei, D., Dirksen, S., Whan, K., and Schmeits, M.: Improving probabilistic wind speed forecasts with deep learning, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-10797, https://doi.org/10.5194/egusphere-egu24-10797, 2024.

X3.41

EGU24-11694

Optimal blending of ensemble forecasts for probabilistic precipitation nowcasting

Martin Weissmann, Simon Köhldorfer, Tobias Necker, and Alexander Kann

Precipitation nowcasting is essential for mitigating the negative effects of severe weather, necessitating accurate and timely forecasts. Our study introduces a novel Local Ensemble Transform Kalman Filter (LETKF)-based blending approach that effectively combines ensemble forecasts to provide cheap probabilistic nowcasts. Our method optimally weights ensemble probabilities by considering precipitation/radar observations as the ground truth. It shifts computed weights in time and refines neighborhood probabilities (NPs). The upscaling step for computing NPs introduces two free parameters, neighborhood size and precipitation rate, which can be selected based on the forecasters needs. Demonstrated within GeoSphere Austria's regional forecasting system using AROME ensemble forecasts over Austria, our technique was especially beneficial for short lead times of up to 2 hours. Longer lead times require to incorporate signal propagation, which is possible by shifting weights in space and time. Our study underscores the effectiveness of data assimilation techniques in enhancing ensemble blending. The proposed approach affordably improves the robustness and accuracy of short-term probabilistic forecasts and holds the potential for extending it to multi-model ensemble blending.

How to cite: Weissmann, M., Köhldorfer, S., Necker, T., and Kann, A.: Optimal blending of ensemble forecasts for probabilistic precipitation nowcasting, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-11694, https://doi.org/10.5194/egusphere-egu24-11694, 2024.

X3.42

EGU24-18642

Copula-based statistical post-processing for multi-site temperature forecasts

Elisa Perrone, Maurits Flos, and Irene Schicker

Modern weather forecasts are typically in the form of an ensemble of forecasts obtained from multiple runs of numerical weather prediction models. Ensemble forecasts are usually biased and affected by dispersion errors, and they should be statistically corrected to gain accuracy. This is often done following a two-step approach: first, we correct the univariate forecasts, and then, we reconstruct the dependence structure non-parametrically via empirical copulas. The parametric correction of the dependence structure is limited to Gaussian copula-based methods. In this work, we propose a novel approach based on a more general parametric class of copulas called Archimedean copulas. We test the new method in both a simulated scenario and a case-study setting for multi-site temperature forecasts from the ALADIN-LAEF ensemble system in Austria. Our findings show that the state-of-the-art non-parametric techniques perform well in the simulation study. However, Archimedean copulas outperform the existing techniques, especially Gaussian copula approaches, and output well-calibrated forecasts in the real-case study. Our analysis demonstrates the usefulness of including advanced parametric copula methods in the post-processing context and the need of a more realistic simulated framework to test new methodology.

How to cite: Perrone, E., Flos, M., and Schicker, I.: Copula-based statistical post-processing for multi-site temperature forecasts, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-18642, https://doi.org/10.5194/egusphere-egu24-18642, 2024.

X3.43

EGU24-19254

Post-processing for an on-demand extremes digital twin – a multi-model approach for wind and solar energy production

Irene Schicker, Petrina Papazek, Pascal Gfäller, Iris Odak Plenkovic, Ivan Vujec, Alexander Kann, and Kristian Horvath

The amount of wind and solar energy fed into the European power grid increases rapidly and with the transition to a fossil fuel-free energy production, relying heavily on renewable energy sources, more accurate predictions for both high-resolution temporal and spatial scales are needed to ensure, most of all, grid stability. This is even more the case with extreme events, both extremes in weather across the nowcasting to weeks ahead time scale and combined and non-necessarily extreme weather, events such as Dunkelflaute or longer lasting solar/wind droughts. Accurate, frequently updated and especially on-demand available predictions of the expected power production are needed. Post-processing methods enable targeted forecasts of meteorological parameters both at site-location and regional level, which can server for a conversion to power production, particularly a direct conversion of NWP predictions and observations to power production.

For an on-demand extreme digital twin forecasting system, fast and transferable post-processing methods, able to account for the upper/lower bounds of the respective distributions are needed. Furthermore, they need to be able to either generate on-the-fly (semi-)synthetic power production data or a reduced set of both observation and NWP input data. The latter is essential when moving towards hyper-resolution NWP simulations with only a limited set of training data available.

Within the on-demand extreme digital twin initiative, several post-processing methods, statistical and (deep) machine learning, were implemented and applied to selected use cases for on/offshore wind and solar production extreme events. Here, we demonstrate (i) the capability of the Kalman filter, the analogs method, IrradPhyDNet, a sequence-2-sequence LSTM, Random Forest, and other machine learning/statistical methods in extreme event prediction, (ii) evaluate the methods skills by using heterogeneous (multiple NWP models, observations, climatologies, etc.) and varying length input data and real/(semi-)synthetic power data as target, and (iii) present a workflow for an on-demand prediction for wind and solar energy production including user-interaction.

How to cite: Schicker, I., Papazek, P., Gfäller, P., Odak Plenkovic, I., Vujec, I., Kann, A., and Horvath, K.: Post-processing for an on-demand extremes digital twin – a multi-model approach for wind and solar energy production, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-19254, https://doi.org/10.5194/egusphere-egu24-19254, 2024.

X3.45

EGU24-4610

ECS

Statistical post-processing and generation of spatially correlated precipitation forecasts with convolutional neural networks

Syed Ahmad Siffat, Quan Jun Wang, Kirien Whan, and Erik Weyer

The raw forecasts from a numerical weather prediction (NWP) model cannot be directly used because of systematic biases. Statistical calibration is performed to produce reliable and accurate ensemble forecasts. However, this is usually done on a grid-cell by grid-cell basis, followed by the use of empirical copula to embed a realistic spatial structure in the calibrated ensemble members. One drawback of these approaches is that it is difficult to select the empirical copula. In this paper, we propose Convolutional Neural Network (CNN) based models for post-processing precipitation forecast fields and generating ensemble forecasts. Unlike the traditional approaches which are applied to individual grid-cells, the model is applied to the whole precipitation field. Monte-Carlo (MC) dropouts are used to estimate uncertainty and generate ensemble forecasts. These ensemble forecasts preserve the inherent spatial structure, thereby eliminating the need for ensemble reordering. The model is applied to NWP forecasts of Brisbane drainage basin in eastern Australia. It is evaluated on all precipitation events, including no, low and high precipitation amounts. The results show that, for all levels of precipitation, the ensemble forecasts are skillful at both the grid-cell and basin scale, and the uncertainty is estimated reliably.

How to cite: Siffat, S. A., Wang, Q. J., Whan, K., and Weyer, E.: Statistical post-processing and generation of spatially correlated precipitation forecasts with convolutional neural networks, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-4610, https://doi.org/10.5194/egusphere-egu24-4610, 2024.

X3.46

EGU24-5541

ECS

Machine learning-based discrete post-processing of visibility ensemble forecasts

Maria Nagy-Lakatos and Sandor Baran

In aviation meteorology, as well as in water and road transportation, the accurate and
reliable prediction of visibility is of utmost importance. Despite various meteorological
services offering ensemble forecasts for visibility, the predictive accuracy and reliability for
this parameter are notably lower compared to variables like temperature or wind speed.
Therefore, it is strongly recommended to implement some form of calibration, typically
involving the estimation of the predictive distribution through parametric or non-parametric
methods, including machine learning techniques. The World Meteorological Organization
suggests that visibility observations should be reported in discrete values, turning the
predictive distribution into a discrete probability law. Consequently, the calibration process
can be simplified to a classification problem. This study investigates the predictive
performance of locally, semi-locally, and regionally trained proportional odds logistic
regression (POLR) and multilayer perceptron (MLP) neural network classifiers using
visibility ensemble forecasts from the European Centre for medium-range weather
forecasts. The findings reveal that while climatological forecasts surpass the raw
ensemble, post-processing leads to a substantial improvement in forecast skill. Overall,
POLR models exhibit superiority over their MLP counterparts.

Reference
Baran, S., Lakatos, M., Statistical post-processing of visibility ensemble forecasts.
Meteorol. Appl. 30 (2023), paper e2157, doi:10.1002/met.2157.

*Research is supported by the ÚNKP-23-3 New National Excellence Program of the
Hungarian Ministry for Culture and Innovation from the source of the National Research,
Development and Innovation Fund and the Hungarian National Research, Development
and Innovation Office under Grant No. K142849.

How to cite: Nagy-Lakatos, M. and Baran, S.: Machine learning-based discrete post-processing of visibility ensemble forecasts, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-5541, https://doi.org/10.5194/egusphere-egu24-5541, 2024.

X3.47

EGU24-6281

ECS

Tail calibration of probabilistic forecasts

Sam Allen, Jonathan Koh, and Johanna Ziegel

X3.48

EGU24-7702

ECS

On statistical calibration of dual-resolution precipitation forecasts

Marianna Lakatos-Szabó, Estíbaliz Gascón, and Sándor Baran

Recently, all leading meteorological centers release ensemble forecasts that vary in terms of ensemble size and spatial resolution, even when covering the same area. These factors significantly impact the forecast accuracy and computational resources required. In the last few years, the plans of upgrading the configuration of the Integrated Forecast System of the European Centre for Medium-Range Weather Forecasts (ECMWF) from a single forecast with 9 km resolution and a 51-member ensemble with 18 km resolution induced an extensive study of the forecast skill of both raw and post-processed dual-resolution predictions comprising ensemble members of different horizontal resolutions.

We investigate the predictive performance of the censored shifted gamma (CSG) [1] ensemble model output statistic (EMOS) approach for statistical post-processing with the help of dual-resolution 24h precipitation accumulation ensemble forecasts over Europe with various forecast horizons. The high-resolution operational 50-member ECMWF ensemble is supplemented by a 200-member low-resolution (29-km grid) experimental forecast. The various dual-resolution combinations, which are equivalent in computational cost to the operational ensemble, show improved forecast skill after EMOS post-processing compared with raw ensemble combinations [3]. Additionally, the differences between these combinations are significantly reduced as a result of this post-processing technique. Moreover, the semi-locally trained CSG EMOS is fully able to catch up with the state-of-the-art quantile mapping [2] and provides an efficient alternative without requiring additional historical data essential in determining the quantile maps.

References:

[1] Baran, S. and Nemoda, D. (2016). Censored and shifted gamma distribution based EMOS model for probabilistic quantitative precipitation forecasting. Environmetrics 27, 280–292.
[2] Gascón, E., Lavers, D., Hamill, T. M. , Richardson, D. S., Ben Bouallègue, Z., Leutbecher, M. and Pappenberger, F. (2019). Statistical postprocessing of dual-resolution ensemble precipitation forecasts across Europe. Quart. J. Roy. Meteor. Soc. 145, 3218–3235.
[3] Szabó, M., Gascón, E. and Baran, S. (2023) Parametric post-processing of dual-resolution precipitation forecasts. Weather Forecast., 38(8), 1313–1322.

How to cite: Lakatos-Szabó, M., Gascón, E., and Baran, S.: On statistical calibration of dual-resolution precipitation forecasts, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-7702, https://doi.org/10.5194/egusphere-egu24-7702, 2024.

X3.49

EGU24-10787

ECS

Statistical Post-Processing for Wind Gust and Precipitation Extremes: Insights from a Pre-Operational System

Harun Kıvrıl, Bastien François, Maurice Schmeits, Kirien Whan, Eva van der Kooij, and Antonello Squintu

Effective forecasting of weather events, especially extremes, is critical for minimizing potential damage and ensuring public safety. Yet, ensemble forecasts that allow to represent uncertainty in weather predictions like ECMWF-ENS suffer from biases and dispersion issues. These shortcomings decrease the forecasting skill and introduce the need for statistical post-processing methods like Ensemble Model Output Statistics (EMOS) and Quantile Regression Forest (QRF) to enhance the forecast quality. While these methods increase overall forecasting performance in general, their capability to handle extreme events varies. To strengthen the forecasting of these critical events, a closer examination and potential refinement of the statistical post-processing methods are necessary, along with the development of methods tailored to extreme weather events. This study analyzes the performance of pre-operational post-processing models of ECMWF-ENS wind gust and precipitation forecasts in the Netherlands using EMOS and/or QRF techniques. The skill of the models (using Continuous Probability Ranked Skill Score (CRPSS) and Brier Skill Score (BSS)) for both wind gusts and precipitation is demonstrated. Besides, by focusing on windstorm Poly (July 5th, 2023) and a heavy rain case (June 22nd, 2023), the capabilities of the methods in forecasting specific extreme events are investigated. While the QRF forecasts for the heavy rain case show better skill than for the raw forecasts, for the Poly storm case the post-processed models performed worse than the raw forecasts initially. Thus, this work concentrates on the underlying reasons for the limited performance on that event and proposes an error modeling approach to enhance the post-processing performance of the storm event without compromising the overall forecasting performance.

How to cite: Kıvrıl, H., François, B., Schmeits, M., Whan, K., van der Kooij, E., and Squintu, A.: Statistical Post-Processing for Wind Gust and Precipitation Extremes: Insights from a Pre-Operational System, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-10787, https://doi.org/10.5194/egusphere-egu24-10787, 2024.

X3.50

EGU24-16157

ECS

Enhancing Renewable Energy Forecasting: A Comprehensive Evaluation of Weather Forecast Models and Post-Processing Methods for Belgium

Ruoke Meng, Aaron Van Poecke, Geert Smet, Jonathan Demaeyer, Hossein Tabari, Peter Hellinckx, Joris Van den Bergh, and Piet Termonia

As renewable energy sources continue to account for an increasing proportion of Belgium's energy production, decision making in renewable energy production increasingly relies on accurate numerical weather prediction forecasts. For general applications, forecast validation often focuses on direct comparisons to observations for the whole domains of interest, while in this study we assess model performance specifically related to renewable energy productions. We perform extended verification of relevant variables (wind speed, temperature, solar radiation, etc.) from multiple high-resolution deterministic and ensemble weather forecast models operated in Belgium for the period of May 2021 - June 2023. The forecasts are verified with observational datasets collected from on- and offshore weather stations, masts, lidars, and wind farm observations to comprehensively understand the capabilities of the models, making use of various deterministic and probabilistic skill scores. The results show that during lead times up to two days, although verification metrics differ among models, there are systematic errors in their forecasts for different observation sites. Such errors can often be eliminated by post-processing techniques. Therefore, we extend our verification dataset, with post-processed forecasts corrected by several methods including member-by-member and AI-based approaches. The results of this work will lead to an enhanced understanding of current forecasting skills of the operational models, help to evaluate the effectiveness of goal-oriented post-processing methods, and provide a reference for Belgian sustainable energy stakeholders.

How to cite: Meng, R., Van Poecke, A., Smet, G., Demaeyer, J., Tabari, H., Hellinckx, P., Van den Bergh, J., and Termonia, P.: Enhancing Renewable Energy Forecasting: A Comprehensive Evaluation of Weather Forecast Models and Post-Processing Methods for Belgium, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-16157, https://doi.org/10.5194/egusphere-egu24-16157, 2024.

Posters virtual: Thu, 18 Apr, 14:00–15:45 | vHall X3

Display time: Thu, 18 Apr 08:30–Thu, 18 Apr 18:00

Chairpersons: Stéphane Vannitsem, Maxime Taillardat, Sebastian Lerch

vX3.2

EGU24-1665

Optimisation of Regional Weather Forecasts for Northern Algeria Using a Convolutional Neural Network and AROME Model Analysis.

Islam Bousri

This study introduces an innovative approach aimed at enhancing the accuracy of regional weather forecasts from the AROME model, covering northern Algeria. By leveraging AROME analysis, a refined representation based on real observations and widely used for monitoring and validating our model, our primary objective was to precisely correct surface parameters, including temperature at 2 meters, humidity, wind force, and sea-level atmospheric pressure (MSLP). This correction was performed based on their forecast ensemble, all while preserving spatial resolution.

This methodology has yielded promising results, demonstrating a significant improvement in the accuracy of regional weather forecasts. The presentation will delve into the detailed integration process of the Convolutional Neural Network (CNN) and AROME analysis, highlighting the successes achieved in correcting essential surface parameters. These advancements strengthen the reliability of regional meteorological models, with positive implications for resource planning and management in the northern region of Algeria.

How to cite: Bousri, I.: Optimisation of Regional Weather Forecasts for Northern Algeria Using a Convolutional Neural Network and AROME Model Analysis., EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-1665, https://doi.org/10.5194/egusphere-egu24-1665, 2024.

Discussion Post a comment