IMPROVER tools for physically consistent realizations

Title: IMPROVER tools for physically consistent realizations

Authors: Max White, Jasmine Beaver, Gavin Evans

IMPROVER (Integrated Model Post-Processing and Verification) is an open-source, Python-based toolbox developed by the UK Met Office for post-processing ensemble weather forecasts. It provides a wide range of capabilities, including physical and statistical corrections, model blending, regridding, thresholding, and ancillary data generation. To date, much of the development has focused on producing skilful probabilistic forecast products. However, many applications – such as driving hydrological and impact models – also require spatially and temporally coherent forecast scenarios, often referred to as physically consistent realisations.

To address this need, we have implemented a suite of tools within IMPROVER for generating physically realistic realisations from post-processed, multi-model ensemble forecasts across different spatial domains, resolutions, and lead times. These methods aim to retain the calibrated statistical properties of probabilistic forecasts while producing coherent forecast evolutions suitable for downstream applications.

We will present the open-source, reusable Python functionality added to the IMPROVER toolbox to support this capability, including:

• Generation of additional realisations: expansion of a dataset’s realisation dimension to increase ensemble size while maintaining physical consistency.
• Stochastic noise generation: the addition of spatially correlated noise using a short-space Fourier transform to break grid point ties (for example, zero precipitation occurring in multiple ensemble members). This avoids physically unrealistic, spatially uncorrelated artifacts.
• Realisation clustering and matching: k-medoids clustering of realisations (ensemble members) from a primary forecast source (for example, a lower-resolution global model used at longer lead times), followed by matching to realisations from secondary forecast sources such as nowcasts or higher resolution ensembles. This defines a trajectory for each realization across the lead time range. Jumps between different forecast sources can be smoothed using temporal interpolation.
• ECC-Q and ECC-T mapping: calibration using Ensemble Copula Coupling, either by sampling at evenly spaced quantiles (ECC-Q) or using a parametric gamma distribution (ECC-T), allowing the spatial characteristics of the raw ensemble members to be better represented after sampling.
• Period disaggregation and temporal interpolation utilities: tools for disaggregating accumulated diagnostics (for example, from 3-hourly to 1-hourly) and temporally interpolating forecast fields using linear methods or machine-learning-based approaches such as Google FILM.
• Deterministic realisation selection: extraction of a deterministic realisation from a clustered ensemble, providing a physically plausible single forecast trajectory.