Weather Data Streaming with Kerchunk: Strengthening Early Warning Systems

The Ensemble Prediction System (EPS) provided by global weather forecast centres generates vast amounts of data that is crucial for early warnings of extreme weather and climate. However, regional and national meteorological services often face challenges in processing this data efficiently, particularly during regional downscaling and post-processing. Conventional methods of downloading and storing GRIB-format data have become increasingly inefficient and unsustainable. The Strengthening Early Warning Systems for Anticipatory Actions (SEWAA) project aims to address these challenges by exploring the use of cloud native operations and GenAI-cGAN driven post-processing systems.   

Kerchunk provides a groundbreaking solution for real-time weather data streaming, catering to the transition towards open and free to use cloud-based object storage from global weather forecasting centres. Kerchunk, in conjunction with GRIB index files, enables efficient, real-time access to weather data, fostering more sustainable workflows in weather and climate services, thus strengthening early warning systems.  

This study developed a workflow for streaming forecast data using Kerchunk with two primary objectives:  

1. Using GRIB index files to reduce redundant readings and generate Kerchunk reference files.  

2. Through streaming-like access, convert the reference files into virtual Zarr datasets and utilise Dask compute for scalable data handling   

The methodology utilised recent improvements in the Kerchunk library that integrate GRIB scanning with its index files. This allowed the system to sample subsets of the GRIB corpus instead of processing entire Forecast Model Run Collections (FMRC), significantly optimising performance.  

The workflow was implemented using cloud-based compute operations via Coiled python library and its service on the Google Cloud Platform. Dask cluster, managed through Coiled, enabled the creation of Zarr virtual datasets for analysis and visualisation. This streaming approach efficiently loads weather data into memory on demand, avoiding unnecessary data downloads and duplication.   

We validated the solution with NOAA GFS/GEFS datasets stored in AWS S3 bucket as open datasets. The optimised workflow demonstrated remarkable efficiency, requiring only <5% of the original GRIB data to be read, with the rest replaced by index files as input for reference file creation. This is followed by the step of Kerchunk reference files to virtual Zarr conversion by Dask clusters to process on a regional scale, such as East Africa’s in minutes supporting near real-time applications across spatial and temporal scales.  

This approach significantly enhances post processing workflows for EPS weather forecast, bolstering early warning systems and anticipatory action. Future work will focus on using the method to scaling training datasets and improving the cost efficiency of cGAN training to advance operational early warning systems. This innovative solution directly addresses the challenges faced by meteorological services in processing massive weather datasets, providing a scalable, cost-effective, development foundation for applying GenAI based post-processing and improving early warning systems.