Pencil-chunking for cloud-native extreme value analysis of rainfall and drought thresholds

Extreme value analysis (EVA) is widely used to derive thresholds for flood and drought monitoring and anticipatory action, but its application at high spatial and temporal resolution is constrained by data access and computational limitations. This work presents a cloud-native approach for performing pixel-wise EVA on large meteorological datasets, highlighting the role of cloud-optimised data zarr formats, chunking strategies, and parallel computation.

Two datasets are used: NOAA CDR CMORPH rainfall at 30-minute resolution (1998–2024) and ECMWF ERA5-derived SPI and SPEI indices (1940–2025). Both provide long time series at fine grid resolution (0.08° and 0.25°), which are required to estimate return periods and derive actionable thresholds. The primary challenge lies in efficient data access and processing rather than in the statistical methods themselves.

Traditional HPC workflows rely on domain decomposition distributed across processors via MPI or OpenMP. The cloud-native approach presented here follows a similar pattern but shifts the optimisation focus from compute parallelism to data layout and streaming efficiency. By aligning chunking strategies with the temporal requirements of EVA, the workflow enables efficient access to long time series without repeated data loading or complex file orchestration.

The workflow uses cloud-based object storage and Dask distributed computing. Datasets are accessed through cloud-native formats: Kerchunk-based references generated via VirtualiZarr, Parquet catalogs, and Zarr-compatible Icechunk stores. Initial ingestion uses spatial-first ("pan") chunking within an Icechunk data store, which is efficient for spatial subsetting. This layout is rechunked into time-contiguous ("pencil") chunks using Dask's peer-to-peer shuffle-based rechunking, which redistributes data directly between workers with stable memory usage — eliminating separate preprocessing workflows.

For both CMORPH and ERA5 datasets, threshold estimation completes within hours using modest cloud resources. The workflow minimises repeated data scanning and limits data movement by collocating computation with storage. Results demonstrate that chunking strategy — particularly the transition to time-contiguous pencil chunks — is a critical enabler of scalable extreme value analysis for operational climate services in East Africa.