Building Scalable Meteorological Data Pipelines with Dask, Zarr, and Cloud-Native Infrastructure

The Finnish Meteorological Institute (FMI) is modernizing its data processing workflows to address increasing data volumes and evolving user requirements. Current post-processing relies on a well-established C++-based system, where limited development capacity has become a bottleneck for extending functionality and supporting modern high-resolution and ensemble datasets. At the same time, the current setup restricts development of data processing workflows to a small group of specialists, limiting the ability of meteorologists, scientists, and other domain experts to directly contribute to algorithm development and data products.

To support scalable data processing and delivery, we are developing a Python-based platform built on the Dask ecosystem. Using tools such as Xarray and Zarr, the system enables parallel, distributed processing of large gridded datasets while remaining accessible to both developers and domain experts. The platform is designed to support diverse use cases, including operational product generation, algorithm development, and future machine learning workflows.

The architecture is designed around cloud-native principles, with initial deployment on FMI's OpenShift platform. Data workflows emphasize cloud-friendly formats, with Zarr as a candidate internal format for scalable processing. In addition to georeferenced outputs such as Cloud Optimized GeoTIFF for visualization and distribution, the system must also support efficient generation of time series products for downstream applications and users. Support for additional input formats, as well as integration with FMI’s existing data servers and APIs (e.g. OGC EDR), is under active development.

A prototype workflow using selected FMI datasets demonstrates how distributed processing and modern data formats can improve scalability, streamline data pipelines, and improve integration between data production and downstream services and users, while lowering the barrier for domain experts to directly contribute to data processing and product generation.