Reproducible dual-path cloud-native architecture for working with crowdsourced weather observations

Integrating crowdsourced weather observations into national meteorological service (NMS) infrastructures remains a complex challenge, driven by heterogeneous data formats, inconsistent access methods, and the need for reproducible workflows across research and operational environments. These obstacles often lead to duplicated effort and hinder the transition from exploratory analysis to production‑grade applications, especially when working with large‑scale geospatial datasets that demand modern Python‑based, cloud‑native, and GIS‑enabled processing workflows. 

To address this, we developed a proof-of-concept (PoC) with three connected components: open-source virtual research environment (i.e. SWIRRL), a spatial processing service (i.e. GRID), and an internal data platform (i.e. Party Platform) enabling data analysis directly in the cloud infrastructure. 

• SWIRRL is a framework for VREs that supports JupyterLab-based workspaces, in combination with other interactive visualization tools and analysis workflows. It supports the automated collection of data and the execution of workflows that populate and transform those workspaces. Provenance is captured using standard provenance technologies allowing users to trace changes in data and tools. To support collaboration and Open Science, SWIRRL offers snapshot capabilities that allow users to publish their notebooks as a reproducible Jupyter Binder repository on GitHub. 

• GRID is an internal KNMI development intended to encapsulate well-consolidated gridding methods used for current products and services. GRID exposes interpolation methods to convert datapoints into a continuous field and create contours lines based on the continuous field. The usage of GRID implies that the identical processing logic is applied across environments, hence aligning research and operational outputs.

• Party Platform is a serverless data platform that uses standard SQL. This platform provides a uniform interface to heterogeneous datasets, simplifying integration and ensuring reproducible access across research and operational contexts. Cloud-native architecture and Python integration allow rapid scaling of queries and processing tasks without impacting operational stability. 

The novelty of our approach lies in combining serverless, SQL-accessible data with reusable spatial processing services. Together, these components form a dual-path architecture that supports both research and operations on shared infrastructure while remaining logically isolated. This design enables rapid experimentation on production-grade infrastructure without compromising operational stability, hence closing the gap between prototyping and deployment. Future work will add full PROV-O provenance for all GRID processing steps, delivering [or improving] end-to-end transparency. Taken as a whole, this approach shows how unifying serverless data access with reusable geospatial processing can transform the integration of crowdsourced observations and accelerate both research innovation and operational impact.