Contributions can include innovative data pipelines, integration of diverse datasets, visualization strategies, reproducible analysis workflows, and operational applications. The session aims to foster knowledge exchange on practical implementations, highlight cutting-edge approaches, and demonstrate how modern computational techniques can enhance the usability and impact of weather and climate data.
In recent years, Python has become an integral component of operational data processing pipelines in meteorology. Its growing ecosystem of scientific and domain-specific libraries has enabled flexible, scalable, and maintainable solutions for handling diverse data types, including in situ observations, remote sensing products, and numerical weather prediction (NWP) outputs. For meteorological service providers, Python offers a powerful framework to bridge research developments and operational applications.
In this contribution, we present practical experiences from integrating Python-based tools into the workflows of providers of meteorological data services. We highlight how a combination of widely used scientific libraries and community-driven meteorological packages has been employed to address key challenges such as data ingestion, format harmonization, quality control, and post-processing. Particular attention is given to the handling of heterogeneous data formats (e.g., GRIB, NetCDF, BUFR) and the efficient processing of large datasets in near-real-time environments.
We further discuss how Python libraries support the implementation of user-oriented services, including tailored forecast products, data visualization, and verification workflows. The modularity of the Python ecosystem allows rapid prototyping and iterative development, which is essential for responding to evolving user requirements and incorporating new data sources, such as AI-based forecasts.
At the same time, the use of Python in operational settings introduces challenges related to performance, dependency management, and long-term maintainability. We share lessons learned regarding the balance between flexibility and robustness, including strategies for testing, deployment, and integration with existing infrastructure.
By providing concrete examples from our operational environment, this contribution aims to illustrate both the benefits and limitations of Python as a core technology for meteorological service providers, and to inform best practices for its effective use in production systems.
How to cite: Schulze, D. and Hensel, T.: Experiences using Python libraries as a meteorological service provider, EMS Annual Meeting 2026, Utrecht, Netherlands, 6–11 Sep 2026, EMS2026-619, https://doi.org/10.5194/ems2026-619, 2026.
Air traffic management requires timely and reliable weather information to support decision-making at airports. To address this need, MeteoSwiss has developed adMET (Aerodrome Meteorological Forecast Information Service), an operational system that integrates meteorological forecast data and post-processing workflows to provide tailored weather guidance for air traffic controllers and other aviation stakeholders. The system currently supports the major Swiss airports of Zurich (ZRH) and Geneva (GVA).
The adMET workflow combines heterogeneous meteorological datasets, including numerical weather prediction output and derived forecast products, and transforms them into aviation-relevant information on wind, visibility, thunderstorms, temperature, pressure, and precipitation. Python-based processing pipelines are used to select suitable forecast sources and apply post-processing steps that transform raw model output into operationally useful products. Examples include spatial aggregation around airports and polar representations of vector quantities such as wind. These transformations allow aviation users to interpret forecast information directly in the context of arrival and departure planning. From a technical perspective, the system is implemented as a modular, containerized workflow deployed on OpenShift with additional showcase services running on AWS (Amazon Web Services). Data processing is driven by eventing to ensure optimal timeliness. Individual processing components are orchestrated through automated CI/CD pipelines and executed as independent jobs, enabling flexible deployment, scaling, and maintenance. Monitoring and operational oversight are implemented through cloud-native observability tools, including Prometheus-based metrics collection and Grafana/AWS CloudWatch dashboards. Data products are distributed through a standardized data exchange infrastructure, allowing seamless integration with external aviation stakeholders such as air navigation service providers.
The system demonstrates how modern software engineering practices - including reproducible workflows, containerized applications, and automated deployment pipelines - can support the operationalization of meteorological data processing. By bridging meteorological forecasting systems and aviation decision-support workflows, adMET illustrates how scalable Python-based infrastructures can enhance the usability and operational value of weather data for high-impact applications such as aviation.
How to cite: Landmann, J. M., Attinger, R., Aznar Siguan, G., Barras, H., Reiniger, T., and Wehrli, K.: Operational data pipelines for aviation weather decision support: the adMET system at MeteoSwiss, EMS Annual Meeting 2026, Utrecht, Netherlands, 6–11 Sep 2026, EMS2026-110, https://doi.org/10.5194/ems2026-110, 2026.
Integrating crowdsourced weather observations into national meteorological service (NMS) infrastructures remains a complex challenge, driven by heterogeneous data formats, inconsistent access methods, and the need for reproducible workflows across research and operational environments. These obstacles often lead to duplicated effort and hinder the transition from exploratory analysis to production‑grade applications, especially when working with large‑scale geospatial datasets that demand modern Python‑based, cloud‑native, and GIS‑enabled processing workflows.
To address this, we developed a proof-of-concept (PoC) with three connected components: open-source virtual research environment (i.e. SWIRRL), a spatial processing service (i.e. GRID), and an internal data platform (i.e. Party Platform) enabling data analysis directly in the cloud infrastructure.
- SWIRRL is a framework for VREs that supports JupyterLab-based workspaces, in combination with other interactive visualization tools and analysis workflows. It supports the automated collection of data and the execution of workflows that populate and transform those workspaces. Provenance is captured using standard provenance technologies allowing users to trace changes in data and tools. To support collaboration and Open Science, SWIRRL offers snapshot capabilities that allow users to publish their notebooks as a reproducible Jupyter Binder repository on GitHub.
- GRID is an internal KNMI development intended to encapsulate well-consolidated gridding methods used for current products and services. GRID exposes interpolation methods to convert datapoints into a continuous field and create contours lines based on the continuous field. The usage of GRID implies that the identical processing logic is applied across environments, hence aligning research and operational outputs.
- Party Platform is a serverless data platform that uses standard SQL. This platform provides a uniform interface to heterogeneous datasets, simplifying integration and ensuring reproducible access across research and operational contexts. Cloud-native architecture and Python integration allow rapid scaling of queries and processing tasks without impacting operational stability.
The novelty of our approach lies in combining serverless, SQL-accessible data with reusable spatial processing services. Together, these components form a dual-path architecture that supports both research and operations on shared infrastructure while remaining logically isolated. This design enables rapid experimentation on production-grade infrastructure without compromising operational stability, hence closing the gap between prototyping and deployment. Future work will add full PROV-O provenance for all GRID processing steps, delivering [or improving] end-to-end transparency. Taken as a whole, this approach shows how unifying serverless data access with reusable geospatial processing can transform the integration of crowdsourced observations and accelerate both research innovation and operational impact.
How to cite: Hooijer, T., van Ekris, J., Cornelis, L., Spinuso, A., Garcia Marti, I., van der Neut, I., and van der Ven, M.: Reproducible dual-path cloud-native architecture for working with crowdsourced weather observations, EMS Annual Meeting 2026, Utrecht, Netherlands, 6–11 Sep 2026, EMS2026-382, https://doi.org/10.5194/ems2026-382, 2026.
Since 1998, The European Climate Assessment & Dataset (ECA&D) forms the backbone of the Climate Data node of the Regional Climate Center (RCC) for Regional Association (RA) VI, enabling the sharing of daily meteorological surface observations between National Meteorological and Hydrological Services (NMHSs) across Europe and with scientists worldwide. The dataset is managed and updated by our team of researchers and engineers at KNMI (The Royal Netherlands Meteorological Institute).
The dataset has been steadily growing, driven in part by new EU data-sharing regulations, and ECA&D is updated monthly with contributions from NMHSs and other data-holding institutions, such as universities. Today, it encompasses daily data from over 24,000 stations across Europe, South Caucasus, and the Middle East.
Managing and updating a dataset of this scale requires processes that are efficient, timely, and reproducible. In this talk, we discuss essential software best practices that we have implemented over the past year to achieve this goal. While these best practices are broadly applicable, we demonstrate their implementation using Python workflows. Our aim is to inspire others to transform their research projects into scalable, production-ready solutions.
Using ECA&D as a case study, we highlight the following software best practices:
- Automation. Of the 33 monthly data ingestion processes, 26 are now fully automated. This has significantly reduced manual effort and the risk of human error, while improving reproducibility. It also eliminates reliance on implicit, individual knowledge of “how things are done,” making processes more transparent and maintainable.
- Monitoring. We set up basic monitoring for our data pipeline, allowing us to detect issues quickly and address them before the dataset is shared with users.
- Version control. Nearly all of our code is now managed with Git, improving traceability, reproducibility, and collaboration.We will share tips on how to set this up for existing non-tracked codebases, which contain a mix of programming languages.
- Automated testing and CI/CD pipelines. All new code undergoes unit and integration tests, which are executed automatically in the CI/CD pipeline before changes can be merged into the main codebase. We will elaborate on some Python tools such as pre-commit hooks, mypy and ruff.
By following these best practices, researchers and engineers can improve the reliability, reproducibility, and scalability of their data workflows, transforming them into robust, production-ready systems.
How to cite: Zaanen, K., van den Besselaar, E., Angevaare, J., Vlemmix, T., and van der Schrier, G.: Building Scalable and Reliable Climate Data Pipelines: Software Best Practices for ECA&D (European Climate Assessment & Dataset), EMS Annual Meeting 2026, Utrecht, Netherlands, 6–11 Sep 2026, EMS2026-333, https://doi.org/10.5194/ems2026-333, 2026.
Many European countries have developed climate services in recent years presenting locally downscaled climate projections. While each such climate service has its own peculiarities, a high degree of overlap and duplication can also be found between these services: the basic workflow of the Danish “Klimaatlas”, starting from the EURO-CORDEX ensemble, bias-adjusting against local datasets, and producing indicators, can also be seen in climate services in other countries such as Norway and Sweden. However, although each of these climate services is doing nearly the same thing, there has traditionally been little exchange of code between even neighbouring countries. The KAPy (Klimaatlases in Python) project aims to provide a platform for climate services to learn from each other, reduce duplication and enable rapid development of new climate services in other regions. KAPy is a climate data processing pipeline built on an open-source software stack centred on the Python language. The workflow control tool Snakemake from the field of bioinformatics provides reproducibility and scalability, xarray, xclim and xsdba provide core functionality, while the open-source paradigm enables collaboration and transparency. In addition to Klimaatlas Denmark, KAPy now also forms the core processing chain of a new climate service in Ghana, “Climate Atlas Ghana” and the current development of a climate service for Greece. To illustrate the ability of this tool to rapidly produce climate service information, I provide a detailed analysis of the efforts required to produce information that could be used for a hypothetical climate service for the town of Utrecht: starting from downloaded data, configuration, bias-adjustment and indicator production was completed in an afternoon working on a moderately sized terminal service. We also illustrate the wide applicability of the tool, showing its application from CMIP to convection-permitting models, and with a wide variety of outputs. We conclude with an open invitation to all to join the KAPy network as both users and developers, and thereby contribute to making climate services more transparent and widely accessible.
How to cite: Payne, M. R.: KAPy (Klimaatlases in Python) – an open-source pipeline for the production of climate services, EMS Annual Meeting 2026, Utrecht, Netherlands, 6–11 Sep 2026, EMS2026-727, https://doi.org/10.5194/ems2026-727, 2026.
Working with meteorological and climatological datasets often means grappling with a patchwork of file formats, incompatible data structures, and complex APIs before any real analysis can begin. Earthkit is an open-source ecosystem of Python-based tools designed to remove these obstacles, offering high-level, concise APIs that let researchers and operational users focus on their science rather than getting bogged down in the details of software.
Led by ECMWF, earthkit builds on decades of experience developing operational meteorological software to provide a modern, accessible Python toolkit. It integrates seamlessly with essential data science packages like xarray, pandas, and geopandas, bridging the gap between specialised meteorological capabilities and the tools data scientists already use.
Earthkit provides a consistent interface for accessing data from a wide range of sources - from local files and URLs to in-memory streams and remote web services - and across diverse formats including GRIB, netCDF, Zarr, and GeoTIFF, without requiring users to master the intricacies of each. Earthkit also brings capabilities to the open-source ecosystem that have previously been hard to find, including regridding across complex grid types such as HEALPix, Reduced Gaussian, ORCA, and ICON, as well as meteorological and hydrological algorithms, with GPU acceleration for computationally intensive operations.
This presentation introduces the earthkit ecosystem and demonstrates through practical examples how its components work together to support reproducible, format-agnostic, end-to-end workflows - from data retrieval and analysis through to publication-quality visualisation. Earthkit is designed for anyone who works with weather and climate data, and aims to make modern earth science data more accessible, interoperable, and actionable.
How to cite: Varndell, J.: Earthkit: Open-source tools for seamless earth science workflows, EMS Annual Meeting 2026, Utrecht, Netherlands, 6–11 Sep 2026, EMS2026-775, https://doi.org/10.5194/ems2026-775, 2026.
Supporters & sponsors
