Building Scalable and Reliable Climate Data Pipelines: Software Best Practices for ECA&D (European Climate Assessment & Dataset)

Since 1998, The European Climate Assessment & Dataset (ECA&D) forms the backbone of the Climate Data node of the Regional Climate Center (RCC) for Regional Association (RA) VI, enabling the sharing of daily meteorological surface observations between National Meteorological and Hydrological Services (NMHSs) across Europe and with scientists worldwide. The dataset is managed and updated by our team of researchers and engineers at KNMI (The Royal Netherlands Meteorological Institute).

The dataset has been steadily growing, driven in part by new EU data-sharing regulations, and ECA&D is updated monthly with contributions from NMHSs and other data-holding institutions, such as universities. Today, it encompasses daily data from over 24,000 stations across Europe, South Caucasus, and the Middle East.

Managing and updating a dataset of this scale requires processes that are efficient, timely, and reproducible. In this talk, we discuss essential software best practices that we have implemented over the past year to achieve this goal. While these best practices are broadly applicable, we demonstrate their implementation using Python workflows. Our aim is to inspire others to transform their research projects into scalable, production-ready solutions.

Using ECA&D as a case study, we highlight the following software best practices: 

• Automation. Of the 33 monthly data ingestion processes, 26 are now fully automated. This has significantly reduced manual effort and the risk of human error, while improving reproducibility. It also eliminates reliance on implicit, individual knowledge of “how things are done,” making processes more transparent and maintainable.
• Monitoring. We set up basic monitoring for our data pipeline, allowing us to detect issues quickly and address them before the dataset is shared with users. 
• Version control. Nearly all of our code is now managed with Git, improving traceability, reproducibility, and collaboration.We will share tips on how to set this up for existing non-tracked codebases, which contain a mix of programming languages.
• Automated testing and CI/CD pipelines. All new code undergoes unit and integration tests, which are executed automatically in the CI/CD pipeline before changes can be merged into the main codebase. We will elaborate on some Python tools such as pre-commit hooks, mypy and ruff. 

By following these best practices, researchers and engineers can improve the reliability, reproducibility, and scalability of their data workflows, transforming them into robust, production-ready systems.