User-turned-developer: Scientific software development for a national nutrient policy impact monitoring in Germany

Many countries in the EU fail to protect their waters adequately from nitrogen and phosphorus inputs (European Environment Agency. 2024), often originating from agricultural sources (Sutton 2011). Germany was found guilty by the European Court of Justice for insufficient implementation of the EU Nitrates Directive, for protection of waters from nutrient pollution from agriculture (European Court of Justice 2018). In response, Germany introduced a monitoring system for assessing the impact of the recently updated application ordinance, which implements the EU Nitrates Directive. This monitoring creates time series of pollution-related spatial indicators ranging from land use to modelled nutrient budgets. Input data on land use sources the Integrated Administration and Control System. The results are used by German authorities for reporting to the EU as well as national and regional water protection policy.

We present the technical concept, infrastructure and workflows established for this data-intensive, long-term project and discuss challenges and limitations when operating in the science-policy nexus. We aim to share good practices in modularization, automation, and reproducibility, and discuss strategies for efficient maintenance of scientific software development in context of long-term, policy-relevant monitoring projects.

Our system is designed to handle heterogeneous data with different levels of data protection requirements related to General Data Protection Regulation (GDPR). A modular structure was chosen to enhance usability and maintenance. Reproducibility is ensured through version-controlled, script-based software development. For efficiency, consistency and the streamlining of workflows reporting is automated and an ever-growing set of user-faced functions is bundled into a package. To ensure the possibility of advances in data preparation and modelling, a submission-based approach was chosen, recalculating all indicator times series each reporting year. This requires robust data management, reproducibility, and resilient workflows to accommodate evolving input data.

We still face challenges in handling Open Science principles, political stakeholder interests as well as GDPR. Similarly, scientific advances lead to updated results which may conflict with the need for clear and unambiguous outcomes of the authorities. Regular deadlines and stakeholder needs resulted in an organically grown code base, and sometimes cause neglection of quality checks and unit testing. Additionally, interaction between reproducible, script-based solutions and “traditional” workflows based on Microsoft Word are inefficient. The changing structure of the yearly gathered data hinders automatization of data processing. Due to this and the annual advances in the processing of the input data, maintaining the database is also challenging.  This we would like to share and discuss with other teams facing similar problem

Our system is tailored to handle heterogeneous and sensitive data of different sources producing reliable results and accommodating advances in data preparation and modelling in the long run. However, navigating technical limitations, good scientific practice and policymakers’ interests is challenging for us.

Literature

European Court of Justice (2018). European Commission against Federal Republic of Germany. Infringement Proceedings ‐ Directive 91/676/EEC.

European Environment Agency. (2024). Europe's state of water 2024: the need for improved water resilience. Publications Office.

Sutton, Mark A. (Ed.) (2011). The European nitrogen assessment. Sources, effects and policy perspectives. Cambridge 2011.