A modularized workflow for processing heterogeneous agricultural land use data

In Germany each federal state is committed to collect required information on funding, farming practices and land use with an “Integrated Administration and Control System” (IACS) (Deutscher Bundestag 2014).

Based on the land parcel identification system (LPIS) as one of the core elements of IACS (European Commission, 2025), georeferenced data along with ancillary data are collected annually since 2005. Mandatory requirements for checks and on-site validations ensure a high data quality which makes IACS data very suitable for research purposes (Leonhardt 2024). Our goal is to create a nation-wide timeseries based on IACS data, that contains detailed information on land use, animal husbandry and farm statistics and can be used for comprehensive land use, soil, agricultural-policy and biodiversity research. Despite this, IACS data remain underused for scientific research due to the following challenges:

• Data protection: Obtaining and handling IACS data requires a legal agreement between the research project and the respective federal state including Data Usage Agreements.
• Data heterogeneity: All federal states have unique data processing workflows and historical changes in processing practices resulting in different data-types, -formats, structure, keys, encodings, etc.
• Data volume: Large storage volume, processing capacities and back-up systems with high security levels are required. Efficiency and data minimization is an important framework for the design of the processing workflows.

 

In this contribution we as user-turned-developers, want to show how we utilize our toolbox of open-source software (Linux, Bash, R, PostgreSQL/PostGIS, Python, GitLab), for a suitable modularized workflow to meet these challenges.

The first module is tailored to pre-process the data to its specific federal state qualities. Module two and three contain more general functions to grant machine readability. All data is then processed in a data cleaning workflow and imported into our PostgreSQL/PostGIS database.

We use our database for data harmonization by implementing modularized functions to handle different use cases.

The resulting harmonized datasets are provided to research teams with data protection clearance for federal state and year respectively. Harmonized tables are versioned as releases, to either grant reproducibility as well as to provide necessary updates.

Figure 1 Modularized workflow for IACS data processing towards a nation-wide harmonized timeseries

Reproducibly is granted by using script-based procedures that are stored and versioned in GitLab as well as extensive code documentation and automized file-based processing documentation.

Our modularization process lays the foundation for sustainable handling of complex administrative agricultural data and is a first step towards a software development approach.

Literature

European Commission (2025): Integrated Administration and Control System (IACS). Online available  https://agriculture.ec.europa.eu/common-agricultural-policy/financing-cap/assurance-and-audit/managing-payments_en

Deutscher Bundestag (2014): Gesetz über die Verarbeitung von Daten im Rahmen des Integrierten Verwaltungs- und Kontrollsystems nach den unionsrechtlichen Vorschriften für Agrarzahlungen. InVeKoS- Daten-Gesetz - InVeKoSDG, vom 5 (2019). Online available: https://www.gesetze-im-internet.de/invekosdg_2015/

Heidi Leonhardt, Maximilian Wesemeyer, Andreas Eder, Silke Hüttel, Tobia Lakes, Henning Schaak, Stefan Seifert, Saskia Wolff (2024): Use cases and scientific potential of land use data from the EU’s Integrated Administration and Control System: A systematic mapping review, Ecological Indicators, Volume 167, ISSN 1470-160X, https://doi.org/10.1016/j.ecolind.2024.112709.