From Field to File: challenges and recommendations for handling hydrological data

In catchment hydrology, long-term data collection often starts as part of a (doctoral) research project. In some cases, the data collection continues on a limited budget, often using the field protocol and data management plan designed for the initial short-term project. Challenges and issues with the continued data collection are likely to arise, especially when there are multiple changes in the people involved. It is especially difficult for researchers who were not directly involved in the fieldwork to understand the data and must therefore rely on field notes and archived data. They then often encounter issues related to inconsistent metadata, such as inconsistent date-time formats and inconsistent or missing units, missing calibration files, and unclear file and processing script organization.

While the specific issues may sound very case-dependent, based on our own and other’s experiences from various research projects, it appears that many issues recur more frequently than one might expect (or be willing to admit). In this presentation, we will share our experiences with bringing spatially distributed groundwater level data collected in Sweden and Switzerland from the field to ready-to-use files. Additionally, we provide recommendations for overcoming the challenges during field data collection, data organization, documentation, and data processing using scripts. These include having a clear, detailed protocol for in the fieldwork and the data processing steps, and ensuring it is followed. Although protocols are often used, they are frequently not detailed enough or are not used as designed. The protocols might also not take into account the further use of the data, such as for hydrological modelling, beyond field collection.