EGU24-16013, updated on 31 Jul 2024
https://doi.org/10.5194/egusphere-egu24-16013
EGU General Assembly 2024
© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.

Potential of Deep Learning based quality control methods for soil moisture time series in an operational data service

Wolfgang Korres1, Tunde Olarinoye2, Fay Boehmer3, Kasjen Kramer4, Stephan Dietrich2, Marcel Reinhardt1, and Matthias Zink2
Wolfgang Korres et al.
  • 1Department M5 Geodesy and Remote Sensing, Federal Institute of Hydrology (BFG), Koblenz, Germany
  • 2International Center for Water Resources and Global Change (ICWRGC), Koblenz, Germany
  • 3Department M4 Geodata Center / WasserBLIcK / GRDC, Federal Institute of Hydrology (BfG), Koblenz, Germany
  • 4Department Z2 Information Technology and Information Management, Federal Institute of Hydrology (BfG), Koblenz, Germany

Soil moisture and its measurements are important in various fields and applications, from agriculture and hydrology to climate modelling, ecology and ecosystem health management. The monitoring of soil moisture gained widespread recognition in the early 2000s as an integral component of the hydrological and meteorological observation systems. This momentum was accelerated with the establishment of several soil moisture monitoring networks. The collection of data from different networks with a diversity of sensors and data formats, harmonization, quality control, archiving of in situ soil moisture data and ensuring the free accessibility of this data for end-users underline the motivation behind the foundation of the International Soil Moisture Network (ISMN) in 2009.

All in situ data sourced from the different providers undergo two quality checks. First, a visual inspection of the data excluding near real-time data and second, a rule-based automatic quality control procedure before inclusion in the ISMN database to ensure high quality research-ready soil moisture data for end-user. Thirteen different plausibility checks are applied to every singular hourly observation, which is flagged then as dubious if one of these checks fail, otherwise as “good”. These plausibility checks can be categorized into: i) a geophysical range verification, detecting the exceedance of certain thresholds (e.g., soil moisture values below 0 Vol.-%), ii) geophysical consistency methods, taking either ancillary in situ data if available or NASA’s GLDAS Noah data into account. An example is the flagging of soil moisture where soil temperature is negative). And iii) spectrum-based approaches, using the first and second derivatives of the entire soil moisture timeseries to detect dubious soil moisture patterns (i.e., spikes, breaks, and plateaus).

Publications in recent years point to the great potential of Deep Learning (DL) based methods for identifying anomalies in time series data. In this study, the potential of Long Short-Term Memory (LSTM) and Transformer models for anomaly detection in soil moisture time series is being investigated. Therefore, randomly selected time series from the ISMN are manually (visually) quality flagged (labelled). In order to be able to label these data we developed a guidance how to visually quality control in situ soil moisture data. Different Deep Learning methods in combination with varying external data sets (e.g. precipitation time series) are validated against the manually labelled data and compared to the previously implemented flagging method. The method will be further developed and evaluated for its use in ISMN operations. The incorporation of additional flagging information, especially when enhanced by Deep Learning methods, is anticipated to lead to a better usability of soil moisture data, as well as promoting a more robust quality control by the ISMN for its users in the future.

How to cite: Korres, W., Olarinoye, T., Boehmer, F., Kramer, K., Dietrich, S., Reinhardt, M., and Zink, M.: Potential of Deep Learning based quality control methods for soil moisture time series in an operational data service, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-16013, https://doi.org/10.5194/egusphere-egu24-16013, 2024.