Intelligently Gapfilling Earth Observations: Towards a coherent observational view of Land Hydrology

Earth observations have many missing values. Their complex patterns of missingness can be a significant hurdle for studying Earth system dynamics and climate change impacts. To overcome this issue, missing values are regularly imputed, i.e. infilled, using techniques such as interpolation. However, the common practice to do this for each variable separately can negatively affect the covariance between different data products, resulting in biased estimates. Moreover, relying solely on interpolation for infilling missing values makes only inefficient use of information that may be available from other variables at the same location in space and time.

Here we propose a modular gap-filling algorithm that exploits the multivariate nature of Earth system observations and builds upon the notion that if a value is missing, it is likely that some other variables will be observed at the same location and time and their relationship can be learned. To this end, the algorithm expands upon simple interpolation by additionally applying a statistical imputation method that is designed to account for covariance across variables.

The algorithm is tested using gap-free reanalysis data of relevant variables to land surface processes: ground temperature, precipitation, terrestrial water storage and soil moisture. These variables were masked to match missingness patterns of remote sensing observations. Subsequently, the gap fill estimates can then be compared to the original reanalysis values to assess the merit of the gap fill.

Overall, estimates of the proposed algorithm have lower bias and higher correlation compared to simple interpolation. Furthermore, we demonstrate that the multivariate core of the algorithm improves the physical consistency across the considered variables. In case studies focussing on large-scale droughts, extreme values are correctly reconstructed even in cases of high fraction of missing values. The algorithm can thus be used as a flexible tool for gapfilling remote sensing and in-situ observations commonly used in climate and environmental research and create a coherent observational dataset of a flexible set of observational products.