Machine learning emulation of stereo-based cloud-top height retrieval from Sentinel-2

The majority of supervised machine learning pipelines, particularly in the popular domains of natural language processing and computer vision, rely on manually annotated data. In geoscience applications, however, reference data are not necessarily derived from human annotation but could come as the output of explicit physical models or algorithms. These algorithms typically rely on simplifying hypotheses about the underlying physical processes and may be computationally expensive or applicable only to a limited subset of observations. In such circumstances, machine learning can be used to emulate explicit algorithms, with the objective of reproducing their outputs while potentially exploiting wider information pathways present in the data.

Beyond computational considerations, this hypothesis-light, data-driven framework allows for counterfactual testing by selectively removing input information and evaluating the model’s ability to recover similar predictions. For instance, in computer vision, color information can be removed by averaging RGB channels, while semantic or contextual information can be limited by progressively reducing the input patch size or by exploiting the inductive biases of different neural network architectures. In this way, one can identify additional cues in the input data linked to the physical property of interest, but also assess whether the model reproduces biases inherent to the reference algorithm. 

We explore this approach for high-resolution cloud-top height (CTH) estimation within the Clouds Decoded project, which uses Sentinel-2 (S2) multispectral observations (originally intended for land monitoring) to retrieve cloud properties. CTH can be estimated from S2 imagery using a stereo-based method that leverages the instrument’s geometry and inter-band delays. While effective, this approach is computationally demanding and relies on assumptions that restrict its applicability across diverse cloud scenes. We assess whether a neural network can learn to approximate this stereo-based CTH retrieval and analyse which textural, spectral, high-level semantic, or even geolocation-related cues the model might use to infer cloud height.