Hydrological surfaces classification with Deep Learning using multiple sensors and exogeneous data

Traditional approaches for classifying hydrological surfaces - water, turbid water, salt pan, snow, and ice - usually rely only on one remote sensing dataset (often optical data like Sentinel-2). They face limitations under cloud cover areas and often confuse similar surface types (snow & salt pan, water & shadows). To overcome this, the study explores the use of Convolutional Neural Networks that can integrate spatial context, trained with multiple data sources like SAR (e.g., Sentinel-1), optical imagery, and exogenous inputs (weather, elevation). 

Deep Neural Networks are well-suited for texture extraction in remote sensing imagery and can efficiently handle inputs with multiple spectral bands. However, processing data from various sensor modalities introduces the challenge of aligning these inputs within a shared feature space where correlations can be effectively captured. To address this challenge, we developed a classical encoder-decoder architecture and explored the use of multiple encoders feeding into a single shared decoder. Two types of encoder families – EfficientNet and Swin Transformer – and two types of decoders – UNET and FPN – alongside various fusion methods were tried and showed similar performances.

For this study, a global multimodal database was gathered using open-source data from the Copernicus program. Initial trials with 17 labelled scenes (50GB) showed poor generalisation capabilities, leading to the extension of the dataset to 57 different scenes worldwide. Additional products were integrated, including Sentinel-1 (GRD VV+VH) data, 30m digital elevation models (ASTER GDEM), and meteorological data (from the ECMWF) to build the final 350GB database. Segmentation masks were generated semi-automatically (using a first version of our DL network) and then refined through visual inspection of Sentinel-2 images.

Results showed improved classification performance for all target classes when elevation data was included, and a dedicated dual-encoder-decoder model architecture proved particularly effective. On the other side, the integration of Sentinel-1 SAR data did not improve performance, likely due to the low temporal correlation between Sentinel-1 and Sentinel-2 acquisition (3-days average). Similarly, adding meteorological information did not enhance results, as our experiments showed that the model consistently disregarded scalar inputs regardless of integration approach.

Our model demonstrated notable robustness on the global database and was compared to existing CNES classification chains, including SurfWater (surface water detection) and Let-It-Snow (snow segmentation in mountains). Classification performance was comparable to SurfWater, though snow classification showed limitations in comparison to Let-It-Snow, particularly in the French Pyrenees.

The findings from this study underscore the potential of a multimodal approach in improving hydrological surface classification, particularly by incorporating data such as elevation. Future work could focus on increasing the volume of labelled data used to train the network to further enhance the model’s global applicability and precision across varied geographic and climatic conditions. Additionally, to fully leverage SAR imagery, reworking the database with more precise, directly annotated products would be essential. Finally, other approaches have to be tried in order to take into account meteorological data, for example using seasonality or more complex inputs. Other exogenous data could be added like terrain shadows.