Automatic Land Cover Segmentation from Perfect Sensor Stereo Images with Height Information

The improved ability of imaging sensors to capture very high resolution (VHR) remote sensing images has been boosted by recent enhancement in the data processing algorithms. This improvement raises the potential of providing precise scene understanding for many applications. For instance, the task of semantic segmentation is an important one in the context of 3D building modelling. In this work, the main objective is to show the potential of using images of a stereo pair as inputs to a neural network trained for semantic segmentation into different urban classes. Actually, instead of increasing artificially the amount of training data, the use of stereo pair images can be seen as a realistic data augmentation, where the model will be trained to see the same object from different acquisition angles. From an experimental point of view, the results show that the method achieves better performances and gives a greater ability to generalize compared to the use of a single view. 

The segmentation process is performed using an encoder-decoder network architecture, namely the U-net network which includes an EfficientNet for the encoder part and a RefineNet for the decoder stage. The model is trained on Pleiades images involving different sources of ground truth (OpenStreetMap, IGN databases and in-house LCLU AI4GEO hierarchical labelled data). Additionally to the spectral information, height information is also considered to enhance the segmentation accuracy. This latter information is obtained using digital surface model (DSM). Indeed, classes identifying urban areas (building class for example) can be more easily discerned according to their height information. 

Furthermore, since Pleiades images are used as inputs of the proposed model, some geometrical issues need to be handled. To remove this complexity, a simulated imaging geometry of a perfect instrument is designed with no optical distortion and no high attitude perturbations. Resulting geometry is commonly called perfect sensor geometry. Since then, to avoid problems of geometric offsets between different data sources (satellite images in perfect geometry), terrain geometry of DSM/DTM and various ground truth databases, several tools have been developed to allow conversion between different geometries. The ortho-rectification is a commonly used geometrical correction that aims at presenting images as if they had been captured from the vertical. Therefore, this correction requires the availability of a HR digital terrain model (DTM) and may result on some distortions. In particular, some area may be occluded and others may arise a spreading effect of buildings.

 To address this issue, and preserve the native image information of the perfect sensor geometry, one key contribution of this work is to map the DSM data and ground truth image into the perfect sensor geometry. By doing this geometric processing and object positioning during the training process, better overlays between different data sources (stereo pair images, DSM model and ground truth data) are ensured and geometrical distortions and offsets can be avoided. In addition, the inferences can be done directly on the perfect sensor geometry without having to go through terrain geometries which requires high resolved DSM/DTM models.