A Domain-Change Approach to the Semantic Labelling of Remote Sensing Images

For many years, image classification – mainly based on pixel brightness statistics – has been among the most popular remote sensing applications. However, during recent years, many users were more and more interested in the application-oriented semantic labelling of remotely sensed image objects being depicted in given images.

In parallel, the development of deep learning algorithms has led to several powerful image classification and annotation tools that became popular in the remote sensing community. In most cases, these publicly available tools combine efficient algorithms with expert knowledge and/or external information ingested during an initial training phase, and we often encounter two alternative types of deep learning approaches, namely Autoencoders (AEs) and Convolutional Neural Networks (CNNs). Both approaches try to convert the pixel data of remote sensing images into semantic maps of the imaged areas. In our case, we made an attempt to provide an efficient new semantic annotation tool that helps in the semantic interpretation of newly recorded images with known and/or possibly unknown content.

Typical cases are remote sensing images depicting unexpected and hitherto uncharted phenomena such as flooding events or destroyed infrastructure. When we resort to the commonly applied AE or CNN software packages we cannot expect that existing statistics, or a few initial ground-truth annotations made by an image interpreter, will automatically lead to a perfect understanding of the image content. Instead, we have to discover and combine a number of additional relationships that define the actual content of a selected image and many of its characteristics.

Our approach consists of a two-stage domain-change approach where we first convert an image into a purely mathematical ‘topic representation’ initially introduced by Blei [1]. This representation provides statistics-based topics that do not yet require final application-oriented labelling describing physical categories or phenomena and support the idea of explainable machine learning [2]. Then, during a second stage, we try to derive physical image content categories by exploiting a weighted multi-level neural network approach that converts weighted topics into individual application-oriented labels. This domain-changing learning stage limits label noise and is initially supported by an image interpreter allowing the joint use of pixel statistics and expert knowledge [3]. The activity of the image interpreter can be limited to a few image patches. We tested our approach on a number of different use cases (e.g., polar ice, agriculture, natural disasters) and found that our concept provides promising results.  

[1] D.M. Blei, A.Y. Ng, and M.I. Jordan, (2003). Latent Dirichlet Allocation, Journal of Machine Learning Research, Vol. 3, pp. 993-1022.
[2] C. Karmakar, C.O. Dumitru, G. Schwarz, and M. Datcu (2020). Feature-free explainable data mining in SAR images using latent Dirichlet allocation, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 14, pp. 676-689.
[3] C.O. Dumitru, G. Schwarz, and M. Datcu (2021). Semantic Labelling of Globally Distributed Urban and Non-Urban Satellite Images Using High-Resolution SAR Data, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 15, pp. 6009-6068.