Understanding and recognition of geo-scenes based on multimodal spatial semantics to monitor complex urban systems

Urban geo-scenes (UGS) are an abstraction of the basic units of cities. Understanding and functional recognition of UGS is crucial to balancing and optimizing urban spatial layout, rationally allocating urban resources, and enhancing urban resilience and vitality. To construct UGS, urban geo-objects (UGO) e.g., derived from remote sensing must be combined with semantic information, which has seldom be done so far.  Consequently, this study designed a UGS recognition framework based on multimodal deep learning. First, we use very high-resolution satellite data to derive UGOs. Second, the self-built SE-DenseNet branch is used to mine deep physical visual features and social semantics from satellite image data and auxiliary data (POI, building footprints from UGOs). Finally, we build an urban fabric graph model to mine spatial semantics between UGOs.  In addition, a spatial semantic fusion module is introduced for the collaboration and interaction of multi-modal and multi-scale features. We evaluate the effectiveness of the proposed framework in the complex Beijing and Shenzhen regions of China. The overall accuracy is 91.35% and 90.24% respectively, which is higher than the state-of-the-art multimodal methods. In addition, our study also emphasizes the key role of spatial relationships and distribution patterns of UGO in UGS recognition, and the addition of POIs and building heights improves the recognition accuracy. The multimodal UGS recognition framework based on urban fabric can more effectively understand urban functions, thereby achieving urban planning and management.