SHAFTS – A deep-learning-based Python package for Simultaneous extraction of building Height And Footprint from Sentinel Imagery

Buildings are common components in the urban environment whose 3D information is fundamental for urban hydrometeorological modeling and planning applications. In order to monitor building footprint and height across large areas on a regular basis, recent earth observation research has witnessed promising progress in mapping such information from publicly available satellite imagery by statistical methods using regression between multi-source remotely sensed data and target variables. However, most of them often involve tedious feature preprocessing, which constrains their capability to establish a comprehensive representation of an ever-changing and multi-scale urban system efficiently.

Considering this bottleneck, this work develops a deep-learning-based (DL) Python package-SHAFTS (Simultaneous building Height And FootprinT extraction from Sentinel Imagery) to estimate 3D building information at various scales. SHAFTS provides Convolutional Neural Networks (CNN) with the Multi-Branch Multi-Head (MBMH) structure to automatically learn representative features shared by building height and footprint mapping tasks from multi-modal Sentinel imagery and additional background DEM information. Besides, to leverage the power of big data infrastructures, SHAFTS offers essential functionality including automatically collecting potential reference datasets by web scraping and filtering appropriate input imagery from Google Earth Engine, which can effectively ease model upgrading and deployment for large-scale mapping.

To evaluate the patch-level prediction skills and city-level spatial transferability of developed models, this work performs diagnostic performance comparisons in 46 cities worldwide by using conventional machine-learning-based (ML) models and CNN with the Multi-Branch Single-Head (MBSH) structure as benchmarks. Patch-level results show that DL models successfully produce more discriminative feature representation and improve the coefficient of determination of building height and footprint prediction over ML models by 0.27-0.63, 0.11-0.49, respectively. Moreover, stratified error assessment reveals that DL models effectively mitigate severe systematic underestimation of ML models in the high-value domain. Additionally, within the DL family, comparison in spatial transferability demonstrates that the MBMH structure improves the accuracy of CNN and reduces the uncertainty of building height predictions in the high-value domain at the refined scale. Therefore, multi-task learning can be considered as a possible solution for improving the generalization ability of models for 3D building information mapping.