A Stacking Ensemble Deep Learning Approach for Post Disaster Building Assessment using UAV Imagery

Traditional post-disaster building damage assessments were performed manually by the response team, which was risky and time-consuming. With advanced remote sensing technology, such as an unmanned aerial vehicle (UAV), it would be possible to acquire high-quality aerial videos and operate at a variety of altitudes and angles.  The collected data would be sent into a neural network for training and validating. In this study, the Object Detection model (YOLO) was utilized, which is capable of predicting both bounding boxes and damage levels. The network was trained using the ISBDA dataset, which was created from aerial videos of the aftermath of Hurricane Harvey in 2017, Hurricane Michael and Hurricane Florence in 2018, and three tornadoes in 2017, 2018, and 2019 in the United States. The Joint Damage Scale was used to classify the buildings in this dataset into four categories: no damage, minor damage, major damage, and destroyed. However, the number of major damage and destroyed classes are significantly lower than the number of no damage and minor damage classes in the dataset. Also, the damage characteristics of minor and major damage classes are similar under such type of disaster. These caused the YOLO model prone to misclassify the intermediate damage levels, i.e., minor and major damage in our earlier experiments. This study aimed to improve the YOLO model using a stacking ensemble deep learning approach with a image classification model called Mobilenet. First, the ISBDA dataset was used and refined to train the YOLO network and the Mobilenet network separately, and the latter provides two classes predictions (0 for no damage or minor damage, 1 for major damage or destroyed) rather than the four classes by the former. In the inference phase, the initial predictions from the trained YOLO network, including bounding box coordinates, confidence scores for four damage classes, and the predicted class, were then extracted and passed to the trained Mobilenet to generate the secondary predictions for each building. Based on the secondary predictions, two hyperparameters were utilized to refine the initial predictions by modifying the confidence scores of each class, and the hyperparameters were trained during this phase. Lastly, the trained hyperparameters were applied to the testing dataset to evaluate the performance of the proposed method. The results show that our stacking ensemble method could obtain more reliable predictions of intermediate classes.