3D object detection and tracking in debris flows with cameras and LiDARs

Debris flows are destructive mixtures of water and sediments. In mountain regions, debris flows are a relevant hazard as they threaten people and infrastructure. They move with rapid to extremely-rapid velocities, and often feature a coarse-grained front followed by a liquefied tail. Recently developed LiDAR sensors allow for long-term monitoring of debris-flow dynamics at high spatial (<2 cm) and temporal (10 Hz) resolution in the field, and provide the necessary high-quality data to improve our fundamental understanding of the complex debris-flow behavior.

Here, we present a framework for object detection in debris flows using deep learning algorithms which are trained on 2D camera images, and the results were then fused with LiDAR data to obtain 3D information. We used the YOLOv5 architecture to train a detector of breaking and diffuse surge waves, woody debris, boulders and rolling boulders. The detected objects were then tracked using the SORT algorithm. By subsequently reprojecting the image detections and tracks on the point clouds, 3D information such as velocity was determined. The detector performs very well on the different surge wave types with mean average precisions exceeding 0.9 in the test dataset. The other object types such as woody debris are more difficult to detect and track but still result in mean average precisions around 0.7. Finally, we show how surge waves interact with other objects of the flow by speeding them up and increasing their potential destructive impact. Continued monitoring and application of this method to more debris-flow events will result in an extensive dataset, which would be nearly impossible to obtain with a human operator only. Ultimately, our work will help to improve our understanding of debris-flow dynamics and reduce the hazard associated with this destructive process.