Deep-Pathfinder: Near-real-time detection of mixing layer height based on lidar remote sensing data and deep learning

Following the growing use of AI/ML in atmospheric sciences, this presentation will address the use of machine learning techniques for product generation from atmospheric measurement data. Our Deep-Pathfinder algorithm uses solely ceilometer observations to extract the mixing layer height (MLH), indicating the change between vertical mixing of air near the surface and less turbulent air above. The concept is to represent the sensor data as an image and the MLH profile as a corresponding mask, and directly predict the mask using image segmentation techniques. This concept is generic and, in principle, it can be applied to various sensor types.

Deep-Pathfinder is based on a customised U-Net architecture with MobileNetV2 encoder for fast inference and a nighttime variable to indicate whether a stable or convective boundary layer can be expected. Model training used range-corrected signal data from Lufft CHM15k ceilometers in the Netherlands (2020–2022), supplemented with 50 days of high-resolution annotations. First, input samples were randomly cropped to 224x224 pixels, covering a 45-minute period and maximum altitude of 2240 meters. Then, the model was pre-trained on the Dutch National Supercomputer Snellius using 19.4 million samples of unlabelled data. Finally, the labelled data was used to fine-tune the model for the task of mask prediction. Performance on a test set was compared to MLH estimates from ceilometer manufacturer Lufft and the STRATfinder algorithm, showing Deep-Pathfinder followed short-term fluctuations more closely.

Existing path optimization algorithms have good temporal consistency but can typically only be evaluated after a full day of ceilometer data has been recorded. Deep-Pathfinder retains the advantages of temporal consistency by assessing MLH evolution in 45-minute samples, using the full 12 s x 10 m resolution of the ceilometer. KNMI’s MLOps team is implementing the Deep-Pathfinder algorithm to run it in near-real-time on observations of CHM15k ceilometers in the Netherlands. The upcoming monitoring phase will highlight areas where the model could be further improved, using a model lifecycle of enhancing annotations for identified shortcomings, retraining and deployment. Further, we are investigating extending our concept to a multi-class segmentation problem to create atmospheric feature masks. These developments exemplify how machine learning techniques can be deployed towards enhancing operational usage of sensor data.