Deep-Pathfinder: A machine learning algorithm for mixing layer height detection based on lidar remote sensing data

The mixing layer height (MLH) indicates the change between vertical mixing of air near the surface and less turbulent air above. MLH is important for the dispersion of air pollutants and greenhouse gases, and assessing the performance of numerical weather prediction systems. Existing lidar-based MLH detection algorithms typically do not use the full resolution of the ceilometer, require manual feature engineering, and often do not enforce temporal consistency of the MLH profile. Given the large-scale availability of lidar remote sensing data and the high temporal and spatial resolution at which it is recorded, this domain is very suitable for machine learning approaches such as deep learning. This presentation introduces a completely new approach to estimate MLH: the Deep-Pathfinder algorithm, based on deep learning techniques for image segmentation.

The concept of Deep-Pathfinder is to represent the 24-hour MLH profile as a mask (i.e., black indicating the mixing layer, white indicating the non-turbulent atmosphere above) and directly predict the mask from an image with lidar observations. Range-corrected signal (RCS) data at 12-second temporal and 10-meter vertical resolution was obtained from Lufft CHM 15k ceilometers at five locations in the Netherlands (2020–2022). High-resolution annotations were created for 50 days, informed by a visual inspection of the RCS image, the manufacturer's layer detection algorithm, gradient fields, thermodynamic MLH estimates, and humidity profiles of the 213-meter mast at Cabauw.

Our model is based on a customised U-Net architecture with MobileNetV2 encoder to ensure fast inference times. A nighttime variable indicated whether the sample occurred between sunset and sunrise and hence, whether an estimate of the stable or convective boundary layer was required. Model calibration was performed on the Dutch National Supercomputer Snellius. First, input samples were randomly cropped to 224x224 pixels, covering a 45-minute period and maximum altitude of 2240 meters. Then, the model was pre-trained on 19.4 million samples of unlabelled data. Finally, the labelled data was used to fine-tune the model for the task of mask prediction. Performance on a test set was compared to MLH estimates from ceilometer manufacturer Lufft and the STRATfinder algorithm.

Results showed that days with a clear convective boundary layer were captured well by all three methods, with minimal differences between them. The Lufft wavelet covariance transform algorithm contained a slight temporal shift in MLH estimates. Further, it had more missing data in complex atmospheric conditions. STRATfinder estimates for the nocturnal boundary layer were consistently low due to guiding restrictions in the algorithm. In contrast, Deep-Pathfinder followed short-term fluctuations in MLH more closely due to the use of high-resolution input data. Path optimisation algorithms like STRATfinder have good temporal consistency but can only be evaluated after a full day of ceilometer data has been recorded. Deep-Pathfinder retains the advantages of temporal consistency by assessing MLH evolution in 45-minute samples, however, it can also provide real-time estimates. This makes a deep learning approach as presented here valuable for operational use, as real-time MLH detection better meets the requirements of users in aviation, weather forecasting and air quality monitoring.