A Retinex-based zero-shot learning method for permanently shadowed regions’ image enhancement

The moon is the earth's only natural satellite and it has a huge impact on the earth. Many countries have carried out a large number of lunar exploration missions over the years. The Lunar Reconnaissance Orbiter (LRO) launched by the United States in 2009 carried seven scientific instrument payloads to comprehensively map the lunar terrain and explore lunar resources. The optical narrow-angle camera (NAC) captures, processes and transmits back a large amount of image data [1]. These optical image data are intuitive data for studying the moon. Due to the axial tilt and undulating terrain, there are many areas (i.e., permanently shadowed regions (PSRs)) near the lunar south pole and north pole that are never illuminated by sunlight, so the imaging quality of these areas is poor. As shown in Fig. 1, PSR images have low signal-to-noise ratio (SNR), poor visibility, and most of the content is obscured. However, PSR may contain important materials such as water-ice, which makes it critical to study PSR with images. How to use low-light image enhancement methods to obtain high-quality PSR images has become a major challenge in this field.

Fig. 1. Optical image examples of the lunar south pole PSR. These images were captured by LRO NAC and stitched together in post-processing.

Image enhancement is an important task in computer vision, which aims to design algorithms to improve the interpretability of images. Image enhancement in the past was mainly divided into traditional methods and deep learning methods. Traditional methods represented by histogram normalization (HE)[2] and dehazing model theory[3] are limited in enhancement range and physical assumptions, and have low automation capabilities. Deep learning methods are divided into supervised learning methods, GAN-based methods[4] and zero-shot learning methods. Supervised learning relies on pre-made paired datasets consisting of original and optimized images. However, for PSR images, there is no standard optimized image, so supervised learning methods are not suitable for PSR image enhancement. Although GAN-based methods get rid of the restrictions on paired datasets, they still need to select data to meet the needs of the generator and discriminator, which makes the process more cumbersome. The zero-shot learning method, which does not rely on any data and can directly enhance images, is suitable for the enhancement of scarce data such as PSR images. An important category of zero-shot learning methods is the decomposition of low-quality images derived from Retinex decomposition theory. Regarding an image G(x), Retinex theory believes that it can be divided into a combination of reflection map and illumination map, that is:

G(x) = R(x)•I(x)

where x is the pixel position of the image, R and I represent the reflection map and illumination map respectively. For low-light images such as PSR images, I(x) is too dark, and R, which represents image features, should also be adjusted. By adjusting these sub-maps, the Retinex-based method can achieve image enhancement.

However, traditional Retinex only considers the content and brightness of the image (i.e., reflectance map and illumination map), but does not consider image noise. Low-light images such as PSR have a low number of photons transmitted to the CCD during imaging, resulting in dim visual phenomena. But low photons not only reduce the brightness of the image, but also introduce a lot of noise, which further reduces the quality of the image. The traditional Retinex method will expand the original noise while adjusting the brightness, which hinders image content analysis. To this end, we propose a Retinex-based model RZSL-PSR that introduces noise estimation. This model decomposes the PSR image into reflection map, illumination map and noise map. The specific formula is:

G(x) = R(x)•I(x) + N(x)

Based on this formula, the model uses the proposed sub-map learning network to output three sub-maps respectively. The network structure includes downsampling, concatenation fusion, and skip connection structures, which helps to obtain feature maps that integrate multi-scale, strong semantic information, and high-resolution information.    

Subsequently, the three output sub-maps are optimized to obtain the final output enhanced image. This process does not rely on learning image mappings of paired datasets, but instead utilizes a non-reference loss function to supervise model training. The loss function is as follows:

L = L_r + a•L_t + b•L_n

where L_r is the loss function to ensure that the output image conforms to the Retinex decomposition theory, L_t is the loss function to flatten the illumination uniformity of the image, and L_n is the loss function to estimate and suppress the noise in the dark area. Guided by these loss functions, the model learns nonlinear function mappings for enhancing PSR images, including increasing brightness and removing noise.

As shown in Fig. 2, the enhanced PSR image has better visibility than the original image. However, in order to highlight the landforms within the area for research on landforms and materials, we use USM sharpening for processing. The sharpening map results in Fig. 2 show that the landforms inside the PSR are highlighted, which helps to count the internal landforms.

Fig. 2. PSR’s (original image)-(enhanced image)-(sharpened image) comparison example. Among them (a) is the input PSR image, (b) is the enhanced

image, and (c) is the image after USM sharpening of the enhanced image.

Acknowledgements

This work was financially supported by the National Natural Science Foundation of China under No. 42030110, 41874010 and 41804025, the Natural Science Foundation of Hubei Province of China under No. 2022CFB123 and the Space Optoelectronic Measurement and Perception Lab in Beijing Institute of Control Engineering under No. LabSOMP-2023-09.

References

[1] Vondrak R, Keller J, Chin G, et al. Lunar Reconnaissance Orbiter (LRO): Observations for lunar exploration and science[J]. Space science reviews, 2010, 150: 7-22.

[2] Pisano E D, Zong S, Hemminger B M, et al. Contrast limited adaptive histogram equalization image processing to improve the detection of simulated spiculations in dense mammograms[J]. Journal of Digital imaging, 1998, 11: 193-200.

[3] Dong X, Pang Y, Wen J. Fast efficient algorithm for enhancement of low lightingvideo[M]//ACM SIGGRApH 2010 posters. 2010: 1-1.

[4] Meng Y, Kong D, Zhu Z, et al. From night to day: GANs based low quality image enhancement[J]. Neural Processing Letters, 2019, 50: 799-814.