Resolution enhancement using deep learning methods: an integrated workflow applied to real-world Back-Scattered Electron (BSE) data

Recent technological advances allow geoscientists to generate high-resolution (HR) imagery using a variety of different beam-forming mechanisms (e.g. visible light, X-Rays, or charged particles such as electrons and ions). One of the main limitations in producing HR data is the required acquisition time at high magnifications. For example, back-scattered electron (BSE) mapping of a standard petrographic thin section at a resolution of 50nm/pixel takes approximately 60 days and is associated with a storage requirement in the order of 700 GB. Deep-learning methods have proven effective for resolution enhancement in regular photographic images, and in this work we present an integrated image registration and upscaling workflow to enhance image resolution, using real-world BSE datasets.

The proposed workflow requires the acquisition of one, or multiple, HR regions within a region that is imaged at low-resolution (LR). Next, close to pixel-accurate image registration is performed by using the successive implementation of two concepts: i) first the precise location of the HR region within the LR region is determined by using a Fast Fourier Transform algorithm (Lewis, 2005), and ii) final image registration is achieved by iteratively calculating a deformation matrix that, using Newton’s method of optimization, is aiming to minimize an error function describing the differences between both images (Tudisco et al., 2017).

Subsequently, matching HR and LR image pairs are fed into a Generative Adversarial Network (GAN) that learns to produce HR images from the LR counterparts. A GAN consists of two neural networks, a generator and a discriminator. The generator produces synthetic HR data based on LR input, and the discriminator attempts to classify the data as either real HR or synthetic HR. The two networks are trained together in an adversarial process, with the goal of the generator producing synthetic data that the discriminator cannot distinguish from real data.

We demonstrate our method on a variety of large real-world datasets and show that it effectively increases the resolution of full-size BSE maps up to a factor of four, while being able to resolve important features. The upscaling of BSE data, with a factor of four, is associated with a 90% reduction in beamtime and a factor 16 reduction in storage requirements. Image registration, preprocessing, and model training on a high-performance workstation takes 12-24 hours. Having a trained model, inference can be done using a regular laptop.

[1] Lewis, J. P. "Fast normalized cross-correlation, Industrial Light and Magic." unpublished (2005).

[2] Tudisco, Erika, et al. "An extension of digital volume correlation for multimodality image registration." Measurement Science and Technology 28.9 (2017): 095401