Portable Fourier Ptychography Microscope (FPM) with Deep Learning Capability for Planetary Exploration

Portable Fourier Ptychography Microscope (FPM) with Deep Learning Capability for Planetary Exploration

Thuc Phan¹, Thanh Nguyen^1,2, George Nehmetallah¹, and Shahid Aslam²

¹Catholic University of America, EECS, Washington DC, 20064, United States of America

²NASA, Goddard Space Flight Center, Greenbelt MD, 20771, United States of America

We present a portable FPM to perform wide field-of-view, high spatial-resolution imaging for biological samples. An NVIDIA Jetson-Nano board is used to computationally construct the images. The FPM has the potential to be compatible with space-based payloads [1].

The compact FPM experimental system is shown in Fig. 1. The FPM is controlled by an NVIDA Jetson nano board with state-of-the-art machine learning processing capability. The camera captures low-resolution bright field and dark field intensity images corresponding to multiple illuminations of each light from the LED array. Different diffracted orders of light serve as the sub-complexed fields in the Fourier domain and are stitched under a constraint iterative process to perform the high-resolution reconstruction. The LED positions based on setup coordinates were digitally calibrated.

Figure 1. A portable inferencing FPM device. (left) Optical schematic, and (right) portable FPM tabletop setup. The NVIDIA Jetson Nano board enables the development of an inferencing deep learning pipeline to provide near real-time monitoring. The device is being tested by using a Lenovo Legion laptop and the following software packages: Python 3.8, Tensorflow/RT, CUDA 11.2, SPI-spidev, Open CV 4.0.

High-NA illuminators are needed for large space-bandwidth product imaging [2]. To achieve high-angle illumination with a sufficient signal-to-noise ratio in the darkfield region, the illuminators has a dome-like shape rather than planar shape [3]. A programmable dome array is made of five separate planar LED arrays that can illuminate up to 0.80 NA with discrete control of the RGB LEDs (Red: 625 nm, Green: 530 nm, and Blue: 485 nm) was used. The working distance is 65 mm. A 4x Nikon objective with 0.13 NA and a working distance of 17.2 mm was placed under the sample stage. This results in a larger synthetic NA of 0.93.

The device can be operated in a traditional FPM mode capturing intensity images and reconstructing 3D phase images with no deep learning (DL) capability (see Fig. 2). The normal mode is used to provide data for the DL mode to perform fresh training and testing. The DL model automatically detects if the sample is out-of-focus and then mechanically transits the sample to the focus plane using a translation stage or by digitally translating it into the focus position using DL. To implement near real-time monitoring, a DL model is used to speed up the DL-based FPM reconstruction by using 1 in 10 of the LED’s. This enhances the overall throughput of the system [4].

Figure 2. Deep learning framework for FPM inferencing. The generator network was trained under conditional generative architecture to predict the phase map. The trained generator model then was loaded on NVIDA Jetson Nano board to perform inferencing.

 

References

[1] Nguyen, T., et al. “Portable flow device using Fourier ptychography microscopy and deep learning for detection in biosignatures,” SPIE Defense and Commercial Sensing, Vol. 11401 (2020)

[2] Sun, J., et al. “Resolution-enhanced Fourier ptychographic microscopy based on high-numerical-aperture illuminations.” Sci. Rep. 7, 1187 (2017)

[3] Eckert, R., et al. “Efficient illumination angle self-calibration in Fourier ptychography.” Appl. Opt. 57, 5434-5442 (2018)

[4] Tian, L., et al. “Multiplexed coded illumination for Fourier Ptychography with an LED array microscope,” Biomed. Opt. Express 5, 2376-2389 (2014)