Toward a Robust Pipeline for the 3D Reconstruction Of The Hermean Surface: The SIMBIO-SYS STC Automated Processing Framework

Introduction

Next year, in 2027, the BepiColombo joint mission between the European Space Agency (ESA) and the Japan Aerospace Exploration Agency (JAXA) dedicated to the comprehensive exploration of the planet Mercury is set to enter its nominal phase. The SIMBIO-SYS instrument suite will act as the mission’s ‘eyes’, designed to provide comprehensive imaging and spectroscopic observations of the planet's surface through three independent optical channels.

The STereo imaging Channel (STC) is a core component for the SIMBIO-SYS suite objectives, designed specifically to achieve stereo global 3D mapping of Mercury and targeted color observations.  STC utilizes an innovative push-frame concept using a single detector but two separate optical paths, one tilted forward and one backwards, allowing stereo pairs to be captured by utilizing the movement of the satellite.  The instrument will generate a massive amount of data, in the order of hundreds of thousands of frames to complete the global mapping. This means that processing such large volumes of data for mosaicking and stereogrammetry will require a pipeline that is as robust and automated as possible, currently being developed by the INAF OAPd team.

 

Processing Pipeline Overview

Due to the orbital geometry and planning conditions, the frame footprints of the two channels will not align perfectly, so the mosaicking and stereometric processing will be carried out in blocks. This also has the advantage of simultaneously improving attitude and position information through bundle block adjustment, thereby mitigating artefacts caused by inaccuracies in kernels and surface models. In the pipeline, the Mercury’s surface will be dynamically tessellated into rectangles of approximately 5 x 5 frames, varying according to the geometric conditions, for each channel and with an appropriate lateral overlap to avoid geometric inconsistencies (such as gaps) and facilitate subsequent mosaicking. The pipeline will ingest the images after being radiometrically corrected, as well as the initial orbital pointing data expressed in the Mercury body-fixed Cartesian frame. Intrinsic camera parameters (focal length, principal point, distortion) are obtained from validated pre-flight calibration data.

The frames block is geometrically refined through a progressive “middle-out” bundle adjustment strategy. Starting from the central image and expanding outward bilaterally (both horizontally and vertically), successive adjacent frame pairs are matched using SIFT-based feature detection [3]. Corresponding image points are back-projected onto a reference Mercury global DTM (at 663 m/px) [4]  to establish absolute 3D tie-point coordinates, anchoring the adjustment to the hermean topography. A Ceres-based nonlinear least-squares solver [5] minimizes reprojection residuals jointly over the expanding set of image poses, while previously corrected frames are held fixed to prevent sequential drift. The resulting projection matrices define the rectified mosaics geometry.

Following the mosaicking process, a second alignment stage allows any residual errors between the two mosaics to be corrected. In this stage, the mosaic from the second channel is aligned with the first using a similar pre-alignment and bundle adjustment procedure as for the single frames.  

For an initial, coarse matching, a tile-based SIFT is used, whereby the mosaics are divided into subsets that are compared in parallel. This allows tie points to be defined as uniformly and evenly as possible across the images being processed. The stereo processing part is a direct evolution of the 3DPD software, which has been under active development since 2018 [1][2], for processing DTMs from the CaSSIS (Colour and Stereo Surface Imaging System) instrument aboard ESA’s Trace Gas Orbiter. The CaSSIS instrument, in fact, acquires stereo imagery using a similar push frame architecture, in which the two acquision of the stereo pair are read out during a single spacecraft overpass, yielding a forward/backward stereo pair using a rotation mechanism. 

In the 3DPD, the seed points information obtained with SIFT is expanded to the whole image using Delaunay triangulation, resulting in a TIN surface of parallaxes, represents the first sparse disparity maps used as initialization for the subsequent dense matching process. The dense matching phase employs multi-scale processing, utilizing image pyramids to implement a coarse-to-fine strategy. This approach effectively constrains search space and minimizes matching outliers (blunders) under the hypothesis that the target is continuous. The process begins at the coarsest pyramid level using Normalized Cross-Correlation (NCC) for initial matching. These results are then propagated to successively higher resolutions: at each stage, the disparities for new points are estimated through bilinear interpolation of their neighbors. This iterative refinement continues until the highest resolution level is reached, where Least-Squares Matching (LSM) is applied to achieve final sub-pixel precision [6].

Using the previously corrected projection matrices, the point cloud derived from dense matching can finally be projected into a chosen reference system, thereby refined in the post-processing and interpolated into a DTM and orthorectified image blocks.

Testing

Active development of the pipeline is facilitated by two different test datasets, which enable its effectiveness to be assessed and fine-tuning to be carried out before the arrival of STC data: synthetic images generated using the PLanetary Image Simulator (PLAS) [7]  and the images of Mars from the CaSSIS instrument. PLAS enables the creation of synthetic images that faithfully reproduce the format and characteristics of STC data. The images are generated by faithfully reproducing the lighting and attitude conditions derived from the SPICE kernels, using as the surface three-dimensional surface models of the Moon and Mercury. Furthermore, the simulated images allow for the controlled testing of pointing errors and noise, to quantify their impact. Regarding CaSSIS, although the target (Mars) and ground resolution differ by approximately an order of magnitude (approximately 4.5 m/px CaSSIS), the use of real data allows for the handling of unforeseen errors in the kernels and the presence of unexpected noise.

[1] Simioni, E., Re, C., Mudric, T., et al. (2021). PSS 198, 105165. https://doi.org/10.1016/j.pss.2021.105165

[2] Re, C., Fennema, A., Simioni, E., et al. (2022). PSS 219, 105515. https://doi.org/10.1016/j.pss.2022.105515

[3]  Lowe, D.G. (2004). IJCV 60, 91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94

[4]  Becker, K. J., Robinson, M. S., Becker, T. L et al. (2016). 47^th LPSC

[5] Agarwal, S., Mierle, K., & The Ceres Solver Team. (2023). Ceres Solver (Version 2.2). https://github.com/ceres-solver/ceres-solver

[6] Gruen (1985) S. Afr. J. Photogramm. Remote Sens. Cartogr. 14.3 175-187.

[7] Re, C., Tullo, A., La Grassa, R., et al. (2024). EPSC2024-851. https://doi.org/10.5194/epsc2024-851