Platform Engineering for Earth Observation: A Unified Approach to HPC and Cloud Systems

The growing volume of Earth Observation (EO) and Earth modeling data makes it increasingly impractical to download and analyze it locally. Furthermore, as cloud-native data formats and AI/ML-driven models gain popularity, the community requires powerful computing and storage solutions to efficiently process and analyze EO data. High-performance computing (HPC) and cloud infrastructures can help accomplish this, but both bring significant challenges in maintaining those resources, putting additional workloads on the scientists and developers.

In this paper, we will present our solution, which uses cloud-native technologies and a “Control Plane” approach to seamlessly interact with HPC scheduling endpoints like SLURM and PBS, as well as cloud infrastructure resources, allowing HPC jobs to be submitted and monitored directly from a Kubernetes-based infrastructure. In contrast to traditional IT architecture, Platform Engineering is concerned with lowering operational complexity by introducing control planes to provide self-service capabilities. By abstracting away the complexities of the underlying infrastructure, this method gives teams a customized, scalable, and dependable environment to suit their unique requirements. We will thoroughly analyze existing technologies, including their methodologies, strengths, limits, and potential as universal solutions. Furthermore, we will assess their adaptation to various cloud and HPC infrastructures, providing insights into their suitability for larger applications. 

We will conclude our discussion with practical examples showing how the technical benefits of these two computing paradigms, combined with the Platform Engineering approach, may be effectively used in real-world EO data processing scenarios.