KuHOP: Kubernetes-Based Orchestration for Hybrid Earth Science Computing

Numerical Weather Prediction (NWP), Earth Observation (EO), and Earth System Modeling workflows commonly span high-throughput computing (HTC) and high-performance computing (HPC): EO ingestion and pre-processing on HTC, data assimilation and model execution on HPC, and verification back on HTC. Most infrastructures lack integrated mechanisms to coordinate these heterogeneous environments, leading to manual workflow orchestration and ad-hoc data transfers.

We present KuHOP (Kubernetes-orchestrated Hybrid Operations Platform), an architecture that applies cloud-native orchestration concepts to hybrid HTC/HPC workflows. KuHOP uses Kubernetes as a unified control plane to describe, schedule, and monitor workflows across heterogeneous environments while preserving existing SLURM schedulers through native job submission. Containerized services and Kubernetes operators translate declarative workflow specifications into scheduler-specific jobs and manage data movement between clusters, enabling consistent observability without replacing established resource managers.

By treating HTC and HPC systems as backends within a single orchestration framework, KuHOP aims to improve portability and reproducibility of hybrid workflows through version-controlled, declarative definitions. The modular design supports GPU-accelerated AI/ML components and envisions multi-tiered resource federation. The Kubernetes control plane also allows institutions to deploy complementary services such as data streaming pipelines, AI inference endpoints, and custom dashboards as their needs evolve.

Developed at SURF, the Dutch national digital infrastructure provider, KuHOP targets both operational and non-operational Earth science workflows where hybrid computing is essential. It offers institutions a practical path to automate HTC/HPC coordination without abandoning existing infrastructure or losing the specialized capabilities of each environment.