Seamless Upscaling Research from Cloud to HPC using eWaterCycle

The eWaterCycle platform provides hydrologists with a platform that allows them to work with each other's models and data without having to become a computer scientist in the process. The eWaterCycle platform supports existing hydrological models and makes them available for scientists using the BMI model interface as a communication layer. Models run in containers for reproducibility and dependency control. Popular hydrological models are readily available (PCRGLobWB, WFLOW, HBV, etc.). Scientists develop their analyses or experiments in the widely known JupyterHub environment. 

While in theory eWaterCycle can be installed and run on any hardware, in practice most users interact with it on the SURF Research Cloud, a cloud computing infrastructure available to the Dutch academic ecosystem. Until recently upscaling from Cloud to HPC infrastructure for larger model runs required extensive knowledge of the HPC system. Here we will present our work on building a seamless workflow that allows scientists to upscale their cloud based work to the Snellius supercomputer and the Spider grid computer without having to worry about technical issues like mounting points for (large) datasets and container engines.

Our workflow opens up the possibility for more scientists to benefit from HPC and Grid resources while focussing on their domain science. We present the workflow in such a format that it should be easily portable to other hybrid cloud - HPC infrastructures, including the DestinE systems.