Auto-EC-Earth: An automatic workflow to manage climate modelling experiments using Autosubmit

The European community Earth system model EC-Earth is based on different and interoperable climate components simulating different processes of the Earth system. This makes it a complex model that requires multiple input data sources for its various model components, which can be run in parallel with multiple configurations and resolutions, demanding different computational resources in each case.

The EC-Earth software contains a minimum set of scripts to manage the compilation and execution of the simulations, but these are not enough to perform all the tasks that experiments demand nor to guarantee the traceability and reproducibility of the entire workflow in a high-productivity scientific environment. For that matter, the Auto-EC-Earth software has been developed at the Earth Sciences department of the Barcelona Supercomputing Center (BSC-ES) relying on Autosubmit, a workflow manager also developed at BSC-ES.

We take advantage of the automatization provided by the workflow manager that allows us to configure, manage, orchestrate and share experiments with different configurations and target platforms. The workflow manager allows the user to split the run into different tasks that are executed on different local and remote machines, like the HPC platform where the simulation needs to be performed. This is achieved in a seamless integration between Autosubmit, the EC-Earth tools, and the different machines where the scripts run, all without any user-input required after the initial setup and the launch of the experiment thanks to the workflow developments. Autosubmit also allows to ensure traceability of the actual runs, to have all the required data available for different kinds of experiments separated and well documented.

However, running the main part of the simulation is a cooperative task between the Autosubmit workflow manager and the different tools used for each model version. Auto-EC-Earth workflow has evolved to adapt the best possible to the EC-Earth model scripts that are present to help with the model runs. In EC-Earth 4, ScriptEngine is used to manage the run, and it has been fully integrated into the Auto-EC-Earth 4 workflow and used to set up the environment, while Autosubmit still manages the submission of jobs to the HPC and the dependencies between them.

Auto-EC-Earth is a great example of a workflow system that has been developed and used throughout the years, well established within the BSC-ES and used in multiple production cases, like multiple CMIP exercises as well as a reference for newer ESM workflows like the one developed in the Destination Earth project. It has also allowed the BSC-ES to collaborate with the EC-Earth community through the testing of the new releases of the model.