AutoML: A Flexible and Scalable HPC Framework for Efficient Machine Learning in Atmospheric Modelling

The integration of Machine Learning (ML) into Earth System Sciences has revolutionized predictive modeling. However, the transition from local prototyping to large-scale deployment is often hindered by fragmented codebases and the manual overhead of managing complex hyperparameter tuning on High-Performance Computing (HPC) clusters. We present AutoML, a framework developed to automate and standardize the ML lifecycle in HPC environments by leveraging the open-source Autosubmit workflow manager.

AutoML employs a configuration-driven architecture that decouples model logic from workflow execution. By utilizing Autosubmit’s proven capability to handle complex dependencies and remote HPC environments, AutoML allows researchers to scale experiments—from initial prototyping to production-level global pipelines—through a single configuration file. This approach directly addresses the challenge of experiment reproducibility and efficiency within ML projects. The framework automates critical steps in the typical ML workflow, including hyperparameter search space optimization, multi-node distributed training, and dynamic resource allocation on heterogeneous HPC architectures.

We demonstrate the framework’s utility through Atmospheric Composition applications at the Barcelona Supercomputing Center (BSC). By providing a standardized structural template AutoML fosters collaboration and ensures that advancements in machine learning for atmospheric science are scalable, computationally efficient, and transferable across research lines.