EGU24-20642, updated on 18 Mar 2024
https://doi.org/10.5194/egusphere-egu24-20642
EGU General Assembly 2024
© Author(s) 2024. This work is distributed under
the Creative Commons Attribution 4.0 License.

Workflow Provenance with RO-Crate in Autosubmit

Bruno De Paula Kinoshita, Daniel Beltran Mora, Manuel G. Marciani, and Luiggi Tenorio Ku
Bruno De Paula Kinoshita et al.
  • Barcelona Supercomputing Center, Earth Sciences, (bruno.depaulakinoshita@bsc.es)

In this talk we present current work in Autosubmit to track workflow provenance using the community maintained open standard RO-Crate. Autosubmit is an experiment and workflow manager designed to conduct climate experiments in different platforms (local, HPC, cloud), and is part of different Earth Digital Twin initiatives (Destination Earth Climate Digital Twin, and the European Digital Twin of the Ocean).

Workflow managers have a central role in receiving user input, processing it with local and remote jobs that run on different platforms and that generate output data. RO-Crate enables tracking of workflow prospective (what should happen, e.g. workflow configuration, Slurm job settings) and retrospective (what happened, e.g. log files, performance indicators) provenance. By adopting an open standard that is used by other workflow managers (e.g. Galaxy, COMPSs, Streamflow, WfExS, Sapporo, and Autosubmit) and tools (e.g. Workflow Hub, runcrate) from various domains we show that it not only improves data provenance in Autosubmit, but also interoperability with other workflow managers and tools.

We also describe recent work to integrate RO-Crate with METACLIP, a language-independent framework for climate product provenance that was used in IPCC Atlas. METACLIP uses ontologies such as PROV to track the provenance of climate products. We describe how that relates to RO-Crate, and how we are integrating both via JSON-LD.

How to cite: De Paula Kinoshita, B., Beltran Mora, D., G. Marciani, M., and Tenorio Ku, L.: Workflow Provenance with RO-Crate in Autosubmit, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-20642, https://doi.org/10.5194/egusphere-egu24-20642, 2024.

Comments on the supplementary material

AC: Author Comment | CC: Community Comment | Report abuse

supplementary materials version 1 – uploaded on 17 Apr 2024, no comments