Workflow Provenance with RO-Crate in Autosubmit

In this talk we present current work in Autosubmit to track workflow provenance using the community maintained open standard RO-Crate. Autosubmit is an experiment and workflow manager designed to conduct climate experiments in different platforms (local, HPC, cloud), and is part of different Earth Digital Twin initiatives (Destination Earth Climate Digital Twin, and the European Digital Twin of the Ocean).

Workflow managers have a central role in receiving user input, processing it with local and remote jobs that run on different platforms and that generate output data. RO-Crate enables tracking of workflow prospective (what should happen, e.g. workflow configuration, Slurm job settings) and retrospective (what happened, e.g. log files, performance indicators) provenance. By adopting an open standard that is used by other workflow managers (e.g. Galaxy, COMPSs, Streamflow, WfExS, Sapporo, and Autosubmit) and tools (e.g. Workflow Hub, runcrate) from various domains we show that it not only improves data provenance in Autosubmit, but also interoperability with other workflow managers and tools.

We also describe recent work to integrate RO-Crate with METACLIP, a language-independent framework for climate product provenance that was used in IPCC Atlas. METACLIP uses ontologies such as PROV to track the provenance of climate products. We describe how that relates to RO-Crate, and how we are integrating both via JSON-LD.