Implementation of a FAIR Compliant Automated Workflow for Infrastructures

The European infrastructure In-service Aircraft for a Global Observing System (IAGOS) (www.IAOGS.org) has implemented an automatic workflow for data management organizing the dataflow starting at the sensor towards the central data-portal located in Toulouse. The workflow is realized and documented using the web-based Django framework with a model-based approach using Python.

This workflow performs all necessary data processing and QA/QC tests to automated upload NRT processed data and serves the PI as basis for approval decisions. This includes repeated cycles for different stages of data maturity. The PI can monitor the status of all tasks by web-based reports produced by the Task Manager.  An automated reprocessing is possible by storing metadata on all steps as well as decisions of the PI. The implementation of the workflow is one big step to make IAGOS data handling compliant with the FAIR principles (findable, accessible, interoperable, reusable).

The workflow is easy adaptable to manage the workflow of other Infrastructures or research institutes. Thus, we will open the development under MIT license and invite other datacenters to contribute to the development.

Acknowledgments:

This work was supported by European Union's Horizon 2020 research and innovation programme under grant agreement No 824068 and by Helmholtz STSM Grant “DIGITAL EARTH”