Establishing FAIRness through all-actor approaches to data pipelines: Frameworks for successful development of data standards and pipelines at the UK’s National Centre for Atmospheric Sciences

The FAIR data principles are a common theme in many discussions and focus of work within research data management. Such work often focuses on particular parts of the data management lifecycle, for example: FAIR through data management planning, FAIR through data discovery and, more recently, areas of FAIR as applied to software and machine learning. 

However, whilst there are many successful attempts at enhancing metadata and data FAIRness for specific parts of the data lifecycle, there may be issues that only arise when considering the overall interconnections between the various stages and the associated actors. For example, a domain may follow common file and metadata conventions for data interoperability, such as CF conventions, enabling research to take place utilising multiple data sources, but pertinent metadata to long-term curation or wider end-usability may not be presented or indeed captured at source. This can have ongoing issues around the level that wider (true?) FAIRness that can be reached and present additional overheads for other actors wishing to handle such data resources, such as manual effort needed for full long-term curation or missed opportunities for data re-use in other spheres.

Recognising these issues and, crucially,  the interplay between all actors along the data lifecycle, the UK’s National Centre for Atmospheric Science (NCAS) have developed the frameworks to ensure all actors’ needs are considered. These are succinctly captured in the ‘NCAS Data Pyramid’, where each corner represents a given actor (data provider, long-term archive, those creating tools aiding data flows and utilisation, end-user community), whilst the sides explore the interconnections between these actors. All parts of the pyramid (corners and sides) provide a range of use-cases and requirements that need to be supported. This approach has enabled NCAS to then develop a range of data standards to enhance data FAIRness for surface and remote sensing data (including from ships and aircraft), imagery data and, in due course, laboratory data.

Furthermore, to aid establishing new data standards NCAS has developed data standards development framework, utilising the ‘Scope -> Define -> Develop -> Sustain’ data standard lifecycle:

• Scope: Identify community groups. Assess their needs. Determine the scope for the standard.

• Define the standard by: ensuring all stakeholder needs are covered; defining user-focused data products that it will deliver; and the underpinning standards to be drawn on for wider interoperability. 

• Develop: provider tools (including checkers for compliance); data delivery pipelines (including those workflows to capture internal/external metadata required for data use/contextualisation of data (e.g. project info); develop end-user data exploitation(visualisation) tools

• Sustain: having developed standards and workflows have a governance structure to maintain and manage future iterations of the standards development cycle. This must ensure that it refers back to the community groups (as in step 1). 

The approach also keeps wider inter-standards interoperability a key focus throughout. The success of this approach is demonstrated through the establishment of data pipelines aiding data to flow with associated metadata from provider to end-user and has seen wider adoption of NCAS data standards within the wider atmospheric community.