The Science Application Store of the ESA Datalabs

Context:

The era of Big Data is transforming the way scientists approach their research, and ultimately how science progresses and discoveries are made. This is particularly true for both Space Sciences and Earth Observation, where current missions have highlighted a critical need for the creation and development of infrastructures capable of acquiring, processing, analysing and distributing Big Data from space. Current satellites have started to deploy customized systems for enabling preservation and to ensure easy access to processing systems, but despite having similar needs, these have been vertical approaches. Concomitantly, recent and newer missions, benefiting from technological progresses, have raised the level of collected data to unprecedented levels. Advances in observation systems and instruments require equal advances in data management and analysis.

 

The ESA Datalabs

The European Space Agency (ESA) is seeking to establish a reference framework for space mission exploitation by scientific communities: the ESA Datalabs [1] (see figure below). The fundamental principle of the ESA Datalabs is to move the user to the data and tools. The European Science Astronomy Center (ESAC) has started developing systems and building capacity to cater to the heterogeneous mission data exploitation system needs, while outlining also an umbrella architecture to aggregate generic functions, develop common sub-systems and interoperability concepts to rationalise and gather material and intellectual data assets.

 

The Science Application Store and datalabs

In this presentation we will present one of the modules of the ESA Datalabs, the Science Application Store (SCIAPPS).  The SCIAPPS will enable users to publish so-called datalabs within a trusted environment (the ESA Datalabs [2]), close to the scientific data, and permitting the whole scientific community to discover new science products in an open approach.  

A datalab includes an application (e.g. processors, codes, pipelines, analysis and visualisation tools) and its specific relationship with data assets which can be linked from the ESA Data Discovery portal also within the ESA Datalabs platform.

In a nutshell, the SCIAPPS provides scientific developers with a friendly, fast and guided process to publish datalabs in the ESA Datalabs platform, and provides a catalogue so that scientific users can search and run datalabs.

Datalabs

 Each datalab has its own specific function and purpose as defined by their developers. Practically a datalab consists of the source files, the metadata file (which capatures a synopsis of the function, purpose, scope and relation to data assests if any), and the build Docker image. No knowledge of docker file/images is required, SCIAPPS takes all the complexity out from the user which will be able to create datalabs even from just Jupyter file.

While each datalab is self-contained, specified data assets (e.g. from specific missions in the ESA archive) which may be mounted at runtime. 

SCIAPPS main functionalities

Scientific user will be able to search for datalabs of interest, query datalabs relevant to a specific data collection for a single mission or multi-mission or instrument/s. Users will be allowed to rate, leave feedback on datalabs, bookmarked datalabs of interest, and of course to run the datalabs (using resources from the ESA Datalabs platform).

Datalabs can be created by ESA Datalabs users with Scientific Developer permission. Developers can create, build, test, validate, and export their own datalabs. They can also share them with specific user/s or group/s or share them publicly with all users from the ESA Datalabs. To make a datalab public in the main catalogue, datalabs will undergo a moderation process transparent to the user.

 Summary

In the era of big data and data-intensive analaysis, the ESA Datalabs brings you the opportunity to bring your code directly to ESA’s infrasutructure, the ESA Datalabs. Here we will present the Science Application Store on the ESA Datalabs, which  provides with a simple, fast, and efficient way for users to publish applications close to the data, and provides a datalab catalogue for users to search and run datalabs. 

 

[1] ESA DATALABS: Towards a collaborative e-science platform for ESA. C. Arviset, V. Navarro, R. Alvarez et al., 5th Planetary Data and PSIDA 2021 (LPI Contrib. No. 2549)

[2] https://datalabs.esa.int