ESA’s Planetary Science Archive: Improved interoperability

Introduction: The Planetary Science Archive (PSA)^[1] of the European Space Agency (ESA) has, over the past year, incorporated new interfaces for accessing the data. These new interfaces (an improved SFTP^[2], the PSA PDS^[3] API^[4] and ESA Datalabs^[5]), along with enhancements to the existing ones (TAP/EPN-TAP^[6] and UI^[7]), have considerably contributed to expand the interoperability mechanisms of the archive.

Here we describe various use cases (for both PDS3^[8] and PDS4^[9] data formats) in which all or part of these interfaces are used, showing different ways to obtain results from the same source archive.

Example Overview: The use case will simply be to access some PDS3 and PDS4 products by their logical identifier. These are the missions and instruments we will use for this (both Martian missions).

Mission	Format	Instr.	Product identifier
Mars Express	PDS3	HRSC	MEX-M-HRSC-3-RDR-EXT9-V4.0:DATA:HO799_0000_S23.IMG
ExoMars TGO	PDS4	ACS	urn:esa:psa:em16_tgo_acs:data_raw:acs_raw_hk_nir_20180613t180000-20180613t235959::1.0

 
Depending on the interface, the result will be either the download of the data product or the associated metadata (bundles/datasets, collections, label files, etc.)

GUI: The PSA Graphical User Interface (PSA GUI), which has been recently refurbished using the Angular framework, offering a modern and responsive design, allows the user to visualize data in a friendly way and search for PDS3 or PDS4 products via a rich set of filters. For our use case, we will use the Product ID field to search by the logical_identifier parameter (last version will be given by default).

Figure 1: Getting a PDS3/4 product in the PSA GUI and displaying it in the Mars map view.

TAP/EPN-TAP: The Table Access Protocol (TAP) service, running as a web application, allows searching PDS3 or PDS4 data by means of ADQL^[10] queries. This service is used as back-end for the GUI shown before; in addition, it can be accessed directly through a browser, programmatically (with Bash, Python, etc.) or from a client application such as Topcat. For the PDS3 use case, we will use the EPN-TAP service (Europlanet extension based on TAP) via the curl command:

$ curl -X 'GET' \ 'https://psa.esa.int/psa-tap/tap/sync?LANG=ADQL&REQUEST=doQuery&FORMAT=json&QUERY=select%20*%20from%20%20psa.epn_core%20where%20(obs_id=%27MEX-M-HRSC-3-RDR-EXT9-V4.0%3ADATA%3AHO799_0000_S23.IMG%27)' \ -H 'accept: *'

This returns a JSON file with expected metadata such as mission, instrument, bundle, collection, target, geometrical information, processing level, download path, etc.

SFTP: The new SFTP service, based on CrushFTP^[11], provides a rich web client to download data, as well as a standard SFTP access. It uses a virtual volume behind the scenes, to get the available public/private data using the FUSE^[12] technology. In this interface, the user may explore the structure of missions, datasets/bundles up to the desired level. For the use case, we will run the URL in the browser to get the same PDS3 product as before: https://psaftp.esac.esa.int/#/MARS-EXPRESS/HRSC/MEX-M-HRSC-3-RDR-EXT9-V4.0/DATA/O799/HO799_0000_S23.IMG

This will result in the download of the selected product to the user’s machine.

Figure 2: Accessing a PDS3/4 product and browsing its contents with the PSA new SFTP.

 PDS API: The PDS API implemented by NASA makes use of the PDS4 harvester to access PDS4 data of the PSA (although PDS3 data is not yet available in PDS, the PDS3 to PDS4 migration activity of the PSA coming up soon will cover the retrieval of these datasets). The API can be accessed from a browser (Swagger^[13]), command line, Jupyter notebook^[14], etc. This API allows the user to search for PDS4 products and their references by their lidvid (logical identifier and version id), collections, specifying the returned fields and filtering by any of the metadata contained in their label files. This ability to query arbitrary meta-data makes it very powerful.

To get the example PDS4 product via the PDS API we would just use the following URL:
https://pds.nasa.gov/api/search/1/products/urn:esa:psa:em16_tgo_acs:data_raw:acs_raw_hk_nir_20180613t180000-20180613t235959::1.0

Figure 3: Getting metadata of a PDS4 product through the PSA PDS API (via Swagger API).

The response body includes product’s information such as the mission, instrument, observation dates, targets, etc. It also includes the PSA label (XML file) URL, which links to our PSA FTP repository.

Various response formats are supported, including CSV, XML, JSON and the original PDS4 label.

ESA Datalabs: Finally, this new framework allows the user to work with planetary data without needing to download it. In the example, a Jupyter Notebook is used to access and display the MEX HRSC product. For reading PDS3 data, the PDR^[14] Python library is used.

Figure 4: Getting and plotting a PDS3 product with ESA Datalabs.

Acknowledgments: We are grateful to the PSA development team for their invaluable assistance in creating and refining the PSA software. We also thank our advisors and supporters for their guidance and encouragement throughout the process.

References:

[1] S. Besse et al. (2018), ESA's Planetary Science Archive: Preserve and present reliable scientific data sets, Planetary and Space Science, Volume 150, p. 131-140.

[2] Secured FTP: https://psaftp.esac.esa.int/

[3] Planetary Data System: https://pds.nasa.gov/

[4] PDS PSA: https://pds.nasa.gov/api/search-psa/1/

[5] ESA Datalabs: https://datalabs.esa.int/

[6] TAP/EPN-TAP: https://psa.esa.int/psa-tap/tap/

[7] PSA User Interface: https://psa.esa.int

[8] PDS3 Archiving Guide:

https://www.cosmos.esa.int/documents/772136/977578/ESDC-PSA-TN-0008.pdf

[9] PDS4 Archiving Guide:

https://www.cosmos.esa.int/documents/772136/977578/ESDC-PSA-TN-0002+Iss2Rel5-5.pdf

[10] Astronomy Data Query Language (ADQL):          https://www.ivoa.net/documents/ADQL/20180112/PR-ADQL-2.1-20180112.html

[11] CrushFTP: https://www.crushftp.com/index.html

[12] FUSE: https://github.com/libfuse/

[13] Swagger: https://swagger.io/

[14] Jupyter Notebook: https://jupyter.org/

[15] PDR python library: https://github.com/millionconcepts/pdr