ESA’s Planetary Science Archive: ways to download data

Introduction

The European Space Agency’s Planetary Science Archive (PSA, https://psa.esa.int) is a multi-mission system curating, preserving and serving the data of ESA’s planetary missions. All data are stored in the Planetary Data System (PDS) format and can be searched and retrieved through a unified user interface and APIs. Here we highlight the options to download data from the different interfaces (UI, TAP/EPN-TAP, SFTP, and the PDS API)  that PSA supports.

Use Case Overview

The use case will simply be the download of a particular PDS4 ExoMars Trace Gas Orbiter (TGO) product from the Atmospheric Chemistry Suite (ACS) instrument by using its logical identifier (and version identifier if needed). For this example, the product LIDVID (logical identifier and version identifier) is the following:

urn:esa:psa:em16_tgo_acs:data_raw:acs_raw_hk_nir_20180613t180000-20180613t235959::4.0

The result will be the download of the data product and the associated metadata (bundles/datasets, collections, label files, etc.). For simplicity, the example below will only download the label file of the product and its data.

Also, note that if the version identifier is not provided, the PSA will give the latest one ingested in the system by default.

Of course, all the downloads explained here are also applicable to PDS3 products.

Finally, the PSA controls the access to download private data to specific privileged users from any interface, if applicable.

Graphical User Interface

The PSA Graphical User Interface (PSA UI)

• Direct download (synchronous).
• Scripting download (asynchronous): This offers both wget, curl and bash scripting files, with file granularity. Like this example

  • # WARNING: This script requires execute permission.

    # To grant execute permission, run the following command:

    # chmod +x script_name.sh

    curl --create-dirs -o 'PSA_11-05-2026_141738/em16_tgo_acs/data_raw/Science_Phase/Orbit_Range_2400_2499/Orbit_2476/acs_raw_hk_nir_20180613T180000-20180613T235959__4_0.tab' 'http://psadev03.ebe.lan/psa-tap/data?retrieval_type=GEN_PRODUCT&id=urn:esa:psa:em16_tgo_acs:data_raw:acs_raw_hk_nir_20180613t180000-20180613t235959::4.0&FILENAME=acs_raw_hk_nir_20180613T180000-20180613T235959__4_0.tab'

    curl --create-dirs -o 'PSA_11-05-2026_141738/em16_tgo_acs/data_raw/Science_Phase/Orbit_Range_2400_2499/Orbit_2476/acs_raw_hk_nir_20180613T180000-20180613T235959__4_0.xml' 'http://psadev03.ebe.lan/psa-tap/data?retrieval_type=GEN_PRODUCT&id=urn:esa:psa:em16_tgo_acs:data_raw:acs_raw_hk_nir_20180613t180000-20180613t235959::4.0&FILENAME=acs_raw_hk_nir_20180613T180000-20180613T235959__4_0.xml'

• Send the products to the download manager: This provides a visual tool to enable/disable products by category and referenced products (auxiliary, documents, calibration, SPICE, browse, geometry, etc).

All these offer zip or tar.gz and the possibility to flatten the directory structure as options.

TAP/EPN-TAP

In case of the Table Access Protocol (TAP), this service (running as a web application) allows the search for PDS3 or PDS4 data via ADQL. This is used as the back-end for the UI (as seen before) but of course can be used through the command line. Through TAP, both PDS3/PDS4 data, and both observational and auxiliary products can be downloaded. The first ones contain the primary data from a scientific or engineering observation, such as an image, spectrum, table, etc, while the auxiliary ones provide supporting information needed to interpret, calibrate, locate, or use those observations, such as SPICE kernels, documentation, geometry, etc. With curl (via the POST method), we can then download the PDS4 observational product used for the example:

curl -X POST "http://psadev03.ebe.lan/psa-tap/data?retrieval_type=GEN_PRODUCT&error_format=TEXT" \

     -H "Content-Type: application/zip" \

     -d '{"id": "urn:esa:psa:em16_tgo_acs:data_raw:acs_raw_hk_nir_20180613t180000-20180613t235959::4.0"}' \

     -OJ

This will return a zip file with the expected data and metadata. The inventory of the output should be like:

em16_tgo_acs/data_raw/Science_Phase/Orbit_Range_2400_2499/Orbit_2476/acs_raw_hk_nir_20180613T180000-20180613T235959__4_0.tab

em16_tgo_acs/data_raw/Science_Phase/Orbit_Range_2400_2499/Orbit_2476/acs_raw_hk_nir_20180613T180000-20180613T235959__4_0.xml

SFTP

An SFTP service based on the CrushFTP tool, provides a rich web client to download data, as well as a standard SFTP access. Behind the scenes, it uses a virtual volume to get the public/private data by using the FUSE technology. In this interface, the user may explore the structure of missions, datasets/bundles until reaching the desired one. For the use case, we will run the URL in the browser to download the PDS4 label file:

https://psaftp.esac.esa.int/ExoMars2016/em16_tgo_acs/data_raw/Science_Phase/Orbit_Range_2400_2499/Orbit_2476/acs_raw_hk_nir_20180613T180000-20180613T235959__4_0.xml

Also, it can be retrieved via command line:

$ sftp anonymous@psaftp.esac.esa.int

Connected to psaftp.esac.esa.int.

sftp> cd /ExoMars2016/em16_tgo_acs/data_raw/Science_Phase/Orbit_Range_2400_2499/Orbit_2476/

sftp> get acs_raw_hk_nir_20180613T180000-20180613T235959__4_0.xml

Fetching /ExoMars2016/em16_tgo_acs/data_raw/Science_Phase/Orbit_Range_2400_2499/Orbit_2476/acs_raw_hk_nir_20180613T180000-20180613T235959__4_0.xml to acs_raw_hk_nir_20180613T180000-20180613T235959__4_0.xml

acs_raw_hk_nir_20180613T180000-20180613T235959__4_0.xml                                                                            100%   14KB 654.8KB/s   00:00

sftp> exit

PDS API:

The PDS API implemented by NASA makes use of the PDS4 harvester to access the PDS4 data (PDS3 data is currently not available through this API, but the PDS3 to PDS4 migration activity will eventually allow these datasets to be ingested). The API can be accessed from the browser (Swagger), Jupyter notebook, command line, etc. This API allows users to search for PDS4 products and their meta-data by querying arbitrary data stored in the label or, in this case, by simply using the LIDVID as an identifier. This ability to query arbitrary meta-data makes it very powerful.

 

To access the PDS4 product making use of the PDS API:

https://pds.nasa.gov/api/search/1/products/urn:esa:psa:em16_tgo_acs:data_raw:acs_raw_hk_nir_20180613t180000-20180613t235959::4.0

API responses include basic meta-data, and also give a direct access URL. From the given metadata output, there is the chance, from the PSA, to download both:

• the label file: https://archives.esac.esa.int/psa/repo/esa/psa/em16_tgo_acs/data_raw/2024-07-05/acs_raw_hk_nir_20180613t180000-20180613t235959/4.0/acs_raw_hk_nir_20180613T180000-20180613T235959__4_0.xml
• and the data file: https://archives.esac.esa.int/psa/repo/esa/psa/em16_tgo_acs/data_raw/2024-07-05/acs_raw_hk_nir_20180613t180000-20180613t235959/4.0/acs_raw_hk_nir_20180613T180000-20180613T235959__4_0.tab

 

Conclusion

PSA offers several mechanisms to retrieve data products given their identifier or other meta-data. The UI, TAP and SFTP offer access to both public and private data, whilst only released data are ingested into the PDS registry and API. A few basic examples are shown here, but much more complex queries and downloads are possible.