Testing cloud-optimized formats for future data archival & distribution

The seismological community is producing ever more datasets, and datasets themselves increase in size and complexity. To support the community in FAIRly archiving this data, seismological data centers must find solutions for the storage and distribution of such large and diverse data. Different aspects related to the FAIRness of these datasets are being considered, among others during the GeoInquire project: appropriate metadata to describe experiments, access control for both data and metadata, and improved data formats for raw data. For the first two topics, two proposals are currently under review by FDSN Working Groups, and a positive decision is expected in the coming months. Access control is particularly important for distributed acoustic sensing (DAS) data, given the sensitivity of the detailed cable location information.

Concerning improved raw data storage, one possible way forward is to combine cloud storage, already in place at several data centers, with asynchronous services that provide direct links to data. Ideally, this will allow users to load specific segments of data from the cloud storage using high-level languages like Python and massively parallelize their processing. In such a setting, analysis-ready, cloud-optimized data might provide advantages over traditional miniSEED archives; previous studies also suggest advantages over the HDF5 formats commonly output by distributed acoustic sensing (DAS) devices and currently used to store the majority of DAS data.

In this contribution, we report on ongoing collaborative work to systematically evaluate cloud-optimized formats on commercial and on-premise (university or institute) cloud storage services to evaluate their usability for archival, distribution and analysis-ready access to large datasets. We tested I/O performance and storage aspects in Zarr, tileDB and Apache Iceberg on AWS and self-hosted S3 buckets. We will report on test results and a first scientific use case that utilizes data on an on-premise cloud. We will also compare challenges and opportunities of these storage solutions for DAS and for large-N nodal data.

The zarr format is already used in the Earth Science community and, combined with rich metadata and the xarray library, turns out to provide very user-friendly access and data slicing for DAS data. The TileDB format provides similarly good access and slicing, but is less well known in the Earth Science community and requires careful engineering of data ingestion and maintenance. With this presentation, we aim to provide updates on the ongoing collaboration, show first usage examples for scientific workflows, and to stimulate discussion about future seismological data archives.