CloudTracker Observatory: analysing a large number of aerosol-polluted cloud tracks in the AWS compute cloud

One of the biggest challenges in working with satellite data is the vast volume of data. It makes downloading larger chunks slow, and keeping a local copy for infrequent analysis is often impractical. This is a well-known issue, and several institutions are creating cloud-based solutions. In addition to moving data to the cloud, new file formats and processing tools are emerging. However, there are data which are stored in the cloud in non-cloud-friendly file formats. For example, MODIS cloud optical properties are stored in HDF4 file format in the AWS cloud, but effective software tools for processing such data in the compute cloud are limited.

In this presentation, we discuss planned workflows within CloudTracker Observatory for efficient processing of MODIS data in HDF4 format in the AWS cloud. We use detection and analysis of ship-track-like aerosol-polluted cloud tracks (Toll et al 2019 Nature https://doi.org/10.1038/s41586-019-1423-9) as the main use case. We study both strong visible tracks and weak tracks invisible to the naked eye in the satellite images. We analyse existing software tools and how they could be improved, together with available architectural options in the AWS compute cloud. The Observatory is planned within an ERC-funded project CloudTracker - Tracking Polluted Clouds: the Plausibility of a Strong Aerosol Cooling Effect on Earth’s Climate. Shared cloud-based workflows close to the used satellite data that can be easily extended by any interested research group are likely to foster international collaboration.