PyActiveStorage: Efficient distributed data analysis using Active Storage for HDF5/NetCDF4

Active storage (also known as computational storage) has been a concept often proposed but not often delivered. The idea is that there is a lot of under-utilised compute power in modern storage systems, and this could be utilised to carry out some parts of data analysis workflows. Such a facillity would reduce the cost of moving data, and make distributed data analysis much more efficient.

For storage to be able to handle compute, either an entire compute stack has to be migrated to the storage (with all the problems around security and dependencies) or the storage has to offer suitable compute interfaces. Here we take the second approach, borrowing the concept of providing system reduction operations in the MPI interface of HPC systems, to define and implement a reduction interface for the complex layout of HDF5 (and NetCDF4) data.

We demonstrate a near-production quality deployment of the technology (PyActiveStorage) fronting JASMIN object storage, and describe how we have built a POSIX prototype. The first provides compute “near” the storage, the second is truly “in” the storage. The performance with the object store is such that for some tasks distributed workflows based on reduction operations on HDF5 data can be competitive with local workflow speeds, a result which has significant implications for avoiding expensive copies of data and unnecessary data movement. As a byproduct of this work, we have also upgraded a pre-existing pure python HDF5 reader to support lazy access, which opens up threadsafe read operations on suitable HDF5 and NetCDF4 data.

To our knowledge, there has previously been no previous practical demonstration of active storage for scientific data held in HDF5 files. While we have developed this technology with application in distributed weather and climate workflows, we believe it will find utility in a wide range of scientific workflows.