SC2.19 | From Data to Cube: Building Earth Science Data Cubes with Firecube
From Data to Cube: Building Earth Science Data Cubes with Firecube
Convener: Anna-Lena ErdmannECSECS | Co-convener: Armagan Karatosun

Analysis-ready data cubes make it easier to explore, share, and reuse large volumes of Earth Observation data. But there are challenges with data cubes: How to make your data cubes reproducible and discoverable? How to scale up processing for multi-TB data cubes? How to know which parts of the cubes were written or not? Firecube is an open-source Python framework built at EUMETSAT to address these challenges of creating and maintaining large data cubes.

This session introduces Firecube and gives participants a hands-on tutorial on building zarr data cubes from Earth Observation data in two parts. The tutorial will be guided by the Firecube developers and will include interactive parts where participants are invited to solve some challenges together.

The first part of the session introduces the Firecube plugin framework. Firecube plugins are the recipe that makes a cube reproducible for anyone and are the heart of the software that knows about your dataset. As Firecube plugins are flexible by design, Firecube can be used by any domain across Earth Sciences and beyond. For the short course, we will take an example dataset from the satellite-based Earth Observation domain to create a plugin for a netCDF dataset, introducing functions such as reprojection, subsetting, and regridding.

The second part of the tutorial puts the created plugin into practice. Using a satellite dataset, we will explore Firecube's capabilities to parallelize the reading and writing of files, manage written chunks with Firecube's chunk manager, and create a catalogue to make the cube discoverable.

The session will be hands-on, with opportunities for participants to discuss how Firecube could be applied to their own datasets and workflows. Basic familiarity with Python, xarray, and zarr is recommended. Participants will also have the opportunity to provide feedback on Firecube and contribute ideas for its future development.