eo-grow - Earth Observation framework for scaled-up processing in Python

Analysing EO data is a complex process, and solutions often require custom tailored algorithms. On top of that, in the EO domain most problems come with an additional challenge: How can the solution be applied on a large scale?

Within the H2020 project Global Earth Monitor (GEM) we have updated and extended eo-learn with additional functionalities that allow for new approaches to scalable and cost-effective Earth Observation data processing. We have tied it with the Sentinel Hub’s unified main data interface (Process API), the Data Cube processing engine for constructing analysis-ready adjustable data cubes using Batch Process API, and, finally, the Statistical API and Batch Statistical API to streamline access to spatio-temporally aggregated satellite data.

As part of GEM processing framework, we have built eo-grow which facilitates extraction of valuable information from satellite imagery. eo-grow tackles the issues of scalability by enabling coordination of clusters to run the EO workflows over large areas using  Ray. At the same time the framework provides reproducibility and traceability of the experiments using schemed input configurations and their validation.

In eo-grow a workflow based solution is wrapped into a pipeline object, which takes care of parametrization, logging, storage, multi-processing, data management and more. The pipeline object is configured via a well-defined schema allowing straightforward experimentation and scaling up: going to larger area of interest, running on different time interval, or tweak of any other pipeline parameter becomes just a matter of updating (json) configuration, which additionally serve as record of the experiment.

eo-grow library has been publicly released on GitHub: https://github.com/sentinel-hub/eo-grow. The documentation available in the repository provides the overview of the eo-grow general structure, its core objects, and instructions on installation and using eo-grow with command line interface. Additional repository, https://github.com/sentinel-hub/eo-grow-examples showcases eo-grow on a few use-cases.

In the presentation we will introduce the framework and showcase its usability on concrete examples. We will illustrate how eo-grow is used in large-scale research experiments, explain its role in reproducibility and show how the no-code approach and code reuse facilitate the productionalization of the workflows.