Polytope: Feature Extraction for Improved Access to Petabyte-Scale Datacubes

ECMWF currently produces about 120 TiB of raw weather data from its real-time forecasts every day. With model improvements and higher resolution forecasting however, this raw data is expected to grow to over a petabyte per day over the next few years. Whilst these improvements will in theory help scientists better forecast weather events, distributing such vast amounts of data efficiently will prove to be increasingly difficult with the current data access mechanisms. 

To tackle this challenge, ECMWF is developing a novel feature extraction concept, named “Polytope”. By leveraging tools in the field of higher dimensional computational geometry, Polytope will be able to efficiently cut a wide range of intricate n-dimensional shapes (polytopes) from ECMWF’s high-dimension (6D/7D) weather datacubes. Polytope can be used to perform server-side feature extraction, providing significant data reductions before delivering the data to the user. This not only improves the efficiency of access to petabyte-scale datacubes, but also removes significant post-processing complexity from the user leading to an overall data usability improvement. Practical examples include a user requesting weather data over a 4-dimensional flight path, which crosses three spatial axes as well as a temporal axis. In this example, instead of providing data over the entire bounding box of the path, Polytope will only return the few precise bytes of interest to the user. 

This work is an important contribution to, and is funded by, the EU’s Destination Earth initiative. Within Destination Earth, Polytope will enable efficient access to petabyte-scale datacubes generated by very high-resolution digital twins. This presentation will introduce the Polytope concept and demonstrate its usage for different types of feature extraction.