Too Big to Handle? Hexagonizing LIDAR and Satellite Data in Geoscience Applications

The exponential growth of LIDAR and satellite data in geoscience presents both opportunities and challenges for users. Traditional data handling methods often struggle with the sheer volume and complexity of these datasets, hindering easy accessibility, efficient analysis and decision-making processes. This presentation introduces the Scandinavian Highlands HEX-Responder platform and database structure, a cutting-edge solution that leverages the power of hexagonal discrete global grid system, Uber H3, and developed processes to revolutionize geospatial data management, fast responsive visualization and analysis.

We will showcase real-world applications, highlighting the platform's potential to accelerate scientific discovery and improve decision-making processes using satellite and remote sensing data.

The platform’s approach offers several advantages over conventional methods:

• Efficient data organization and retrieval
• Improved advanced spatial data analyses opportunities
• Seamless integration of multi-scale and multi-dimensional data without losing information
• Enhanced, responsive and fast visualisation capabilities

Our ELT (extract, load, transform) and subsequent visualisation procedure can be applied to any big raster data formats. First, the raw raster data is transformed into optimised parquet files through chunked reading and compression based on a low-resolution H3 hexagon cell index (hexagonization), enabling rapid data import to a column-oriented database management system for big data storage, processing and analytics. The H3 cell organisation is preserved in the database through partitioned fetching for visualisation on the platform. This method allows for horizontal scaling and accurate multi-resolution aggregation, preserving data integrity across scales and significantly overcomes typical computational memory limitations.

The platform's capabilities are exemplified by its approach to LIDAR and satellite emissivity data processing using the H3 grid. High-resolution LIDAR data is efficiently gridded and visualized to H3 resolution level 15 hexagons (0.9m² hexagon cells). The gridding preserves all original pixel raster points while providing aggregated views for seamless zooming.

Another prime example of the capabilities is the handling of NASA’s ASTER Global Emissivity Data (100m resolution). Here, our pipeline transformed 2.1 terabytes of extracted raw CSV-data derived from NASA’s emissivity data into a compressed format based on the H3 index occupying only 593 gigabytes in the database.

This approach not only saves data storage space but also dramatically improves data accessibility and processing speed for the users, allowing users to work in a responsive environment with this massive dataset in ways previously not possible. Each hexagon represents an opportunity to store unlimited amount, types and categories of pre-processed data for more integrative analyses and data insight.

By hexagonizing LIDAR and satellite data, the HEX-Responder platform enables users to explore massive datasets with ease and efficiency in a responsive environment. The integrated procedures allow for detailed information maintenance and retrieval, paving the way for advanced predictive modelling in geoscience applications using earth observation data in a new way.