Compression and Reconstruction of High-Dimensional Weather Simulation Data Using Tensor Decompositions

High-resolution numerical weather and climate simulations increasingly produce very large data with high dimensionality. Such datasets usually span three spatial dimensions, time, multiple physical variables, and ensemble members, leading to six-dimensional (6D) hypervolume datasets. Being grid-based, these datasets can be interpreted as 6D data tensors. The storage, processing, visualization, and analysis of such large data poses significant computational and memory storage challenges. Tensor decomposition and approximation methods have proven to be an efficient tool for compression and reconstruction of such large, high-dimensional scientific datasets. Using rigorous mathematical principles, tensor decompositions are exploiting multi-linear structure and redundancy inherent in scientific data, leading to an effective compression of the datasets while providing visually accurate results.

In this work, we investigate the applicability of tensor decompositions for the compression and efficient representation of 6D weather simulation data. We focus on two of the state-of-the-art low-rank tensor formats, tensor-train (TT) and Tucker decompositions. These methods generalize the singular value decomposition (SVD) to higher-order tensors, enabling compression of spatial, temporal, and physical modes through rank reduction. Therefore, the large high-dimensional tensor is factorized into multiple smaller, rank-reduced tensors with lower dimensionality, reducing the size of the original data significantly while preserving essential features. Such a reduced representation is also called a tensor approximation (TA).

We apply the tensor decompositions to a real-world weather simulation dataset from the Alpine region of Switzerland (COSMO-1E), organized along longitude, latitude, vertical level, time, physical variables (such as temperature), and 11 ensemble dimensions. We evaluate the performance of the compression in terms of storage reduction, relative reconstruction error, peak-signal-to-noise-ratio (PSNR), structural similarity index measure (SSIM), computational costs, and visual comparison to the original data. Our results demonstrate significant compression ratios while preserving high visual accuracy. For example, a TT-based compression with a compression ratio of 1 : 900 provides results with a relative error of only 0.0005. The obtained compression ratio reduces the size of 4GB of the original dataset to 4.6MB for the compressed dataset. Lower compression ratios lead to even higher accuracy.

Beyond efficient data compression, the linear structure of the tensor decompositions allows for efficient application of filters in the tensor domain. The computation of the mean, standard deviation or similar linear operations along user-defined dimensions can directly be performed on the decomposed tensors, without ever having to reconstruct the large 6D dataset. Furthermore, the structure of the tensors allows for efficient partial reconstruction and visualization of slices or subsets of the dataset without reconstructing the complete dataset.

Overall, this work highlights tensor decompositions as powerful tool for managing the growing size and complexity of high-dimensional weather simulation data. Their linear structure, which allows for efficient filter application in the compressed domain, makes them especially suitable for scientific analysis of complex datasets. Their integration into geoscientific data pipelines offers a promising pathway towards scalable and accurate data compression and analysis in numerical weather prediction and climate science.