Effective Data Compression for Geosciences: An open-source solution to combat data storage challenges

Data storage is a critical challenge in science as the amount of data being generated continues to increase. Geosciences and weather are not an exception. To address this challenge, data reduction techniques are required. Even though lossless compression methods might sound ideal, they don't usually work that well when it comes to compressing geoscience data because this data often has a lot of uncertainty resulting in random bits, which makes it hard to compress. Lossy compression methods can do a better job compressing geoscience data by getting rid of the insignificant details that make it hard to compress. Hopefully, research groups have been working on this problem for years and have published open source tools that can provide high compression ratios with acceptable error levels. However, these methods have not yet been widely adopted in geosciences due to concerns about the loss of information that they entail. By combining these best open source lossy compressors into a single easy-to-use package and showing its effectiveness through its use in several weather applications, we have created a powerful and user-friendly open-source tool that effectively helps reduce data storage needs while preserving scientific conclusions.