NetCDF: Performance and Storage Optimization of Meteorological Data

Any software relies on data, and the meteorological field is not an exception. The importance of using correct and accurate data is as important as using it efficiently. GRIB and NetCDF are the most popular file formats used in Meteorology, being able to store exactly the same data in any of them. However, they differ in how they internally treat the data, and transforming from GRIB (a simpler file format) to NetCDF is not enough to ensure the best efficiency for final applications.

In this study, we improved the performance and storage of ARPEGE cloud cover forecasts post-processing with convolutional neural network and Precipitation Nowcasting using Deep Neural Network projects (proposed in other sessions for the EGU general assembly). The data treatments of both projects were studied and different NetCDF capabilities were applied in order to obtain significantly faster execution times (up to 60 times faster) and more efficient space usage.