Histogram compression of large ensemble forecasts

1000 member ensemble forecasts of rainfall are compressed from ~230 MB to ~400 KB using lossy histogram compression. This level of compression allows fast download, analysis and responsive display on a website, even when using obsolete laptop computers or basic smartphones. The information lost to achieve this level of compression is ignored in all but the most specialist of applications, and the algorithm scales to much higher ensemble sizes with negligible additional storage. The method is currently in operation every day with national meteorological centres in East Africa.

 

Physics based weather models are routinely used produce ensemble forecasts with up to 100 members. These ensembles are an advance on single deterministic forecasts, in that they indicate uncertainty. With larger ensembles providing more accurate distributions of forecast variables. The downside of large ensembles is their storage, transmission and processing cost. Furthermore, machine learning models are being used operationally to generate very large forecast ensembles. For example, rainfall forecasts by ICPAC and national meteorology centres in East Africa are now routinely produced with 1000 ensemble members. Analysis and transmission of these forecasts using traditional methods is completely impractical given currently available hardware. Compression is necessary and can be achieved by storing the ensemble as a series of histograms, sacrificing spatial correlation information.