Atmospheric Retrievals in a Modern Python Framework

Modern Machine Learning (ML) techniques applied in atmospherical modeling rely heavily on two
aspects: good quality and good coverage observations. Among others, Satellite
Radiometer (SR) measuremens (Radiances or Brightness Temperatures) offer an excellent trade off
between such aspects; moreover SR observations have been providing quite stable Fundamental Cli-
mate Data Records (FCDR) for years and are expected to continue to do so in the following decades.
This work presents a framework for SR retrievals that uses modern ML standard packages from
the Scipy and Pangeo ecosystems; moreover, our retrieval scheme leverage the powerful
capabilites provided by NWPSAF’s RTTOV and its Python wrapper.
In terms of retrievals we stand on the shoulders of Bayesian Estimation by using Optimal Estima-
tion (OE), popularized by Rodgers for 1D atmospherical retrievals; we use pyOpEst
which is an open source package developed by Maahn. PyOptimalEstimation is structured
following an Object Oriented design, which makes it portable and highly maintainable.

The contribution presented here ranges from the scientific software design aspects, algorithmic
choices, open source contributions, processing speed and scalability; furthermore, simple but effi-
cient techniques such as cross-validation were used to evaluate different metrics; for initial test-
ing we have used NWPSAF’s model data and observation error covariances from SR literature.

The open source and community development philosophy are two pillars of this work. Open source
allows a transparent, concurrent and continuous development while community development brings
together domain experts, software developers and scientists in general; these two ideas allow us to
both profit from already developed and well supported tools (e.g. Scipy and Pangeo) and contribute
for others whose applications might benefit. This methodology has been successfully used all over the
Data Science and ML universe and we believe that the Earth Observation (EO) community would highly benefit in terms of streamlining development and benchmarking of new solutions. Practical examples of success can be found in the Pytroll community.

Our work in progress is directly linked to present and near future requirements by Earth Observa-
tion, in particular the incoming SR streams of data (for operational purposes) is increasing fast
and by orders of magnitude. Missions like the EUMETSAT Polar System-Second Generation (EPS-
SG, 2023) or the Copernicus Microwave Imager Radiometer (CIMR, 2026) will require scalability
and flexibility from the tools to digest such flows of data. We will discuss and show how operational
tools can take advantage of the enormous community based developments and standards and become
game changers for EO.