Least-squares-based formulation of deep learning: Theory and applications to geoscience data analytics

As a specific family of machine learning algorithms, deep learning (DL), successfully applied to several application areas is a relatively new and novel methodology receiving much attention. The DL has been widely applied to a series of problems including email filtering, image and speech recognition, and language processing, but is only beginning to impact on geoscience problems. On the other hand, the standard least-squares (SLS) theory of linear models has been widely used in many earth science areas. This theory connects the explanatory variables to the predicted ones, called observations, through a linear(ized) model in which the unknowns of this relation are estimated using the least squares method. The design matrix, containing the explanatory variables of a set of objects, is usually linearly related to the predicted variables. There are however applications that the predicted variables are unknown (nonlinear) functions of explanatory variables, and hence such a design matrix is not known a-priori. We present a methodology that formulates the deep learning problem in the least squares framework of the linear models. As a supervised method, a network is trained to construct an appropriate design matrix, an essential element of the linear model. The entries of this design matrix, as nonlinear functions of the explanatory variables, are trained in an iterative manner using the descent optimization methods. Such a design matrix allows to employ the existing knowledge on the least squares theory to the DL applications. A few examples are presented to demonstrate the theory.