Adapting Transfer Learning for Multiple Channels in Satellite Data Applications

Transfer learning is a technique wherein information learned by previously trained models is applied to new learning tasks. Typically, weights learned by a network pretrained on other datasets are copied or transferred to new networks. These new networks, or downstream models, are then are then used for assorted tasks. Foundation models extend this concept by training models on large datasets. Such models gain a contextual understanding which can then be used to improve performance of downstream tasks in different domains. Common examples include GPT-3 in the field on natural language processing and ImageNet trained models in the field of computer vision.

Beyond its high rate of data collection, satellite data also has a wide range of meaningful applications including climate impact modelling and sustainable energy. This makes foundation models trained on satellite data very beneficial as they would reduce the time, data, and computational resources required to obtain useful downstream models for these applications.

However, satellite data models differ from typical computer vision models in a crucial way. Because several types of satellite data exist, each with its own benefits, a typical use case for satellite data involves combining multiple data inputs in configurations that are not readily apparent during pretraining of the foundation model. Essentially, this means that the downstream application may have a different number of input channels from the pretrained model, which raises the question of how to successfully transfer information learned by the pretrained model to the downstream application.

This research proposes and examines several architectures for the downstream model that allow for pretrained weights to be incorporated when a different number of input channels is required. For evaluation, models pretrained with self-supervised learning on precipitation data are applied to a downstream model which conducts temporal interpolation of precipitation data and requires two inputs. The effect of including perceptual loss to enhance model performance is also evaluated. These findings can be used to guide adaptation for applications ranging from flood modeling, land use detection, and more.