A Features Reconstruction and Prediction Joint Learning Framework with Incomplete SITS for Agriculture Semantic Segmentation

The dense Satellite Images Time Series (SITS) plays an important role in the agriculture semantic segmentation task. However, in real-world scenarios, cloud contamination and temporary sensor outages can lead to significant data missing in SITS, which declines the performance of models trained on ideal scenarios. A common approach is to reconstruct the complete SITS before the model’s prediction, where the reconstruction is independent of the prediction. This approach not only leads to the error accumulation from reconstruction to prediction, but also the detailed rebuilding of complete SITS may be redundant for the prediction. In this paper, we proposed a features reconstruction and prediction joint learning framework. The collaborative optimization of the two tasks aims to encourage the model to efficiently reconstruct complete features beneficial for prediction from incomplete SITS. Specifically, we simulate the data-missing scenarios with masks. The prediction task of masked data is supervised by labels. Meanwhile, by using the model that is well-trained on ideal scenarios as a teacher, we leverage its extracted temporal features from the data before masking as the target of the feature reconstruction task. The gradient flow of two tasks will be shared, which enables mutual supervision between them. Feature reconstruction prevents the model from acquiring incorrect reasoning ability caused by the shortest path problem during prediction, whereas prediction keeps reliability and reduces redundancy of reconstructed information. Furthermore, after training with the proposed framework, the model architecture remains unchanged and still maintains its robustness of complete SITS, which enhances the model's feasibility in practical applications. The experiments were conducted across multiple agricultural semantic segmentation datasets with incomplete SITS, sourced from Sentinel-2 and Planet satellites. We also validate its robustness for the common model architectures, and visualize the intermediate features to explore the mutual influence between the two tasks.