Advancing Crop Yield Predictions: The Potential of Diffusion Models in Machine Learning for Agriculture

The dual challenges of climate change and a growing population exceeding 9 billion by 2030 necessitate precise regional crop yield prediction models to optimize management, ensure food security, and guide agricultural decisions. Machine learning (ML), leveraging big data and high-performance computing, provides powerful tools for addressing these complexities but faces challenges such as inconsistent data quality and variable algorithm performance. While ML algorithms like Convolutional Neural Networks (CNNs), Random Forests (RF), and Long Short-Term Memory (LSTM) networks show promise in crop yield prediction, their performance can be hindered by data noise and incompleteness. Diffusion (a probabilistic generative model), with its iterative denoising capabilities, offers resilience to these issues and holds significant potential to improve accuracy and reliability in crop forecasting, though their use in this domain remains largely untapped.

This study compared XGBoost (XGB), a state-of-the-art tree-based ML model, with our proposed Diffusion-reg (DR) model. The input data for the models was compiled from multiple sources, including crop calendar data from MIRCA2000, net primary production (NPP) data from WAPOR, soil data from the Soil-Grids database, and maize crop yield data from the FAO database. Climate variables such as precipitation, air temperature, and solar radiation were obtained from ERA5, with all data aggregated into decadal periods. Additionally, Leaf Area Index (LAI) and Normalized Difference Vegetation Index (NDVI) data from MODIS were collected at 16-day intervals. In the subsequent step, maize yield data at the country level from the FAO was spatially disaggregated to produce pixel-scale estimates (250 m resolution, aligned with the soil input data resolution). This process focused exclusively on cropland areas within the five major maize-producing countries in Sub-Saharan Africa.

The evaluation of model performance metrics highlights the consistent superiority of the DR model over XGB across all analyzed countries. The R² values, which measure the proportion of variance explained by the models, indicate higher predictive accuracy for Diffusion-reg in every instance. For example, in Ethiopia, the DR achieves an almost perfect R² of 0.98 compared to XGB’s 0.95, while the largest gap is observed in South Africa, with R² values of 0.86 for DR and 0.76 for XGB. These results highlight the DR model’s ability to effectively capture complex data patterns, even in regions with higher predictive challenges.
Further, the RMSE (Root Mean Squared Error) and MAE (Mean Absolute Error) metrics reinforce the DR model’s superior predictive precision. Across all countries, DR consistently exhibits lower error values, with Ethiopia showing the best performance (RMSE: 0.02, MAE: 0.01). Although South Africa records the highest RMSE (0.25) and MAE (0.13) for the DR model, these metrics still significantly outperform those of XGB. Similar trends in Uganda and Mozambique, where the DR model achieves substantial reductions in error, further validate its robustness and reliability.
In summary, the DR model consistently outperforms XGBoost in diverse regional contexts, highlighting its potential for broader application in predictive tasks requiring high accuracy and resilience.