Transformers for Air Quality: Enhancing PM2.5 Modelling with Deep Attention Mechanisms

In recent years, sustainability has become a global priority, making the mitigation of air pollution—particularly hazardous Particulate Matter (PM)—a paramount societal task. Leveraging air quality data from the Taiwan EPA, this study employs a fine-tuned, pre-trained transformer model to capture the complex, non-linear relationships between various pollutants and PM concentrations. Our results demonstrate that this approach significantly outperforms traditional ANN benchmarks in one-day-ahead predictions. Furthermore, we validate the model’s practical applicability by evaluating its performance under conditions of spatial variability and extreme events. From a statistical and stochastic perspective, the proposed framework can be interpreted as a data-driven approximation of latent stochastic dynamical systems governing pollutant transport and dispersion. This enables probabilistic characterization of forecast uncertainty, tail risks, and rare extreme pollution events, which are critical for risk-sensitive urban environmental governance.

The study offers two main contributions. It applies a large-scale transformer model to capture complex temporal patterns and achieve markedly better PM forecasts than traditional baselines. It also demonstrates strong generalizability through evaluation across varied environmental contexts in Taiwan. The work supports UN SDGs 3, 11,  and 13 by enabling more sustainable urban management, improving public health protection, and strengthening climate resilience, thereby linking advanced AI forecasting with sustainability policy.