Toward a Climate-Aware Large Language Model: A Comparative Study of Methodologies for Source-Grounded Large Language Models

LLMs can effectively simplify complex textual information, yet their application in scientific domains, particularly climate science, remains limited. Climate research relies on dense, technical documents such as assessment reports that are difficult to navigate for non-specialists and time-constrained experts. We have explored the development of a climate-aware LLM that enhances access to such materials by balancing conversational fluency with strict grounding in trustworthy geoscientific sources. In this research, we are studying the different methodologies to develop a climate-aware LLM, to create a model that bridges the gap between complex reports of experts and information. This climate-aware LLM is also envisioned as a foundational component for future, more advanced AI developments in the climate domain.

A major contribution of this work is the development of a curated, large-scale synthetic dataset designed to bridge the gap between LLMs and Earth science. We created a dataset by collecting and preprocessing a vast corpus of Copernicus publications and the Intergovernmental Panel on Climate Change (IPCC) reports, which served as the foundation for generating high-quality Question-Answering pairs. By employing various prompt engineering techniques, we ensured the data covers a wide range of Earth science topics and includes diverse question categories, such as open-ended, closed-ended, and freeform queries, among others. To ensure the practical utility of the model, we also implemented optimizations to reduce generation latency for real-world applications.

Moreover, we systematically evaluate multiple architectural approaches, including retrieval-augmented generation (RAG), retrieval-augmented fine-tuning (RAFT), and full fine-tuning, using a combination of standard semantic and lexical evaluation metrics, domain-specific climate benchmarks such as the ClimaQA Benchmark, and LLM-as-a-judge evaluations to compare model outputs.