Evaluation of GPT-4o and GPT4o-mini for Climate Data Analysis with a novel tool-call software connecting different LLMs with an HPC

As artificial intelligence finds more and more applications within scientific contexts, the question on how to utilize it without sacrificing scientific integrity comes up naturally. In this context, FrevaGPT is a novel system that leverages LLMs such as GPT-4o and GPT-4o-mini to enable users to perform advanced analyses. It allows the loading and analysis of climate datasets by the LLM and moves the basis of truth to generated code, which can be checked by the user. Its backend was developed and deployed using modern software components (e.g. Rust, Python, Podman), focussing on correctness and reliability. The backend of FrevaGPT and its API is presented and the way it integrates into the larger Freva ecosystem as well as the role it plays in the improvements of ad-hoc analyses for climate data is discussed. Additionally, a suite of scientific prompts is explored to evaluate the capabilities of GPT-4o and GPT-4o-mini and how they compare in climate data analysis tasks. The prompts differ both in difficulty and complexity as well as in the requested output type: from a single number, to a graph, to a plot. This evaluation revealed that while both models demonstrated potential, GPT-4o outperformed GPT-4o-mini in handling more complex tasks involving diverse knowledge domains and programming requirements. GPT-4o-mini exhibited a higher tendency for errors and struggled with issues such as mismatched data dimensions, yet it remained a competitive, cost-effective alternative for simpler tasks. The findings highlight FrevaGPT as a significant step towards integrating advanced AI technologies into Earth sciences, bridging the gap between computational complexity and accessibility.