Open-source large language models in astronomical data classification: applications and benchmarking

Recent advances in large language models (LLMs) have opened new possibilities for astronomical data analysis and classification tasks. While multimodal models such as GPT-4o/4.1, Claude 3.5/3.7, and others have demonstrated remarkable capabilities in processing both text and images, their application to astronomy has been limited by substantial operational costs. This work presents a comprehensive evaluation of open-source LLMs, including LLaMA 3.2, Gemma, and DeepSeek, for astronomical data classification tasks, with particular emphasis on mean-motion resonance identification in asteroid dynamics.

In this research, it is demonstrated that open-source models can achieve performance acceptable to the given problem and outperform traditional neural network while significantly reducing operational costs. This approach leverages the inherent pattern recognition capabilities of LLMs to analyze time series data and astronomical images, tasks that traditionally require specialized algorithms and extensive computational resources. Building upon previous work that established the viability of multimodal LLMs for resonance identification, it is shown that careful prompt engineering and model instructing and fine-tuning can yield acceptable accuracy rates even with freely available models that can be launched on a researcher's laptop.

Furthermore, a structured framework for developing standardized benchmarks for astronomical tasks using LLMs is introduced. This framework includes: (1) systematic dataset curation protocols, (2) evaluation metrics adjusted to some astronomical applications, (3) cross-model performance comparison methodologies, and (4) guidelines for prompt engineering technics. These benchmarks enable reproducible performance assessment across different LLM architectures and can facilitate the identification of cost-effective solutions for specific astronomical problems.