Ground-level land surface classification and thermal analysis using foundation models

Foundation models have shown substantial potential in enhancing Earth observation by providing high accuracy while minimizing the need for manual annotation. However, the combined application of multiple foundation models for processing ground-truthing data remains largely underexplored. This study introduces a new approach for ground-level land use classification and ground-truthing by integrating a vision foundation model, the Segment Anything Model (SAM), with a general-purpose large language model (GPT-4o). Using high-resolution thermal and RGB imagery captured from human-eye height with a handheld camera, the proposed method generates object-level land use classifications and surface temperature profiles. Data collection was conducted in the Lusatia region of Germany, covering diverse land use types. SAM was utilized to segment complex landscape structures into meaningful elements such as roads, water bodies, and trees, followed by GPT-4o, which classified these segments into custom-defined land use categories. At a broad level (7 classification types), the workflow achieved approximately 80% accuracy, with high F1 scores for categories such as Road (0.89), Vegetation (0.82), and Built Structure (0.81). At a finer level (28 classification types), the method attained around 64% accuracy, effectively classifying detailed sub-classes such as Asphalt-Concrete Road (F1 = 0.85), Brick Road (F1 = 0.86), Tree (F1 = 0.74), and Arable Land (F1 = 0.68). By overlaying thermal imagery with classified segments, the method revealed distinct microclimatic patterns across land use types, with agricultural land showing the lowest surface temperatures (p < 0.001). The proposed workflow underscores the potential of combining SAM and GPT-4o to deliver robust ground-truthing data using portable cameras, advancing AI-enabled environmental monitoring.