In search of lithological truth – sceptical non-geologists in the non-English speaking world

The main objective of the study was to check how popular, free versions of AI chatbots cope with questions related to lithology. The assumption of the study was that a potential user is not a geologist, does not know how to formulate prompts correctly, and is sceptical enough about new technologies that they avoid logging in. Lithological issues may occur, for example, in descriptions of educational paths. The entire study was conducted in Polish. In order to shorten the study time, all prompts were formulated, and their order was imposed. The aim was, among other things, to see how the answer would differ depending on how precise the question was. In addition, the prompts were deliberately designed not to comply with the rules for asking questions, as we assumed that potential users would lack such knowledge. We asked people with geological knowledge to participate in the study so that they could assess its substantive value after receiving the results.

The rapid expansion of large language models (LLMs) into scientific workflows raises important questions concerning their reliability, transparency, and suitability for specialised disciplines such as the geosciences. This contribution presents the results of a survey-based assessment of selected AI-powered tools conducted in Polish between February and May 2025. The study involved 202 respondents, including professional geologists, academic staff, and students of geosciences, who evaluated AI-generated responses to seven tasks of varying complexity.

The study confirmed that the precise formulation of queries, especially those specifying source requirements and an expert-level perspective, substantially improves the quality of AI-generated content. This effect was particularly evident in questions involving linguistically ambiguous terms, where models often addressed only one interpretation while omitting alternative meanings relevant to geological sciences. Such omissions may result in incomplete or misleading answers when the user lacks sufficient domain knowledge to identify inaccuracies.

The opinions expressed in the Polish-language survey present an ambivalent picture. While the functional benefits and efficiency gains offered by AI tools are widely recognised, substantial methodological, substantive, and ethical limitations remain. The competence and awareness of the user have been identified as pivotal factors in determining whether the adoption of AI results in the creation of genuine value or the dissemination of errors and misinformation. The study emphasises the necessity for enhanced citation practices, the prioritisation of peer-reviewed literature, an augmentation in the number of high-quality non-English open geological publications, an enhancement in the semantic understanding of specialised terminology, and the development of regionally adapted language models. These measures are considered essential for ensuring transparent, reliable, and responsible use of AI in geoscientific research and communication.