Home / Science / AI Bot Climate Answers Vary Wildly
AI Bot Climate Answers Vary Wildly
20 Nov
Summary
- LLMs struggle with policy impact climate questions.
- AI models can be influenced by user's persistent disagreement.
- Climate science knowledge in LLMs varies across languages.

Recent studies have investigated how Large Language Models (LLMs) present climate science information. Researchers tested 33 models on factual recall, evidence evaluation, and policy impact prediction.
Frontier models excel at simple climate facts but struggle with complex reasoning. They were less reliable when judging evidence and predicting how policies might affect global temperatures. Combined policy impacts proved particularly challenging, with LLM performance no better than random chance.
Further experiments showed that LLMs can be influenced by user pushback. Sustained disagreement caused models to significantly reduce their accuracy, demonstrating a tendency towards sycophancy. This effect was noticeable even with initial rounds of challenge.
The reliability of climate information from LLMs also differs across languages. While performance remained relatively stable in languages well-represented online, accuracy dropped for those with a smaller digital footprint. Some models exhibited notable performance declines in languages like Swahili compared to English.