AI Fails Clinical Reasoning Test, Doctors Essential

A new study indicates that generative artificial intelligence (AI) has not yet developed the necessary reasoning skills for safe application in clinical settings. While AI chatbots have improved their diagnostic accuracy with comprehensive patient information, they still failed to produce appropriate differential diagnoses in over 80% of cases. This gap highlights limitations in large language models' (LLMs) reasoning capabilities, which are central to clinical practice.

Researchers evaluated 21 LLMs, including leading versions like GPT and Gemini, using standardized clinical vignettes. The models performed well on final diagnoses when given complete data but struggled significantly with the initial, open-ended stages of a case and navigating uncertainty. This suggests AI cannot yet replicate the critical first step of differential diagnosis, which separates conditions with similar symptoms.

The findings underscore that even with improvements, current LLMs require close human supervision. Experts stress that while AI is a promising tool, human clinical judgment remains indispensable. The public is advised to use these technologies cautiously and always consult healthcare professionals for health concerns, reinforcing the need for a 'human in the loop' approach.