How often do large language models accept fake medical claims?

Large language models accepted made-up health claims approximately 32% of the time across various models tested in a recent study.

Do medical-focused AI models perform better than general ones with fake claims?

No, the study found that medical fine-tuned models consistently underperformed compared with general AI systems when presented with fake medical claims.

What influences an AI model's acceptance of a medical claim?

The study indicates that AI models are more influenced by how confidently a medical claim is written rather than its factual accuracy.

Home / Technology / AI Trusts Fake Health Claims: A Study Reveals Vulnerability

AI Trusts Fake Health Claims: A Study Reveals Vulnerability

10 Feb

•

Summary

Large language models accept fake medical claims 32% of the time.
Model performance varies, with weaker systems falling for claims more often.
Medical fine-tuned models underperform compared to general AI systems.

AI Trusts Fake Health Claims: A Study Reveals Vulnerability

Large language models (LLMs) demonstrate a significant vulnerability to medical misinformation, a new study published in The Lancet Digital Health reveals. These advanced AI systems can mistakenly repeat false health information when it is presented in realistic medical language. Researchers analyzed over a million prompts across 20 leading AI models, including those from OpenAI, Meta, and Google.

The study found that LLMs accepted made-up health claims approximately 32% of the time. However, the susceptibility varied greatly among models. Less advanced systems believed false claims over 60% of the time, while more robust models like ChatGPT-4o accepted them only about 10% of the time. Notably, AI models specifically fine-tuned for medical applications consistently underperformed compared to general-purpose models.

This research underscores that LLMs often prioritize the confident presentation of a medical claim over its factual accuracy. Examples of accepted misinformation include claims that Tylenol causes autism, rectal garlic boosts immunity, or tomatoes are as effective as blood thinners. The findings highlight an urgent need for robust safeguards within AI systems to verify medical claims before they are integrated into healthcare, ensuring patient safety and maintaining trust in AI applications.