AI's Reliability Crisis: Fundamental Design Flaw

Recent research indicates that large language models (LLMs), the technology behind AI tools like ChatGPT, suffer from a significant hallucination problem. This means they frequently generate inaccurate or fabricated information.

The issue is exacerbated by the amount of data fed into the models. Experiments showed that as input text length increased from 32,000 to 128,000 words, hallucination rates rose substantially. For some models with 200,000 words of input, error rates were extremely high.

This tendency to hallucinate is not a flaw that can be easily fixed. Researchers suggest it originates from the fundamental design of LLMs, which are trained to predict the next word in a sequence, prioritizing fluency over factual accuracy.

The implications are substantial for businesses investing heavily in AI. While "about right" answers might suffice for everyday tasks, they pose considerable legal and financial risks in high-stakes fields such as accounting and law. Past investigations have shown LLMs making errors on tax forms serious enough to constitute evasion.

While the industry is exploring workarounds, fundamentally new approaches may be years away. This reality creates a challenging landscape for AI companies, potentially limiting the most lucrative applications of LLMs while facing competition from lower-cost alternatives.