AI's Silent Knowledge Drain: Experts Vanish

For AI systems to continue advancing in knowledge work, they require either autonomous self-improvement or skilled human evaluators. The industry has heavily favored the former, largely ignoring the latter. This approach overlooks the limitations of AI in dynamic environments with ambiguous reward signals, unlike stable systems like game-playing AI. Knowledge work, with its ever-changing rules and delayed outcomes, necessitates human oversight to close the evaluation loop.

The automation of entry-level roles, which traditionally build expertise, creates a "formation problem." This could lead to knowledge fields atrophying not from external catastrophe but from incremental economic decisions. Fields like advanced mathematics or theoretical computer science risk losing their capacity for novel insight as the economic incentives and career paths disappear.

While methods like rubric-based evaluation and Constitutional AI reduce reliance on humans, they cannot fully capture nuanced judgment. Rubrics scale explicit knowledge but miss the deeper, intuitive sense of what is correct, which is gained through experience. Without active efforts to preserve human evaluation infrastructure, the underlying human capacity to validate and extend AI expertise could disappear, even as AI outputs continue to perform well on benchmarks.