Home / Technology / AI Scans All Books: Authors' Data Fears Grow
AI Scans All Books: Authors' Data Fears Grow
25 Aug
Summary
- AI models train on books, raising concerns among authors and publishers.
- Anthropic's 'Project Panama' aims to destructively scan all the world's books.
- Companies purchase books at scale for high-quality AI training data.
Many widely used artificial intelligence systems are trained on material sourced from books. To achieve necessary competence, large language models (LLMs) require substantial data. Companies have utilized various sources, including internet content and synthetic data, alongside printed literature.
Concerns arose from allegations of AI companies using books from pirate ebook sites for LLM training. In response, authors, writers, and publishers have voiced significant worry. The Authors Guild has issued comprehensive guidelines on AI licensing, addressing the legalities and potential harm to the book ecosystem from continued data harvesting.
More recently, the focus has shifted to companies that buy and scan books on an industrial scale. Court documents revealed Anthropic's 'Project Panama,' a scheme aiming to 'destructively scan all the books in the world.' This initiative was codenamed to avoid public knowledge.
Anthropic sought a large, high-quality dataset for training its AI model, Claude. Consequently, the company began purchasing books in industrial quantities due to the superior quality of writing found in books compared to online content.
