AI boom threatens web's historical memory

The Internet Archive, responsible for preserving the web's history through its Wayback Machine, is confronting significant challenges exacerbated by the rise of artificial intelligence. The insatiable demand for computing power driven by AI has led to a drastic increase in the price of large hard disk drives, with costs tripling and availability plummeting.

These high-capacity drives are crucial for storing the Wayback Machine's ever-expanding collection of 210 petabytes of web page snapshots, which grow by approximately 100 terabytes daily. This storage crisis, costing the archive time and money, is compounded by another AI-related issue: news websites are increasingly blocking the archive's web crawler. This measure, intended to prevent AI companies from scraping content, inadvertently hinders the Wayback Machine's efforts to archive the live web.

While the Internet Archive is finding workarounds with donor support to mitigate the immediate impact of soaring hardware costs, the ongoing threat to data collection remains a serious concern. The organization is exploring resolutions with news outlets regarding the crawler blocking. For now, the Wayback Machine's ability to serve as a readily accessible, living archive of the internet faces unprecedented hurdles.