Idle GPUs Plague AI Firms, Prompting Compute Shifts

A recent survey indicates that 79% of companies running AI in production experience GPU idle times of 50% or more. This widespread underutilization is occurring even as many of these same organizations plan to acquire or switch AI compute platforms within the next 12 months. The data suggests a mismatch between readily available GPU capacity and the actual deployment of AI models, particularly frontier models accessed via APIs.

Enterprises are facing rising AI costs, partly due to securing GPU capacity in anticipation of future demand and partly due to the per-token costs of advanced models from providers like OpenAI and Anthropic. These models cannot be easily run on a company's own idle GPUs, necessitating a move towards open-weight models for cost savings, which presents its own challenges.

Companies are exploring various strategies to optimize AI spending, including measuring task costs, optimizing agent context, and shifting routine workloads to open models. The upcoming VentureBeat Agentic Infrastructure Stack Summit on October 28 in San Francisco will address these critical choices regarding agent coordination, context management, inference costs, and building internal AI infrastructure.

The survey data, collected between July and September 2026, reflects responses from 218 companies. The findings suggest that many organizations are grappling with the economics of AI, aiming to maximize the value derived from existing compute and token expenditures before committing to new contracts.