Home / Technology / AI Factories: Underutilized GPUs Spark New Infrastructure Race
AI Factories: Underutilized GPUs Spark New Infrastructure Race
23 Oct
Summary
- Most companies use 50% or less of their GPU capacity.
- 71% of these plan new AI infrastructure within 12 months.
- Integration and security are becoming key infrastructure decisions.
A recent survey reveals that most companies operating AI in production are utilizing 50% or less of their GPU capacity. Despite this underutilization, a substantial 71% of these firms plan to adopt new or replacement AI infrastructure within the next 12 months. This suggests a strategic shift, with buyers prioritizing improved scheduling, integration, and economics over raw compute alone.
Industry leaders are urging a move towards 'AI factories' for producing AI, expanding beyond compute to include elements like governed APIs and secure agent environments. Integration with existing cloud and data stacks, along with robust security measures, are emerging as paramount considerations in infrastructure choices. The efficiency and cost-effectiveness of AI deployments are increasingly being measured by metrics like cost per successfully completed task.
"Memory" in AI infrastructure now encompasses institutional knowledge, agent task state, and inference caching, requiring careful architectural decisions. The survey indicates a growing interest in cost per million tokens as a buying factor, signaling a greater emphasis on the economic viability of AI operations. Leaders and builders need a shared metric, such as cost per task, to align technical decisions with economic goals and identify performance bottlenecks before investing in new infrastructure.
