SSD Storage Boosts AI Memory Beyond GPU Limits

GenStorAIGE has unveiled its AI90 inference acceleration platform, taking a novel storage-centric approach to enhance AI memory capacity. This system integrates PCIe Gen5 solid-state drives directly into the memory hierarchy, allowing parts of the Key-Value Cache used by large language models to reside outside of GPU memory.

The AI90 features a three-tier memory architecture combining HBM, system DRAM, and SSDs. By transparently offloading KV Cache data, the platform alleviates pressure on GPU memory, supporting larger workloads and extended context windows. GenStorAIGE claims this architecture can reduce first-token latency from seconds to sub-second response times.

This new architecture also offers substantial performance gains, including up to a 5.1x increase in throughput and a 39% reduction in GPU memory usage. With intelligent peer-to-peer GPU communication, the AI90 can accelerate inference by up to 5.8x on systems with eight Nvidia RTX 5090 cards, effectively mimicking a much larger cluster.

Supporting context windows exceeding 128,000 tokens, the AI90 enables processing of larger documents and more complex conversations. To manage the intensive write demands of constant cache updates, GenStorAIGE developed the PT200Z AI SSD, boasting 14.8 GB/s sequential read speeds and exceptionally low read and write latencies.

This development reflects a growing trend in AI infrastructure towards memory tiering as LLMs increasingly surpass the capabilities of GPU HBM alone. Integrating fast SSD storage provides a scalable method for increasing context length without requiring more GPUs or larger HBM configurations, though independent benchmarking is needed to verify these performance claims.