AI Models Talk Directly, Skipping Text

A new technique called Cache-to-Cache (C2C) enables AI models to communicate directly via their internal representations, rather than relying on intermediate text. This approach, developed by researchers from Tsinghua University and other Chinese institutions, addresses information loss and inference latency inherent in text-based communication.

Traditional multi-LLM systems often use text tokens to pass information between models, leading to potential data compression issues and ambiguity. C2C bypasses these problems by leveraging the key-value (KV) cache, an internal data structure LLMs already generate.

Experiments show C2C boosts accuracy by 3.1 to 5.4 percentage points and significantly cuts down latency. For instance, one test saw a 14.41x speed improvement over text communication. This method allows models to share information without lengthy text generation or processing.

C2C is compatible with models of different architectures and specializations. The technique requires training only a cache-fusion module, not fine-tuning the main LLMs. However, its current implementation necessitates access to model internals, making it most suitable for teams managing their own inference stacks.

This advancement aligns with broader research efforts to explore more efficient inter-model communication methods beyond text. Other techniques, like Nvidia's KV-cache transfer and RecursiveMAS, also aim to reduce reliance on text tokens for faster and more token-efficient AI interactions.