Home / Technology / AI Distillation: China's Secret Weapon?
AI Distillation: China's Secret Weapon?
26 Oct
Summary
- Chinese AI labs distill American models' output for faster learning.
- US firms increasingly use less efficient reinforcement learning.
- Distillation helps China bridge AI gap, potentially doubling if stopped.
The rapid advancement of artificial intelligence sees Chinese laboratories increasingly employing a technique known as "distillation." This method involves training new AI models on the outputs generated by existing, more advanced models, primarily those developed in the United States. This allows Chinese AI systems to learn and improve at an accelerated pace, mimicking the skills and reasoning demonstrated by their American counterparts.
Meanwhile, leading American AI developers are shifting towards more computationally intensive methods like reinforcement learning (RL). This approach involves setting goals and allowing models to experiment to find solutions, consuming significant computing resources. As the internet's text data becomes largely exhausted for initial training, RL represents the frontier for American AI development.
Chinese firms can distill the capabilities that American models acquire through expensive RL training. This effectively acts as a shortcut, enabling Chinese AIs to learn faster. Researchers estimate that China's AI technology is currently about three to six months behind the leading American models. Without distillation, this gap could potentially double.
To counter this, U.S. companies like Anthropic and OpenAI are implementing measures to detect and impede distillation campaigns. These efforts include closing exploits that provide access to model outputs and increasing the cost of failed distillation attempts. These actions aim to protect intellectual property and mitigate national security risks.
However, the long-term implications for American labs of restricting access to their models remain uncertain. Some Chinese AI experts question the wisdom of relying on distillation, suggesting it might prevent them from truly surpassing frontier models and achieving autonomous AI development. ByteDance's founder, Zhang Yiming, has publicly stated the company will not use distillation, aiming for greater self-sufficiency despite potentially falling behind in the short term.
