AI Cost Breakthrough: Overtrain Compact Models!

A recent study introduces Train-to-Test (T) scaling laws, a novel framework designed to optimize the development of large language models (LLMs) by considering both training and inference costs. Traditional guidelines often prioritize training expenses, leading to inefficient real-world applications that struggle with high inference costs.

The new T scaling laws jointly optimize a model's parameter size, its training data volume, and the number of inference samples used during deployment. This research demonstrates that it is more compute-optimal to train substantially smaller models on vastly more data than previously prescribed.

Experiments involving over 100 language models confirmed that highly overtrained compact models consistently outperformed larger models when test-time sampling costs were factored in. This shift from standard scaling laws, like the Chinchilla rule, offers a compute-optimal frontier.

This framework is particularly beneficial for reasoning-heavy applications such as coding, where generating multiple samples is common. While extreme overtraining may present challenges in fine-tuning, the optimal strategy remains skewed towards compact models. The researchers plan to open-source their findings to empower enterprises.