AI Automates Science While You Sleep

Andrej Karpathy has released 'autoresearch,' an open-source project aiming to automate the scientific method using AI agents. This system functions as an autonomous optimization loop, where an AI agent modifies code based on hypotheses, runs experiments, and retains changes that improve performance. In one overnight run, Karpathy's agent reduced a key metric from 0.9979 to 0.9697.

Over two days, an agent tuned a model, making approximately 700 autonomous changes. It identified about 20 additive improvements, transferring to larger models and yielding an 11% efficiency gain on the 'Time to GPT-2' metric. Karpathy noted the agent uncovered oversights in attention scaling and regularization he had missed over two decades. This approach transforms machine learning into an evolutionary process.

The project's impact quickly spread across X, with over 8.6 million views. Varun Mathur at Hyperspace AI distributed the loop across a peer-to-peer network, enabling 35 agents to run 333 experiments unsupervised. These agents rediscovered ML milestones like RMSNorm in just 17 hours, a feat that took human researchers years.

Eric Siu applied autoresearch to marketing, predicting that future teams will run 36,500+ experiments annually instead of 30. The agent modifies marketing assets like landing pages or ad creatives, deploys them, measures results, and retains successful variations. This process builds a proprietary map of audience resonance, with faster experiment loops determining market winners.

Discussions emerged regarding potential issues like 'over-optimization' of validation sets. However, Karpathy emphasized the focus on optimizing performance per compute. Other users reported successes, such as an agent improving by simplifying the model. The future of research, as suggested by autoresearch, involves humans shifting from experimenters to experimental designers, with curiosity becoming the primary bottleneck.