Elon Musk has joined the chorus of AI experts declaring that we’ve hit “peak data.” In a recent livestream, the xAI founder stated that AI models have essentially consumed all of humanity’s accumulated knowledge and there’s very little real-world data left to train AI models on.
“We’ve exhausted basically the cumulative sum of human knowledge…in AI training,” Musk said, adding, “That happened basically last year.”
Demis Hassabis, CEO of Google DeepMind, recently cautioned that the rapid advancements in AI chatbots may be slowing down. He attributed this potential slowdown to a growing scarcity of high-quality data for training AI models. Hassabis said that tech companies have essentially ‘exhausted’ the readily available data on the internet, having already used it to train existing large language models.
In fact, about a month ago, Microsoft AI chief Mustafa Suleyman talked about the use of synthetic data to train AI models because the available data is exhausted – something that OpenAI’s former chief scientist, Ilya Sutskever, warned about.
Synthetic data is the way to go, says Musk
Musk believes the solution lies in synthetic data – data generated by AI itself. He envisions AI models creating their own training data and engaging in a “process of self-learning.”
“The only way to supplement [real-world data] is with synthetic data, where the AI creates [training data],” he said.
“With synthetic data … [AI] will sort of grade itself and go through this process of self-learning,” he added.
This approach is already being adopted by major players like Microsoft, Meta, OpenAI, and Anthropic, who are using synthetic data to train their most advanced AI models.
Google DeepMind has reportedly been using synthetic data to train AL models and it relied on this method to help train a model that can solve Olympiad-level geometry problems.
In fact, the AI arm of Google recently pointed to inference-time compute – a solution to tackle the lack of data to train AI models. This technique allows AI models to tackle complex tasks by breaking them down into smaller, manageable prompts, essentially learning along the way.
“We’re generating vast amounts of synthetic data. That synthetic data is increasingly high quality,” Suleyman told The Verge in an interview last month.
