Reasoning models, heralded as the next major leap in the evolution of large language models (LLMs), have demonstrated remarkable advancements, particularly in domains demanding intricate problem-solving, such as mathematics and computer programming. These sophisticated systems, distinguished by an additional "reasoning training" phase, leverage reinforcement learning to fine-tune their capabilities for tackling complex challenges. OpenAI’s o3 stands out as a pioneering example, showcasing significant performance gains over its predecessor, o1, according to benchmark evaluations. The central question now looming over the field is the sustainability of this progress. Can these models continue to advance at the same rate simply by increasing computational power?
Epoch AI, a research organization focused on the societal impacts of artificial intelligence, has taken on the task of unraveling this question. Josh You, a data analyst at Epoch AI, has undertaken a comprehensive analysis to determine the current levels of computational investment in reasoning training and to assess the remaining potential for expansion.
The Computation Surge Behind Reasoning Models
OpenAI has publicly stated that o3 was trained with ten times the computational resources dedicated to reasoning compared to o1—a substantial increase achieved in just four months. An OpenAI-produced chart vividly illustrates the close correlation between computational power and performance on the AIME math benchmark. Epoch AI hypothesizes that these figures specifically pertain to the second phase of training, the reasoning training, rather than the complete model training process.
To put these figures into perspective, Epoch AI examined comparable models. DeepSeek-R1, for example, reportedly trained with around 6e23 FLOP (floating point operations per second) at an estimated cost of $1 million, achieved benchmark results similar to o1.
Tech giants Nvidia and Microsoft have also contributed to the development of reasoning models, providing publicly accessible training data. Nvidia’s Llama-Nemotron Ultra 253B utilized approximately 140,000 H100 GPU-hours, equivalent to roughly 1e23 FLOP, for its reasoning training phase. Microsoft’s Phi-4-reasoning employed even less computational power, below 1e20 FLOP. A critical factor distinguishing these models is their heavy reliance on synthetic training data generated by other AI systems. Epoch AI emphasizes that this reliance makes direct comparisons with models like o3 more difficult due to the inherent differences between real and synthetic data and its impact on model learning and generalization.
Defining "Reasoning Training": A Murky Area
Another layer of complexity stems from the lack of a universally accepted definition of "reasoning training." In addition to reinforcement learning, some models incorporate techniques like supervised fine-tuning. The ambiguity surrounding the components included in compute estimates introduces inconsistencies, making it challenging to accurately compare resources across different models.
As of now, reasoning models still consume significantly less computational power than the most extensive AI training runs, such as Grok 3, which exceeds 1e26 FLOP. Contemporary reasoning training phases typically operate between 1e23 and 1e24 FLOP, leaving considerable room for potential expansion – or so it seems at first glance.
Dario Amodei, CEO of Anthropic, shares a similar perspective. He posits that an investment of $1 million in reasoning training can yield significant progress. However, companies are actively exploring ways to increase the budget for this secondary training phase to hundreds of millions of dollars and beyond, which suggestsa future where the economics of training shift dramatically.
If the current trend of roughly tenfold increases in computational power every three to five months continues, reasoning training compute could potentially catch up to the total training compute of leading models as early as next year. However, Josh You anticipates that growth will eventually decelerate to approximately a 4x increase per year, aligning with broader industry trends. This deceleration will likely be driven by a combination of factors, including diminishing returns on investment in training, the increasing cost of compute resources, and the limitations of available training data.
Beyond Compute: The Bottlenecks on the Horizon
Epoch AI emphasizes that computational power is not the sole limiting factor. Reasoning training requires substantial quantities of high-quality, challenging tasks. Acquiring such data is difficult; generating it synthetically is even more so. The problem with synthetic data isn’t just authenticity; many argue the quality is poor. Additionally, the effectiveness of this approach outside of highly structured domains like mathematics and computer programming remains uncertain. Nonetheless, projects like "Deep Research" in ChatGPT, which utilizes a custom-tuned version of o3, suggest potential for broader applicability.
Labor-intensive behind-the-scenes tasks, such as selecting appropriate tasks, designing reward functions, and developing training strategies, also pose challenges. These developmental costs, often excluded from compute estimates, contribute significantly to the overall expense of reasoning training.
Despite these challenges, OpenAI and other developers remain optimistic. As Epoch AI notes, scaling curves for reasoning training currently resemble the classic log-linear progress observed in pre-training. Furthermore, o3 demonstrates substantial gains not only in mathematics but also in agent-based software tasks, indicating the versatile potential of this new approach.
The future of this progress hinges on the scalability of reasoning training – technically, economically, and in terms of content. The following points explore several key factors that will determine the future of these models:
- Technical Scalability: Refers to the ability to increase the computational resources used in training without encountering insurmountable technical hurdles. This includes advancements in hardware, software, and algorithms to efficiently utilize larger datasets and more powerful computing infrastructure. As models grow in size and complexity, technical scalability becomes increasingly critical for continued progress. The underlying architecture will need to evolve to keep pace with the sheer scale of the models.
- Economic Scalability: Entails the feasibility of increasing computational resources within reasonable budget constraints. If the cost of training scales linearly or exponentially with model size, it may become prohibitively expensive to pursue further gains. As such, cheaper and more efficient training may be necessary. Innovations in hardware and optimization techniques that reduce the cost per FLOP are crucial for economic scalability. The trend has been to focus on ever-larger models but with a finite budget, the incentives will shift to training the most efficient models.
- Content Scalability: Highlights the availability of high-quality training data that can effectively drive gains in reasoning ability. As models become more sophisticated, more difficult and diverse datasets are needed to challenge them and prevent overfitting. The availability of such datasets is limited, especially in domains that require complex reasoning. Synthetic data generation techniques can help to alleviate this bottleneck, but they must be carefully designed to avoid biases or inaccuracies that could degrade model performance.
The Future of Compute
It’s easy as laypeople to think that we are on the path of infinite compute. However, in reality, it is limited, and in the future, that limit might become more apparent. In this section, we will be exploring a few ways that compute might evolve in the futureand how those changes will affect the LLM industry.
Quantum Computing
Quantum computing represents a paradigm shift in computation, leveraging the principles of quantum mechanics to solve problems that are intractable for classical computers. While still in its nascent stages, quantum computing holds immense potential for accelerating AI workloads, including reasoning model training. Quantum algorithms like quantum annealing and variational quantum eigensolvers (VQEs) could potentially optimize model parameters more efficiently than classical optimization methods, reducing the computational resources required for training. For example, quantum machine learning algorithms could enhance the optimization of complex neural networks, leading to faster training times and potentially better model performance.
However, significant challenges remain in scaling up quantum computers and developing robust quantum algorithms. The technology is still largely experimental, and practical quantum computers with sufficient qubits (quantum bits) and coherence times are not yet readily available. Furthermore, developing quantum algorithms tailored to specific AI tasks requires specialized expertise and is an ongoing area of research. Widespread adoption of quantum computing in AI remains several years away and is only likely to be practical once computers are available and scalable. Current simulation software that can simulate quantum computing still requires massive amounts of computer power using classical computers, so real quantum computers should reduce the computational cost.
The impact on the LLM industry could be substantial, but there are currently a few technical hurdles that need to be resolved. One aspect is to improve error correction in quantum computers so that the calculations can be performed with the same confidence levels of classical computers. Another issue arises because of the quantum annealing as the best approach to enhance the LLM optimization is still being explored. This means that algorithms that are specific to enhancing LLM performance need to be developed and proven effective.
Neuromorphic Computing
Neuromorphic computing mimics the structure and function of the human brain to perform computation. Unlike traditional computers that rely on binary logic and sequential processing, neuromorphic chips utilize artificial neurons and synapses to process information in a parallel and energy-efficient manner. This architecture is well-suited for AI tasks that involve pattern recognition, learning, and adaptation, such as reasoning model training. Neuromorphic chips could potentially reduce the energy consumption and latency associated with training large AI models, making it more economically viable and environmentally sustainable.
Intel’s Loihi and IBM’s TrueNorth are examples of neuromorphic chips that have demonstrated promising results in AI applications. These chips are capable of performing complex AI tasks with significantly lower power consumption compared to traditional CPUs and GPUs. However, neuromorphic computing is still a relatively new field, and challenges remain in developing robust programming tools and optimizing algorithms for neuromorphic architectures. Furthermore, the limited availability of neuromorphic hardware and the lack of widespread expertise in neuromorphic computing have hindered the adoption of this technology in mainstream AI applications.
One key advantage of neuromorphic computing for reasoning models is its ability to handle unstructured and noisy data more effectively compared to traditional computers. This is because the human brain, which neuromorphic computing is modeled after, is inherently designed to process uncertain and ambiguous information. This capability can be particularly useful for reasoning tasks that involve real-world scenarios with incomplete or inconsistent information.
However, it’s important to note that neuromorphic computing is not a one-size-fits-all solution for AI. It is best suited for tasks that exhibit certain characteristics, such as high levels of parallelism, real-time processing requirements, and tolerance for noise. To fully utilize neuromorphic platforms to enhance reasoning models, further research is needed to understand the trade-offs between neuromorphic and traditional computing approaches.
Analog Computing
Analog computing utilizes continuous physical quantities, such as voltage or current, to represent and process information, rather than discrete digital signals. Analog computers can perform certain mathematical operations, such as differential equations and linear algebra, much faster and more efficiently than digital computers, especially in tasks that may be useful for reasoning. Analog computation can be useful for training models or for running inference when needed.
The inherent speed and energy efficiency of analog computing make it an attractive alternative to digital computing for reasoning models. By leveraging analog circuits to perform computationally intensive operations, AI models can more quickly derive conclusions from the inputed data.
However, analog computing faces challenges in precision, scalability, and programmability. Analog circuits are susceptible to noise and drift, which can degrade the accuracy of computations. Scaling up analog computers to handle large and complex AI models is also a technical challenge. Furthermore, programming analog computers typically requires specialized expertise and is more difficult than programming digital computers. Despite these challenges, there is growing interest in analog computing as a potential alternative to digital computing for specific AI applications, particularly those that demand high speed and energy efficiency.
A key area of focus for analog computing in reasoning models is the development of noise-tolerant analog circuits. Researchers are exploring various techniques to minimize the impact of noise and drift on the accuracy of calculations. Another area of active research is the exploration of hybrid analog-digital architectures that combine the advantages of both approaches.
Distributed Computing
Distributed computing involves distributing AI workloads across multiple machines or devices connected by a network. This approach allows organizations to leverage the collective computing power of a large number of resources to accelerate AI training and inference. Distributed computing is essential for training large language models (LLMs) and other complex AI models that require massive datasets and computational resources.
Frameworks like TensorFlow, PyTorch, and Apache Spark provide tools and APIs for distributing AI workloads across clusters of machines. These frameworks allow organizations to scale up their AI capabilities by adding more computing resources as needed. However, distributed computing introduces challenges in data management, communication overhead, and synchronization. Efficiently distributing data across multiple machines and minimizing communication delays are crucial for maximizing the performance of distributed AI systems. Additionally, ensuring that the different machines or devices are properly synchronized and coordinated is essential for achieving accurate and reliable results.
With that said, the distributed nature of the compute can also decrease the attack vectors. It also means that the compute is resilient to any one point of failure. Overall, organizations may be considering moreand more distributed solutions to make themselves robust against malicious attacks.
Conclusion
The trajectory of reasoning models is undeniably intertwined with the availability and scalability of computational resources. While the current pace of progress driven by increased compute is impressive, several factors, including the scarcity of high-quality training data, the increasing cost of compute, and the emergence of alternative computing paradigms, suggest that the era of unbridled compute scaling may be approaching its limits. The future of reasoning models will likely depend on our ability to overcome these limitations and explore new approaches to enhancing AI capabilities. With all of this information, we can assume that the rise in reasoning model capabilities might soon start to slow for one of the numerous constraints discussed. The shift will be necessary in the industry, since current compute consumption is growing at an unsustainable rate.