There is a debate in the tech world about whether AI models are reaching a plateau. The rate of AI model improvement appears to be slowing, but some tech leaders insist there is no barrier. This has prompted a discussion over how companies can overcome AI bottlenecks ethically and efficiently. Business Insider spoke to 12 people at the forefront of the AI boom to find out the path forward.

Silicon Valley leaders deeply invested in the AI boom have a message for critics: their technology has not hit a wall. A fierce debate over whether improvements in AI models have reached their limit has emerged in recent weeks, prompting several CEOs to respond. OpenAI's CEO Sam Altman was among the first to speak out, posting on X this month that "there is no wall."

Dario Amodei, CEO of rival firm Anthropic, and Jensen Huang, the CEO of Nvidia, have also disputed reports that AI progress has slowed. Others, including Marc Andreessen, say AI models aren't getting noticeably better and are converging to perform at roughly similar levels.

This is a critical question for the tech industry. If traditional AI model training methods are yielding diminishing returns, it could undermine the core reason for an unprecedented investment cycle that's funding new startups, products, and data centers — and even restarting idled nuclear power plants.

Business Insider spoke to 12 people at the forefront of the AI industry, including startup founders, investors, and current and former insiders at Google DeepMind and OpenAI, about the challenges and opportunities ahead in the quest for superintelligent AI.

They said that tapping into new types of data, building reasoning into systems, and creating smaller but more specialized models are some of the ways to keep AI progress moving forward. Additionally, companies are focusing on addressing ethical and privacy implications when using private and synthetic data to overcome limitations in publicly available training data.

The Pre-Training Dilemma

Researchers point to two key obstacles that companies may encounter in an early phase of AI development, known as pre-training. The first is access to computing power, specifically specialist chips called GPUs. It's a market dominated by Nvidia, which has faced supply constraints due to nonstop demand.

"If you have $50 million to spend on GPUs but you're on the bottom of Nvidia's list — we don't have enough kimchi to throw at this, and it will take time," said Henri Tilloy, partner at French VC firm Singular.

There is another supply problem: training data. AI companies have run into limits on the quantity of public data they can secure to feed into their large language models, or LLMs, in pre-training.

This phase involves training an LLM on a vast amount of data, typically scraped from the internet and processed by GPUs. That information is then broken down into "tokens," which are the fundamental units of data processed by a model.

While adding more data and GPUs to a model has reliably produced smarter models year after year, companies are exhausting the supply of publicly available data on the internet. Research firm Epoch AI predicts usable textual data could be depleted by 2028.

"The internet is only so large," Matthew Zeiler, founder and CEO of Clarifai, told BI.

Multimodal and Private Data

Eric Landau, cofounder and CEO of data startup Encord, said that other data sources will offer a path forward in the scramble to overcome the bottleneck in public data.

One example is multimodal data, which involves feeding AI systems visual and audio sources of information, such as photos or podcast recordings. "That's one part of the picture," Landau said. "Just adding more modalities of data." AI labs have already started using multimodal data as a tool, but Landau says it remains "very underutilized."

Sharon Zhou, cofounder and CEO of LLM platform Lamini, sees another vastly untapped area: private data. Companies have been securing licensing agreements with publishers to gain access to their extensive collections of information. OpenAI, for instance, has struck partnerships with organizations such as Vox Media and Stack Overflow to bring copyrighted data into their models.

"We are not even close to using all of the private data in the world to supplement the data we need for pre-training," Zhou said. "From work with our enterprise and even startup customers, there's a lot more signal in that data that is very useful for these models to capture."

However, using private data raises significant ethical and privacy concerns. Companies are becoming increasingly cautious about how they handle such data to ensure compliance with regulations and maintain public trust. "We must be careful to respect user privacy and data ownership when leveraging private datasets," said a senior researcher at Google DeepMind. "Establishing clear consent and using data anonymization techniques are essential steps."

A great deal of research effort is now focused on enhancing the quality of data that an LLM is trained on rather than just the quantity. Researchers could previously afford to be "pretty lazy about the data" in pre-training, Zhou said, by just throwing as much as possible at a model to see what worked. "This isn't totally true anymore," she said.

One solution that companies are exploring is synthetic data, an artificial form of data generated by AI.

According to Daniele Panfilo, CEO of startup Aindo AI, synthetic data can be a "powerful tool to improve data quality," as it can "help researchers construct datasets that meet their exact information needs." This is particularly useful in a phase of AI development known as post-training, where techniques such as fine-tuning can be used to give a pre-trained model a smaller dataset that has been carefully crafted with specific domain expertise, such as law or medicine.

In theory, synthetic data offers a useful way to hone a model's knowledge and make it smaller and more efficient. In practice, companies are also addressing the potential risks associated with synthetic data, particularly regarding data privacy and the phenomenon known as "model collapse." A paper published in July in the journal Nature warned of the risk of "model collapse" with indiscriminate use of synthetic data. This has led to increased focus on ensuring synthetic data is used responsibly, with proper human oversight and validation to maintain data integrity and prevent the amplification of biases.

Building a Reasoning Machine

For some, simply focusing on the training portion won't be sufficient.

Former OpenAI chief scientist and Safe Superintelligence cofounder Ilya Sutskever told Reuters this month that results from scaling models in pre-training had plateaued and that "everyone is looking for the next thing."

That "next thing" appears to be reasoning. Industry attention has increasingly turned to an area of AI known as inference, which focuses on the ability of a trained model to respond to queries and information it might not have seen before, using reasoning capabilities.

At Microsoft's Ignite event this month, the company's CEO Satya Nadella said that instead of seeing AI scaling laws hit a wall, he was seeing the emergence of a new paradigm for "test-time compute," which is when a model has the ability to take longer to respond to more complex prompts from users. Nadella pointed to a new "think harder" feature for Copilot — Microsoft's AI agent — which boosts test time to "solve even harder problems."

Aymeric Zhuo, cofounder and CEO of AI startup Agemo, said that AI reasoning "has been an active area of research," particularly as "the industry faces a data wall." He told BI that improving reasoning requires increasing test-time or inference-time compute.

Typically, the longer a model takes to process a request, the more accurate the outcomes it generates. Right now, models are being queried in milliseconds. "It doesn't quite make sense," Sivesh Sukumar, an investor at investment firm Balderton, told BI. "If you think about how the human brain works, even the smartest people take time to come up with solutions to problems."

In September, OpenAI released a new model, o1, which tries to "think" about an issue before responding. One OpenAI employee, who asked not to be named, told BI that "reasoning from first principles" is not the forte of LLMs as they work based on "a statistical probability of which words come next," but if we "want them to think and solve novel problem areas, they have to reason."

Noam Brown, a researcher at OpenAI, thinks the impact of a model with greater reasoning capabilities can be extraordinary. "It turned out that having a bot think for just 20 seconds in a hand of poker got the same boost in performance as scaling up the model by 100,000 times and training it for 100,000 times longer," he said during a talk at TED AI last month.

These advancements in reasoning capabilities are seen as a promising avenue to improve AI performance without solely relying on larger datasets or more compute power. By allowing models to take more time to generate responses, they can perform deeper reasoning and produce more accurate results. This approach can be more cost-effective compared to the exponential costs of scaling up models. However, increasing inference-time compute also presents challenges, such as higher latency and increased computational costs during inference, which companies are working to optimize.

The AI Boom Meets Its Tipping Point

These efforts give researchers reasons to remain hopeful, even if current signs point to a slower rate of performance leaps. As a separate former DeepMind employee who worked on Gemini told BI, people are constantly "trying to find all sorts of different kinds of improvements."

That said, the industry may need to adjust to a slower pace of improvement.

"I just think we went through this crazy period of the models getting better really fast, like, a year or two ago. It's never been like that before," the former DeepMind employee told BI. "I don't think the rate of improvement has been as fast this year, but I don't think that's like some slowdown."

Lamini's Zhou echoed this point. Scaling laws — an observation that AI models improve with size, more data, and greater computing power — work on a logarithmic scale rather than a straight line, she said. In other words, think of AI advances as a curve rather than a straight upward line on a graph. That makes development far more expensive "than we'd expect for the next substantive step in this technology," Zhou said.

She added: "That's why I think our expectations are just not going to be met at the timeline we want, but also why we'll be more surprised by capabilities when they do appear."

Companies are exploring alternative strategies to achieve significant AI advancements without unsustainable resource investment. These include focusing on data efficiency, improving model architectures, leveraging transfer learning, and developing smaller, specialized models tailored for specific tasks. By optimizing existing resources and enhancing algorithmic efficiency, companies aim to achieve better performance without the need for exponentially more data and compute power.

Companies will also need to consider how much more expensive it will be to create the next versions of their highly valued models. According to Anthropic's Amodei, a training run in the future could one day cost $100 billion. These costs include GPUs, energy needs, and data processing.

Whether investors and customers are willing to wait longer for the superintelligence they've been promised remains to be seen. Issues with Microsoft's Copilot, for instance, are leading some customers to wonder if the much-hyped tool is worth the money.

For now, AI leaders maintain that there are plenty of levers to pull — from new data sources to a focus on inference — to ensure models continue improving. Investors and customers just might have to be prepared for advancements to come at a slower pace compared to the rapid progress seen when OpenAI launched ChatGPT two years ago.

Bigger problems lie ahead if they don't.