Artificial intelligence (AI) scaling laws help us understand and predict how model performance improves as model size, training data, and computational resources increase. Often helped to predict performance, practitioners use this data to develop large language models (LLMs), AI systems that leverage and create vast amounts of textual data. With these topics on the minds of consumers and investors alike, questions often arise around the relationship between scaling laws and LLMs and whether the models are reaching diminishing marginal returns.
In our second installment of Navigating the Boom, where we confront GenAI’s most pressing questions, William Blair equity research analysts explore AI's technological advancements and the potential ways experts are scaling them.
The Three Core Levers of AI Scaling
AI scaling laws assert that model performance improves predictably as the number of parameters increases and the amount of training data grows. To see the maximum benefit in model performance, each of the three core levers of AI scaling—compute, algorithms, and data—need to be increased and improved. Throughout the second half of 2024, investors began to worry that AI scaling laws exhibited diminished marginal returns as data available for LLM pre-training seemed to reach a natural limit. As a result, investors and practitioners began to ask, “Does doubling computing capacity no longer provide the same level of model performance improvement as in prior generations?”
In our view, the reality is that as they approach the upper bounds of what is financially and technically feasible, model providers are encountering diminishing returns for the LLMs. The straightforward path of just “making the model bigger” no longer looks to be the best strategy for advancing the model’s capacities. To overcome these limitations, researchers and practitioners are introducing new vectors of model improvement beyond the pre-training of LLMs, including:
- Test-time compute
- Multimodal AI
- New model training techniques
- AI model chaining
- Expanding context windows
- New algorithmic techniques
Even if LLM pre-training is seeing diminishing returns, we see ample room for ever-higher computing levels to drive continued material AI model performance improvements.
Is the LLM Becoming a Commodity?
On one hand, it’s incredibly difficult to create an LLM. The barriers to entry in the LLM market are high, and only those with significant funding and access to computing power can compete in this space. It’s estimated that some platforms cost $100 million to train, with our analysts finding some estimates reaching up to $1 billion. Even if LLMs become more commoditized over time, this doesn’t necessarily mean LLM providers themselves will become commoditized.
Developers of LLMs can take two different routes when creating and releasing their models: closed source and open source. In closed-source models, the underlying code, training data, and parameters aren’t available to the public except through an application programming interface. In contrast, open-source models are created as collaboration software, where the original source code and models are freely available to the public for redistribution and modification. Though closed-source models may see greater adoption in the near term, over time, we expect developers and data engineers may prefer the transparency and flexibility afforded by open-source solutions.
LLMs Versus SLMs
Large-scale models have captured most of the attention in the GenAI market. However, these LLMs require tremendous computational power and can be expensive to implement. As a result, developers are creating smaller, more targeted models for specific use cases. Both types of models offer their own advantages and limitations, shaping the future of GenAI development across both consumer and enterprise markets.
The shift to smaller, more compact models will likely democratize the use of AI and allow more businesses and innovators to leverage their data at reduced costs. William Blair’s research indicates a growing preference among customers for building or fine-tuning their own smaller, more-tailored GenAI models, and we expect enterprises will leverage numerous small language models (SLMs) to address different use cases. The bottom line, however, is that we see room for different-sized models to succeed in the AI market, as LLMs and SLMs are designed for fundamentally different purposes.
William Blair analysts will continue to share important insights as investors and practitioners continue to understand the technological advancements and scale at which we can best use AI.
For more information about William Blair’s technology, media and communications research, please contact us or your William Blair representative.