What is Scaling Laws (in AI)?

Scaling laws in AI are mathematical relationships that describe how a model’s performance improves predictably as you increase compute, data, and model size. Discovered through empirical research, these laws were the theoretical foundation for the AI scaling era — the insight that making models bigger, with more data and compute, reliably makes them smarter.

The Kaplan Scaling Laws (2020)

In 2020, Jared Kaplan and colleagues at OpenAI published a landmark paper showing that language model performance follows power-law relationships with three variables: model parameters, training dataset size, and compute used in training. Double the compute, and you get a predictable performance improvement — across 7 orders of magnitude. This was transformative. Instead of guessing how much to invest in training, researchers could predict results in advance. “Just scale it up” became a justified strategy, producing GPT-3, GPT-4, and their successors.

The Chinchilla Laws (2022)

DeepMind’s Chinchilla paper refined the scaling laws significantly. It showed that previous large models were compute-efficient but data-inefficient — too large relative to their training data. The Chinchilla result: for any given compute budget, you should use a smaller model trained on significantly more data. The optimal ratio is roughly 20 tokens of training data per model parameter. This insight contributed to models like Llama 2 and Mistral.

What Scaling Laws Don’t Tell You

• Emergent capabilities: New abilities that appear suddenly at certain scale thresholds. See What is Emergent Behavior?
• Alignment and safety: A more capable model isn’t automatically safer or more aligned.
• Real-world usefulness: Benchmark performance doesn’t perfectly translate to business value.

Are Scaling Laws Still Holding?

There is ongoing debate. Some researchers argue returns from pure scaling are diminishing — we’re running out of high-quality internet text. Others point to test-time compute as a new scaling frontier. Traditional training-time scaling is slowing, while new approaches (inference-time compute, synthetic data, multimodal data) open new paths. See also Small Language Models and Large Language Models.

Key Takeaways

• Scaling laws describe predictable performance improvements as AI models grow in size, data, and compute.
• The 2020 Kaplan paper and 2022 Chinchilla paper are the foundational references.
• Chinchilla showed that previous large models were undertrained — the data/parameter ratio matters.
• Scaling laws don’t predict emergent behaviors or alignment properties.
• The frontier is shifting from training-time to inference-time scaling approaches.

Frequently Asked Questions

What is the basic scaling law formula?

Performance (measured as loss) scales as a power law of compute, data, and parameters. Roughly: L ~ C^(-0.05) where L is loss and C is compute. The exact coefficients vary by architecture and task.

Do scaling laws apply to image and video AI too?

Yes. Similar scaling relationships have been found in vision models, multimodal models, and some reinforcement learning systems — suggesting these are fundamental properties of learning from data, not quirks of language models.

Why don’t small companies just train smaller models?

They do. Meta’s Llama series and Mistral models are designed for efficiency at smaller scales. But even optimally scaled small models have hard capability ceilings — some tasks require large parameter counts regardless of efficiency.

Has anyone found the limits of scaling?

Not definitively. We haven’t hit a hard wall, but the marginal gains from additional scale appear to be shrinking. This is part of why the industry is diversifying approaches rather than betting everything on bigger models.

What is Chinchilla optimal compute allocation?

For a given compute budget, Chinchilla optimal training allocates roughly equal resources to model size and data. The rule of thumb is ~20 tokens of training data per model parameter — a 7B parameter model needs ~140B training tokens for optimal efficiency.

Sources

• Wikipedia — Scaling Laws in AI Definition
• Kaplan et al. (2020) — Scaling Laws for Neural Language Models (arXiv)
• Hoffmann et al. / DeepMind (2022) — Training Compute-Optimal Large Language Models (Chinchilla, arXiv)

You May Also Like

• What Is Artificial Intelligence
• AI Glossary
• AI for Dummies
• How Does AI Work
• Essential AI Skills

---

Get free AI tips daily → Subscribe to Beginners in AI

Sources

This article draws on official documentation, product pages, and industry reporting. Specific sources are linked inline throughout the text.

Last reviewed: April 2026