The core "rules" that research has established about AI training and capacity fall into a few layers:

## 1. The Power Law (Kaplan et al., 2020 — OpenAI)

Model performance (measured as cross-entropy loss) follows a **predictable power-law curve** as you scale any of three variables:

- **N** — number of parameters
- **D** — dataset size (tokens)
- **C** — compute (FLOPs)

The key implication: you can **predict** a model's quality before training it, just from the budget. The curves are remarkably stable across many orders of magnitude.

## 2. The Chinchilla Correction (Hoffmann et al., 2022 — DeepMind)

The most important refinement: **model size and data size must scale equally** for optimal efficiency.

> Rule of thumb: **~20 tokens per parameter**

This meant most models before 2022 (GPT-3, Gopher, etc.) were **under-trained** — too big for the data they saw. The fix: smaller model, more data, same compute → better performance.

## 3. The Three-Stage Training Pipeline (now standard)

| Stage | What it does |
|-------|-------------|
| **Pre-training** | Learn general patterns from massive text (the "raw intelligence") |
| **Fine-tuning / RLHF** | Align behavior to human preferences |
| **RLVR** (Reinforcement Learning from Verifiable Rewards) | Model practices problems with checkable answers (math, code) and self-corrects — the engine behind 2024–2026 "reasoning models" |

## 4. Test-Time Compute (the new axis)

Since 2024, a **fourth scaling axis** emerged: giving the model more time to "think" at inference (chain-of-thought, tree search, self-verification). This is **separate from training scale** — a smaller model that reasons longer can outperform a larger model that answers immediately.

## 5. What Actually Matters (and what doesn't)

| Finding | Source |
|---------|--------|
| **Architecture details** (width, depth, aspect ratio) matter **less** than total parameter count | Kaplan et al. |
| **Data quality > data quantity** — curated data beats raw web scraping | Multiple 2025 studies |
| **MoE (Mixture-of-Experts)** decouples capacity from cost — store trillions of params, activate only ~20 B per token | Industry standard since 2024 |
| **Generalization spillover** — training on math/code makes models better at unrelated reasoning tasks | DeepSeek-R1, OpenAI o1 |
| **Architecture can substitute for data** — brain-inspired designs show brain-like activity even untrained | Johns Hopkins, Dec 2025 |
| **RL needs a "cold start"** — imitation learning before RL works best | DeepSeek-R1 report |

## 6. The Open Questions

- **Data wall:** We're running out of high-quality unique text. Synthetic data and multi-epoch training are workarounds, but it's unclear how far they extend the curve.
- **Does scaling alone reach AGI?** The power law is an empirical fit, not a law of nature. No one knows if it holds indefinitely or plateaus.
- **Is there a "phase transition"?** Some capabilities (reasoning, code, math) appeared to emerge **suddenly** at certain scale thresholds rather than gradually — the "emergent abilities" debate is still unresolved.

**In one sentence:** The basic rule is that performance improves predictably with scale, but the *optimal allocation* of that scale across parameters, data, compute, and test-time thinking has been the central research question since 2020.