Scaling Paradigm: Next Word Prediction

• Multi-task learning: Solves many problems (Jason Wei)

• Scaling: Improved smoothly over ~7 orders of magnitude (Kaplan et al, 2020):
  • With respect to Model size, Dataset size, Training compute
  • Abilities unlocked at specific scales (Jason Wei)

• Bottleneck: Some next words are very hard to predict

  • System 1 thinking is fast, intuitive
  • Uses same amount of compute to solve easy and hard problems (Jason Wei)

• Chain-of-Thought (CoT) prompting: A work-around (Wei et al, 2022)

  • Prompt LLM to “think step by step”
  • Enforces System 2 thinking, which is conscious, effortful
    • When you do a math problem, you perform intermediate steps (e.g., store variables)
    • CoT forces the model to output these intermediates as tokens (Nathan Lambert)

New Scaling Paradigm: Scaling RL on CoT

• Train LLM to do CoT explicitly (OpenAI blog post, Nathan Lambert)

  • Training data: Has explicitly correct answers (e.g., coding, math)
  • Model: Can sometimes generate a correct solution
  • Grader: Can verify model outputs and give the model a reward for correct solutions
  • Policy: Nudge model weights so that it’s more likely to produce high-reward outputs
  • Sample many different reasoning trajectories on a training example, reward correct one

  

• New scaling law (Noam Brown, Jason Wei)

• Benchmarks getting saturated more quickly (David Rein)

  

Prompting

• Some confusion on o1 models
• Give it briefs (c/o @benhylak)