I. From GPT-4 to AGI: Counting the OOMs

Introduction

• AGI by 2027 is plausible.
• GPT-2 to GPT-4 progressed AI from preschool to high school-level abilities in four years.
• Trends: compute, algorithmic efficiencies, and unhobbling gains.

Key Trends and Developments

The Last Four Years

• GPT-4's capabilities shocked many, continuing a decade of rapid AI progress.
• GPT-2 to GPT-4: Evolution from preschool to high school capabilities.
  • GPT-2 (2019): Preschooler level, impressive language command but limited.
  • GPT-3 (2020): Elementary level, capable of basic tasks and simple coding.
  • GPT-4 (2023): High school level, sophisticated in coding, writing, reasoning.

Deep Learning Trends

• Dramatic progress in AI capabilities
• Models are saturating benchmarks quickly.
• Previous skepticism about AI capabilities often proven wrong.

Counting OOMs (Orders of Magnitude)

• Compute: Significant increases in model training power.
• Algorithmic Efficiencies: Consistent improvements.
• Unhobbling: Unlocking potential with tools like RLHF and CoT.

The Data Wall

• Potential bottlenecks as models exhaust internet data.
• Labs are exploring new methods like synthetic data and RL to overcome constraints.

Unhobbling and Future Progress

• Unhobbling is crucial for realizing AI potential.
• Future improvements could transform AI from chatbots to agent-coworkers.

The Next Four Years

• Expect another significant jump in AI capabilities by 2027.
• Models could evolve to perform complex tasks akin to expert coworkers.

Conclusion

• The pace of AI progress is not slowing.
• AGI by 2027 is a serious possibility based on current trends.
• The transition from GPT-4 to AGI is marked by continued scaling and unhobbling.

Addendum

• The current decade is critical for achieving AGI.
• Future progress might slow after massive gains in this decade.