Nathan Lambert's Talk on Aligning Open Language Models

Introduction

• Speaker: Nathan Lambert, Research Scientist at Allen Institute for AI.
• Focus Area: Reinforcement Learning from Human Feedback (RLHF) and author of Interconnects DOAI.
• Topic: Aligning Open Language Models
• Context: Fine-tuning and alignment space evolution since ChatGPT.
• Q&A: Clarifying questions welcomed, main discussion and questions at the end.

Historical Context of Language Models

• Claude Shannon: Early work on approximating and arranging characters to create language models.
• Loss Function: Auto-aggressive loss function enables predicting sequences of text.
• 2017 Transformer Paper: Introduced the attention mechanism (Attention is All You Need).
• 2018 Developments:
  • Elmo: Contextualized word embeddings.
  • GPT-1 & BERT: Generated text (GPT-1) and classification (BERT).
• Scaling Laws: GPT-2 showed linear decrease in test loss with increased compute (orders of magnitude).

Major Milestones in Language Model Evolution

• GPT-3 (2020): Demonstrated significant capabilities with few-shot and multi-shot learning.
• Stochastic Parrots Paper (2021): Discussed risks of large language models (LLMs) being