Lecture Notes: Generative Pretrained Transformer (GPT)

Overview of GPT

• GPT stands for Generative Pretrained Transformer
  • Generative: Bots that generate new text.
  • Pretrained: Underwent a learning process from vast amounts of data and can be fine-tuned for specific tasks.
  • Transformer: A type of neural network model fundamental to modern AI advances.

Purpose of Lecture

• To visually explain what happens inside a Transformer and the flow of data through it.
• Various models built with Transformers:
  • E.g., models that transcribe audio data or generate synthetic speech from text (like ChatGPT).
  • Dolly or Midjourney: Use text to generate images.
• Original use case for Transformers: Translating text from one language to another (introduced by Google in 2017).

Focus Area

• We'll focus on models like ChatGPT, which read a text and predict what comes next.
• This prediction is a probability distribution over many possible next text sequences.

Process of Text Generation

• A prediction model can generate longer text by iterating predictions and sampling from the generated distribution, appending new text iteratively.
• Example: GPT-2 vs. GPT-3—larger models generate more coherent and meaningful stories.

Detailed Steps

Tokenization and Embeddings

• Input text is broken into tokens (words or parts of words, or character combinations for text; parts of an image or sound for media data).
• Each token is linked with a vector (a list of numbers encoding the piece's meaning).
  • Similar meanings have vectors close together in high-dimensional space.

Attention Mechanism

• Vectors undergo an attention block allowing them to communicate and update based on context.
  • Determines relevant words in context and updates their meanings.
  • Similar to weighting relevant terms in a sentence for nuanced meaning.

Feed-Forward Layers

• Vectors go through a feed-forward layer (or multi-layer perceptron).
  • Operates on all vectors in parallel for uniform updates.
  • Conceptually like answering a list of questions to update the vector.

Repetition and Matrix Multiplications

• Alternating attention and feed-forward blocks, updating through matrix multiplications.
  • Transfer meanings through various network layers to final vectors.

Final Prediction

• The final vector produces a probability distribution over all possible next tokens.
  • Uses an Unembedding matrix to map this vector to a token probability list.
  • The prediction step repeats to generate continuous text.

Softmax Function

• Converts raw input numbers (logits) into a valid probability distribution.
  • Adjusts values for valid probabilities (between 0 and 1, sum to 1).
  • Can be adjusted via temperature parameter to control randomness in output.

Examples and Applications

• Animated demonstration of text generation with GPT-2 and GPT-3 differences.
• Early demos of GPT-3 completing stories based on initial snippets.
  • Use of system prompts to set framework for interactions.

Deep Learning Background

• Deep Learning: Uses data to define model behavior through examples.
  • From simple linear regression (e.g., house prices based on size) to complex models with billions of parameters like GPT-3.
  • Algorithms like backpropagation essential for large-scale model training.

Embeddings and Context

• Embeddings convert tokens to vectors considering both meaning and position in text/context.
  • E.g., the word