Lecture Notes: XG Boost Part 1 - Regression with Trees

Introduction

• Presenter: Josh Stormer
• Topic: XG Boost for Regression
• Prerequisites: Familiarity with:
  • Gradient Boosting for regression
  • Regularization concepts
• Structure:
  • Part 1: XG Boost regression trees
  • Part 2: XG Boost classification trees
  • Part 3: Mathematical details connecting regression and classification

Overview of XG Boost

• Definition: A comprehensive machine learning algorithm
• Intended Use: Large, complex datasets
• Unique Aspect: Utilizes unique regression trees

Initial Steps in XG Boost

• Initial Prediction:
  • Default value: 0.5
  • Applies to both regression and classification

XG Boost Trees

• Construction:
  • Regression trees fit to residuals
  • Different from regular regression trees

Building XG Boost Trees

1. Start with a Single Leaf:
   • All residuals go to this leaf
2. Calculate Similarity Score:
   • Formula: Sum of residuals squared / (number of residuals + lambda)
   • Lambda: Regularization parameter
3. Splitting the Leaf:
   • Evaluate if splitting improves similarity
   • Calculate gain: Sum of similarity scores of leaves - similarity score of root
4. Choosing the Best Split:
   • Compare different thresholds and choose the one with the highest gain

Example

• Data: Simple dataset with drug dosages and effectiveness
• Steps:
  • Initial prediction at 0.5
  • Calculation of similarity scores and gains for different splits
  • Selection of best splits based on gain

Pruning the Trees

• Pruning Based on Gain:
  • Use parameter gamma
  • If gain - gamma < 0, prune the branch

Regularization with Lambda

• Lambda Effects:
  • Reduces prediction sensitivity
  • Affects similarity scores and output values

Making Predictions

• New Predictions:
  • Start with initial prediction
  • Add output of the tree scaled by learning rate (Etta, default 0.3)
  • Build successive trees to further reduce residuals

Summary

• Key Concepts:
  • Calculation of similarity scores and gains
  • Tree pruning with gamma
  • Regularization with lambda affects tree complexity and output values

Next Steps

• Part 2 will cover classification trees
• Encourage review of additional resources for deeper understanding

Closing

• Encouragement to subscribe and support the series
• Links to additional resources and support options available in the description