StatQuest: ROC and AUC

Introduction

• Presenter: Josh Starmer
• Topic: ROC (Receiver Operating Characteristic) and AUC (Area Under the Curve)
• Builds on previous concepts: confusion matrix, sensitivity, and specificity.
• Example is based on logistic regression.

Understanding the Data

• Y-axis: Two categories (Obese and Not Obese)
• Blue dots: Obese mice
• Red dots: Mice that are not obese
• X-axis: Weight
  • Example of classification based on weight:
    • Heavy mouse may be not obese (e.g., muscular mouse).
    • Light mouse may be classified as obese for its size.

Logistic Regression

• Logistic regression curve converts y-axis to probability of being obese.
• Probability threshold of 0.5 used for classification:
  • Mice with probability > 0.5 classified as obese.
  • Mice with probability <= 0.5 classified as not obese.
• Evaluating effectiveness with new known samples of obese and not obese mice.

Confusion Matrix

• Summarizes classifications:
  • True Positives (TP): Correctly classified obese samples.
  • False Positives (FP): Non-obese samples incorrectly classified as obese.
  • True Negatives (TN): Correctly classified non-obese samples.
  • False Negatives (FN): Obese samples incorrectly classified as non-obese.

Impact of Different Thresholds

• Lowering threshold (e.g., to 0.1):
  • All obese mice classified correctly but increases False Positives.
• Raising threshold (e.g., to 0.9):
  • Fewer False Positives, but may miss some obese samples.
• Importance of selecting optimal thresholds based on context (e.g., medical outcomes).

Receiver Operating Characteristic (ROC) Graphs

• Summarizes true positive rates (TPR) vs. false positive rates (FPR).
• Y-axis: True Positive Rate (Sensitivity).
• X-axis: False Positive Rate (1 - Specificity).
• Visualizes performance of different thresholds without confusion matrices.

Creating a ROC Graph

• Start by plotting points based on confusion matrices for different thresholds.
• Connect points to visualize ROC curve.
• Optimum threshold indicated by the curve's proximity to the top left corner of the graph.

Area Under the Curve (AUC)

• AUC quantifies the overall performance of the model:
  • AUC closer to 1 indicates better model performance.
• Useful for comparing different ROC curves:
  • Example: Red ROC curve (logistic regression) vs. Blue ROC curve (random forest).

Other Metrics

• Precision as an alternative to FPR:
  • True Positives / (True Positives + False Positives).
  • Useful in imbalanced datasets (e.g., rare diseases).

Conclusion

• ROC curves and AUC provide insight into model performance and optimal thresholds.
• Summary:
  • Identify better thresholds for classification.
  • AUC to compare methods.
• Encouragement to engage with StatQuest content and support with merchandise.