Frequentist Hypothesis Testing

May 29, 2024

Frequentist Hypothesis Testing

Overview

  • Nation of blobs popular coin flipping game.
  • The aim is to identify cheaters using trick coins.
  • Focus on developing a reliable method to catch cheaters (Frequentist Hypothesis Testing).

Objectives for the Test

  1. Low False Accusation Rate: Assure fair players aren't wrongly accused.
  2. High Detection Rate: Ensure cheaters are caught.
  3. Efficiency: Use the minimum number of coin flips.

Initial Experiment

  • Blobs flip coins five times.
  • Review frequency of heads results.
  • Evaluate suspicion based on streaks of heads.

Observations

  • Probability Calculations:
    • 1 head: 50%
    • 2 heads: 25%
    • 3 heads: 12.5%
  • Thresholds for accusations:
    • 5 heads out of 5: 3.125% chance fair.
    • Accuse based on the probability of streaks.

Designing the Test

  • False accusation set below 5%.
  • Initial test: 5/5 heads.
    • Not sufficient to meet requirements for catching cheaters.

Adding Statistics Terms

  • Positive Result: Test indicates cheating.
  • Negative Result: Test indicates fair play.
  • True Negative/Positive: Correctly identified.
  • False Negative/Positive: Incorrectly identified.
  • Effect Size & P-Values: Importance of setting thresholds.

Improving the Test

  • Move beyond binary outcomes to mixtures of heads and tails.
  • Introducing more flips (e.g., 10 flips, accuse if 7 or more heads).
  • Binomial Distribution: Used to calculate precise probabilities for varied outcomes.
  • Testing example: Accuse if ≥16/23 heads maximizes catching cheaters.

Running the Refined Test

  • Real-life applications; effective balance needed.
  • Example simulations show practical results of various thresholds.

P-Values

  • Help determine the probability of results if an assumption is true.
  • Example: A blob with 17 heads; P-value 1.7%.

Final Thoughts

  • Frequentist Hypothesis Testing forms a base for scientific experiments.
  • Ensures a balance between false accusations and true detection.
  • Importance of validating assumptions.

Conclusion

  • This framework is vastly applicable, reflecting common practices in scientific studies.
  • Introduction into Bayesian Hypothesis Testing as the next step.