Stat Quest: Statistical Power

Introduction

• Presenter: Josh Starburns
• Topic: Statistical Power
• Prerequisites: Familiarity with p-values and normal distribution
• Note: Check other quests for p-values and normal distribution if not familiar

Example with Distributions

• Two distributions:
  • Weights of mice on a special diet (left)
  • Weights of mice eating normal food (right)
• Little overlap => easy to see diet difference
  • Special diet mice weigh less than normal diet mice

Statistical Test and p-values

• Collecting measurements:
  • Small set from special diet
  • Small set from normal diet
  • Plot and compare means
• Example Result: p-value = 0.0004
  • p-value < 0.05 => correctly reject null hypothesis (data sets aren't from same distribution)
• Repetition: High probability that tests will give small p-value, correctly rejecting null hypothesis
• Exception: Sometimes data may overlap => large p-value (> 0.05) => cannot reject null hypothesis

Concept of Power

• Power: Probability of correctly rejecting the null hypothesis
  • High probability of small p-value => large power
• Power Concepts:
  • No difference between diets => power concept doesn't apply
  • Slight difference, with overlap => power applies

Sample Size and Power

• Small Sample Size:
  • Example: 3 mice on each diet
  • Result: p-value = 0.34 => fail to reject null hypothesis
  • Repetition: Most tests fail to reject
  • Exception: Sometimes no overlap => correctly reject null hypothesis
• Power Analysis:
  • Increasing number of measurements => increase power
  • Determines how many measurements needed for good power

Summary

• Power Definition: Probability of correctly rejecting null hypothesis
• High Power: Low overlap between distributions => high chance of rejecting null hypothesis
• Low Power: High overlap & small sample size => low chance of rejecting null hypothesis
• Increasing Power: Collect more measurements
• Power Analysis: Tells number of measurements needed for good power

Conclusion

• Subscribe for more content
• Support options: Patreon, channel membership, original songs, t-shirts, hoodies, donations
• "Quest on!"