Lecture Notes: Understanding Simpson's Paradox in Statistics

Importance of Statistics

• Statistics are highly persuasive and form the basis of important decisions.
• Organizations and countries often rely on statistical data for decision-making.

Problem with Statistics

• Statistics can be misleading due to hidden factors.
• A set of statistics might have underlying issues that can alter conclusions.

Example: Choosing a Hospital

• Scenario: Deciding between Hospital A and Hospital B for surgery based on survival rates.
  • Hospital A: 900 out of 1,000 patients survived.
  • Hospital B: 800 out of 1,000 patients survived.
• Initial impression: Hospital A seems better.

Analyzing by Health Condition

• Patients in Poor Health:
  • Hospital A: 100 arrived, 30 survived.
  • Hospital B: 400 arrived, 210 survived (52.5% survival rate).
• Patients in Good Health:
  • Hospital B has a survival rate of over 98%.
• Conclusion: Hospital B is better regardless of initial patient health.

Introduction to Simpson's Paradox

• What is Simpson's Paradox?
  • A phenomenon where data appears to show opposite trends depending on grouping.
  • Often occurs due to a 'lurking variable,' an influential hidden factor.
• Example: Hospital choice dilemma shows the paradox when data is aggregated.

Real-World Examples of Simpson's Paradox

1. UK Study on Smokers vs. Nonsmokers

   • Initial data: Smokers had higher survival rates over 20 years.
   • Lurking variable: Age group; nonsmokers were older on average and more likely to die.

2. Florida Death Penalty Cases

   • Initial data: No racial disparity in death sentences between black and white defendants.
   • Lurking variable: Race of the victim; black defendants more likely sentenced to death.

Avoiding the Paradox

• There is no universal solution to avoid Simpson's Paradox.
• Data can be misleading if grouped or categorized arbitrarily.
• To mitigate risks, study the actual situations and consider potential lurking variables.
• Be cautious of those who might use data to manipulate or promote agendas.

Conclusion

• Critical analysis of statistical data is essential to avoid misinterpretation.
• Awareness of Simpson’s Paradox helps in understanding complex data trends.