Lecture Notes: Statistical Sins and Course Wrap-Up

Introduction

• Last lecture of the course.
• Focus on discussing statistical sins.
• Wrap-up of key topics from the course.

Global Warming: Fact or Fiction?

• Statistical Sin: Misleading presentation of data by manipulating the y-axis.
  • Example: Temperature data from 1880 to 2014.
  • Caution against charts that don’t start y-axis at zero.
• Comparison of two data presentations:
  • Original chart highlights change in temperature.
  • Adjusted chart starting at zero shows minimal change.

Flu and Fever Data

• Fever and flu example: Plotting fever progression.
  • Misleading to assume temperature can range from 0 to 200 degrees Fahrenheit.
• Moral: Truncate values appropriately but avoid deception.

Time Series and Trends

• Confusion between fluctuations and trends.
• Importance of choosing appropriate time intervals.
  • Global warming trends require long-term data analysis.
  • Short-term weather events are not indicative of climate change.

Cherry-Picking Data

• Statistical Sin: Cherry-picking data points to support a claim.
  • Example: Charts shown in U.S. Senate argument against global warming.
  • Selecting specific years can misrepresent trends.
• Importance of comprehensive data analysis to avoid skewed conclusions.

Context Matters in Statistics

• Statistics without context can be misleading:
  • Example: 99.8% of firearms not used in violent crime.
  • Contextualize numbers to avoid misinterpretation.
• Swine Flu vs. Seasonal Flu Deaths:
  • 159 deaths from swine flu vs. 36,000 from seasonal flu.

Percent Change and Probability

• Caution against presenting statistics without proper context.
  • Example: Skipping lectures increases failure probability by 50%.
  • Need to understand the base probability to assess significance.

Cancer Clusters

• Definition: Greater than expected cancer cases in a geo-area.
• Most reported clusters don't meet statistical significance.
• Example: Hypothetical attorney looking for cancer clusters.
  • Analysis using random simulations to assess probability.
  • Statistical Sin: Multiple hypothesis testing leads to biased conclusions.

Summary of Course Content

1. Key Topics Covered:
   • Optimization problems, stochastic thinking, modeling.
   • Emphasis on programming skills and use of libraries.
2. Optimization:
   • Formulation of objective functions and constraints.
   • Greedy algorithms vs. dynamic programming.
   • Applications: knapsack problems, clustering.
3. Stochastic Thinking:
   • Importance of randomness in modeling.
   • Application of randomness in non-probabilistic problems.
4. Modeling the World:
   • Inaccuracies in models, confidence intervals, and distributions.
   • Introduction to machine learning: clustering and classification.

Final Thoughts

• Importance of skepticism in data interpretation.
• Encouragement to continue programming and applying course learnings.
• Suggestions for future courses (e.g., software engineering, algorithms, machine learning).
• Closing remarks on computing's future:
  • Historical mispredictions about technology.
• Thank you for participation and engagement in the course.