Discussion on Noise in Judgment

Jul 5, 2024

Lecture Notes: Intelligence Squar - Discussion on Noise in Judgment

Welcome and Introduction

  • Event held at Union Chapel
  • Hosted by Intelligence Squar
  • Audience present both in-person and online
  • Theme: Examining human judgment and decision-making accuracy
  • Key Speakers: Daniel Kahneman, Olivier Sibony
    • Kahneman: Nobel Prize winner, author of Thinking, Fast and Slow, Professor at Princeton University
    • Sibony: Professor of Strategy, co-author of Noise, former senior partner at McKenzie

Concept of Noise

Definition of Noise

  • Judgment noise: Variability in human judgments that should be identical but are not
  • Common in all areas involving human judgment
  • **Distinction from bias: **
    • Bias: Systematic error, average error
    • Noise: Variability that should not exist but does

Examples and Impact of Noise

  • Insurance: Underwriters offered inconsistent quotes (as much as 50% variability)
  • Criminal Justice: Variability in sentencing for identical cases; disparity found in U.S. study
    • Impact: Unfairness in justice as similar cases receive different verdicts
  • Medical Field: Differences in diagnoses/treatments
    • Example: Epilepsy diagnosis from EEG recordings showed high variability

Causes of Noise

Three Main Sources

  1. Level Noise: Consistent differences in judgments (severe vs lenient judges)
  2. Occasion Noise: Variability due to mood, weather, or time of day
  3. Pattern Noise: Differences due to personal biases and experiences affecting judgments
  • Example: Judges having different sensitivities based on personal backgrounds

Addressing Noise

  • Decision Hygiene: Processes to reduce noise in human judgment
    • **Types of Decision Hygiene: **
      • Aggregating independent opinions
      • Structuring decisions
      • Using relative judgments
    • Used primarily by organizations rather than individuals

Algorithms vs Human Judgment

  • Advantages of Algorithms: Consistent, Noise-free, less variability
  • Bias Concerns: Reflect biases in training data but more detectable without noise
    • Examples of poor algorithm design replicating human biases (e.g., hiring algorithms)
  • Conclusion: Algorithms have potential to improve decision accuracy but require careful construction to avoid biases

Expertise

  • Performance-Based Expertise: Verified through outcomes (e.g., chess players, weather forecasters)
  • Respected Experts: Based on peer respect, subjective measures (e.g., expert astrologers)

Challenges and Organizational Resistance

  • **Challenges: **
    • Noise perceived as abstract; difficult to identify and address
    • Resistance due to fear of bureaucracy or over-strict procedures
  • Human Agency: Individuals prefer discretion even if it leads to less efficiency

Summary and Final Thoughts

  • Organizations need to design processes to improve decision-making accuracy
  • Awareness of noise and implementation of decision hygiene can reduce variability and improve outcomes despite inherent resistance

Closing Remarks: Importance of independent opinions and structured decision processes to minimize noise in judgments. Emphasis on organizational strategies to mitigate noise rather than individual self-help.