Module 5.4: Statistical and Clinical Significance

HLTH2024 Research Methods in Health

• Focus: Understanding statistical and clinical significance in research.
• Authors: Written by John Bidewell, spoken by Miriam Kannedy.

Key Concepts

Statistical vs Clinical Significance

• Statistical Significance:
  • Refers to the reliability of the effect observed in the study and its generalizability to the population.
  • Involves inferential statistics like confidence intervals and p-values.
• Clinical Significance:
  • Relates to the practical importance of a treatment effect in real-world settings.
  • Concerns whether the effect size is meaningful in a clinical context.

Effect Size

• Represents the difference between sample averages in intervention studies.
• Important in both statistical analysis and determining clinical relevance.

Interpreting Results

• Ability to interpret effect sizes, statistical significance, and apply both to evidence-based practice.

Evaluating Treatment Effects

Criteria for Successful Treatment

1. Size of Effect:
   • Must be large enough to be useful and justify the treatment.
   • Minimum useful effect size should be determined before trials.
2. Generalisability of Effect:
   • Treatment effect should be statistically significant to suggest it can be generalized.

Scenarios in Clinical Trials

• Scenario 1: Zero treatment effect in population; observed effect due to chance, non-significant test.
• Scenario 2: Large sample effect by chance; non-significant test protects against false conclusions.
• Scenario 3: Modest effect combining systematic and chance effects; significant but not clinically useful.
• Scenario 4: Useful treatment effect observed; significant test confirms usefulness.
• Scenario 5: Large effect in sample but non-significant; real-world uncertainty in inferential tests.

Importance of Sample Size

• Larger samples can lead to statistically significant results even for small effects.
• Clinically important significance must be decided by clinicians.
• Adjust sample size to align statistical significance with clinical importance.

Graph Interpretation

• Be wary of different scales on graphs; can mislead interpretation of effect sizes.

Summary

• Treatments should demonstrate both a clinically worthwhile effect size and statistical significance.
• Both conditions must be met for robust, evidence-based treatment conclusions.
• The same evaluation logic applies to other effect size measures (relative risks, odds ratios).

Remember: Clinicians and statisticians must work together to interpret the data effectively for practical application in healthcare.