Overview
This lecture covers quasi-experimental designs, their limitations compared to true experiments, common pitfalls, and recommended approaches for social science research where randomization is not possible.
Introduction to Quasi-Experiments
- Quasi-experiments aim to approximate experiments when random assignment isn't feasible.
- They maintain experimental structure (groups, treatments, outcomes) but lack random allocation of participants.
- The absence of randomization weakens claims about causality.
Designs to Avoid
- One Group Post-Test Only: Only one group receives treatment; outcome observed—cannot attribute effect to treatment.
- Non-Randomized Two Groups: Two groups (treatment and control) but no random assignment; group differences can confound results.
- One Group Pre-Test/Post-Test: Measurements before and after treatment on the same group; cannot rule out alternative explanations (testing effects, maturation, etc.).
- Regression to the Mean Issues: Selecting groups based on extreme pretest scores leads to artificial changes on retest, not due to treatment.
Recommended Quasi-Experimental Designs
- Pretest-Posttest Non-Equivalent Groups Design: Both a treatment and a comparison group measured before and after treatment; helps rule out some threats but selection bias remains.
- Stronger evidence if the treatment group improves more than the control group, especially from a lower starting point.
- Interrupted Time Series Design: Multiple observations before and after treatment; a sudden or systematic change after treatment suggests causality.
- Must inspect for trends that predate treatment or continue regardless of treatment.
- Regression Discontinuity Design: Groups are assigned to treatment based on a cutoff score; if treatment group shows a jump in outcomes at the threshold, it supports treatment effect.
Examples of Natural Experiments
- Media coverage and scientific impact studied via a newspaper strike (papers written but not published, creating control scenario).
- Opium production in Afghanistan: Taliban takeover led to a sharp reduction—analyzed as a natural "intervention" using repeated measures data.
Key Terms & Definitions
- Quasi-Experiment — Study mimicking an experiment without random assignment.
- Randomization — Assigning participants to groups by chance to ensure equivalence.
- Selection Bias — Systematic group differences affecting outcomes beyond treatment effects.
- Pretest-Posttest Design — Measurements before and after a treatment.
- Interrupted Time Series — Multiple observations around a treatment/intervention.
- Regression to the Mean — Extreme values tend to move closer to average on retesting.
- Regression Discontinuity — Assignment to treatment based on a cutoff; discontinuity in outcome suggests effect.
Action Items / Next Steps
- Review the handout on threats to internal validity distributed last week.
- Avoid weak quasi-experimental designs in future projects.
- Inspect research designs for possible validity threats before drawing causal conclusions.