Significance Testing for Website Background Color Change

Objective

• Current Situation: Website has an off-white background, average user time is 20 minutes.
• Goal: Increase user time by changing the background color to yellow.

Steps in Significance Testing

1. Set Hypotheses

• Null Hypothesis (H0): The mean user time remains 20 minutes after the change.
• Alternative Hypothesis (H1): The mean user time is greater than 20 minutes after the change.

2. Set Significance Level

• Significance Level ((\alpha)): Typically set at 1%, 5%, or 10%.
• Chosen Level for Example: 0.05 (5%).

3. Take a Sample

• Sample Size (n): 100 users.
• Sample Mean: Calculate the mean time spent on the site by the sample.
• Sample Standard Deviation: Calculate if population standard deviation is unknown.

4. Calculate P-value

• P-value Definition: Probability of obtaining a sample mean as extreme as the observed one, assuming the null hypothesis is true.
• Conditional Probability: ( P(\text{Sample Mean} \geq 25 \text{ minutes} | H0 \text{ is true}) ).
• Tools: Use t-statistic if the sampling distribution is roughly normal.

5. Decision Rule

• If P-value < (\alpha): Reject the null hypothesis.
• If P-value (\geq) (\alpha): Do not reject the null hypothesis.

Example Calculation

• Observed Sample Mean: 25 minutes.
• P-value: Calculated from the sample data.
• Scenario 1: P-value = 0.03
  • Since 0.03 < 0.05, reject the null hypothesis. Have evidence that the mean user time increased.
• Scenario 2: P-value = 0.50
  • Since 0.50 > 0.05, cannot reject the null hypothesis. No evidence that the mean user time increased.

Important Clarifications

• P-value Interpretation: Probability of getting the sample statistics given that the null hypothesis is true.
• Common Confusion: It is NOT the probability that the null hypothesis is true given the sample data.