Lecture Notes: Method of Moments in Estimators

Recap of Previous Lecture

• Discussed characteristics of estimators:
  • Bias
  • Variance
  • Squared error risk (mean squared error)
• Explored how to calculate these using probability and properties of IID samples.

Introduction to Design of Estimators

• Focus on the design of estimators using the Method of Moments.

Understanding Parameters and Moments

• Given a random variable X distributed according to a function (pdf/pmf), parameters (e.g., θ₁, θ₂) are unknown.
• Moments are calculated from the distribution:
  • Expected value (E[X]), E[X²], etc.
• Examples of distributions and their moments:
  • Bernoulli(p): E[X] = p
  • Poisson(λ): E[X] = λ
  • Exponential(λ): E[X] = 1/λ
  • Normal(μ, σ²): E[X] = μ, E[X²] = μ² + σ²
  • Gamma(α, β): E[X] = α/β, E[X²] = α²/β² + α/β²

Sample Moments

• Given n IID samples, the sample moment Mₖ is:
  • Mₖ = (1/n) Σ(Xᵢᵏ) for k = 1, 2, ...
• Sample moments are random variables and vary with different samples.
• Central Limit Theorem and Weak Law of Large Numbers justify similarity between sample moments and distribution moments.

Method of Moments Procedure

• Equate sample moments to distribution moments:
  • Solve for unknown parameters.
  • Usually, one moment is enough for one unknown parameter.
  • If the moment isn’t a function of the parameter, find another moment.

Examples of Method of Moments

1. Bernoulli Distribution (p):

   • E[X] = p, thus p = M₁.
   • Estimator: p̂ = M₁ (sample mean).

2. Poisson Distribution (λ):

   • E[X] = λ, thus λ = M₁.
   • Estimator: λ̂ = M₁ (sample mean).

3. Exponential Distribution (λ):

   • E[X] = 1/λ, thus 1/λ = M₁ → λ = 1/M₁.
   • Estimator: λ̂ = 1/M₁.

4. Normal Distribution (μ, σ²):

   • E[X] = μ, E[X²] = μ² + σ².
   • System of equations:
     • μ = M₁
     • σ² = M₂ - M₁².
   • Estimators: μ̂ = M₁, σ̂ = √(M₂ - M₁²).

5. Gamma Distribution (α, β):

   • E[X] = α/β, E[X²] = α²/β² + α/β².
   • Equations:
     • α/β = M₁
     • α²/β² + α/β = M₂.
   • Solve for α and β:
     • Express β in terms of α and substitute.
   • Estimators: α̂ = (M₁²)/(M₂ - M₁²), β̂ = (M₁)/(M₂ - M₁²).

6. Binomial Distribution (n, p):

   • E[X] = np, E[X²] = n²p² + np(1 - p).
   • Solve for n and p using sample moments M₁ and M₂.

Practical Applications

• Calculating estimators from samples:
  • Example for Bernoulli: Count successes in samples.
  • Example for Poisson: Count events over intervals.
  • Example for Normal: Compute mean and variance from samples.

Conclusion

• Method of Moments is a straightforward way to estimate parameters using sample data.
• Next lecture will cover Maximum Likelihood Estimation.