Estimators: Method of Moments vs MLE

Aug 7, 2024

Lecture Notes: Method of Moments and Maximum Likelihood Estimators

Overview

  • Discussing examples of deriving Method of Moments Estimators (MME) and Maximum Likelihood Estimators (MLE) for different distributions.
  • Comparing and contrasting MME and MLE.

Exponential Distribution

  • PDF: (f(x) = \lambda e^{-\lambda x}) for (x > 0).

Method of Moments Estimator (MME)

  • ( \hat{\lambda}_{MME} = \frac{n}{x_1 + x_n} )

Maximum Likelihood Estimator (MLE)

  • Likelihood Function: (L(\lambda) = \prod_{i=1}^{n} \lambda e^{-\lambda x_i} = \lambda^n e^{-\lambda (x_1 + ... + x_n)})
  • Log-Likelihood: ( \log L(\lambda) = n \log \lambda - \lambda (x_1 + ... + x_n) )
  • Differentiate and set to 0: ( \frac{n}{\lambda} - (x_1 + x_n) = 0 )
  • Result: ( \hat{\lambda}_{MLE} = \frac{n}{x_1 + x_n} )
  • Both MME and MLE are the same in this case.

Discrete Case: Uniform Distribution (1, 2, 3)

  • Probability: (p_3 = 1 - p_1 - p_2)

Method of Moments Estimator (MME)

  • Moment equations:
    • First Moment: ( m_1 = p_1 + 2p_2 + 3p_3 )
    • Second Moment: ( m_2 = p_1 + 4p_2 + 9p_3 )
  • Substitute (p_3) and simplify to derive (p_1) and (p_2).
  • Results:
    • ( \hat{p}_1 = \frac{6 - 5m_1 + m_2}{2} )
    • ( \hat{p}_2 = 4m_1 - m_2 - 3 )

Maximum Likelihood Estimator (MLE)

  • Likelihood Function: (L(p_1, p_2) = p_1^{w_1} p_2^{w_2} (1 - p_1 - p_2)^{n - w_1 - w_2})
  • Results:
    • ( \hat{p}_1 = \frac{w_1}{n} )
    • ( \hat{p}_2 = \frac{w_2}{n} )
  • MLE gives simpler, more intuitive results compared to MME.

Uniform Distribution (0, (\theta))

  • PDF: (f(x) = \frac{1}{\theta}) for (0 < x < \theta).

Method of Moments Estimator (MME)

  • (E[X] = \frac{\theta}{2}) leads to (\hat{\theta}_{MME} = 2 \bar{x}).

Maximum Likelihood Estimator (MLE)

  • MLE requires (\theta \geq max(x_1,...,x_n)) leading to:
  • Result: (\hat{\theta}_{MLE} = max(x_1,...,x_n))
  • The two estimators differ significantly in this case.

Discrete Uniform Distribution (1, 2, ..., n)

  • MME results in: (\hat{n}_{MME} = 2 \bar{x} - 1)
  • MLE results in: (\hat{n}_{MLE} = max(x_1,...,x_n))

Complex Cases: Gamma and Binomial Distributions

  • Gamma Distribution MLE involves more complexity without closed-form solutions.
  • Binomial MLE becomes complicated when both (n) and (p) are unknown.
  • Likelihood Functions must be simplified and solved numerically.

Key Takeaways

  • MME provides straightforward solutions while MLE can yield more intuitive results.
  • Always verify consistency and realism of estimators derived from MME and MLE.
  • Use numerical methods for complex distributions when closed forms are not available.

These notes summarize the derivations and comparisons of estimators discussed in the lecture, structured to assist in understanding the methodologies and results.

Full transcript