Estimators for Mean & Variance in Probability
This content delves into the estimation of mean and variance in probability, covering topics such as point estimators, sample mean estimators, choices of estimators, and desirable properties of point estimators. It explains how estimators are used to infer parameters of underlying distributions from data samples. The discussion includes definitions, motivations, and considerations for selecting appropriate estimators in statistical analysis.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Chapter 15 Estimators for Mean & Variance "Introduction to Probability for Computing", Harchol-Balter '24 1
Probability vs. Statistics PROBABILITY generate Probabilistic Model Data infer STATISTICS This chapter: Given some data, infer the mean and variance of the underlying distribution. "Introduction to Probability for Computing", Harchol-Balter '24 2
Motivation for Estimation Call this unknown ? Goal: Estimate the mean height of women Obviously can t measure heights of all women, so sample heights of ? women, say ? = 5. Use ?????? ??????? as estimation of ?. 3 "Introduction to Probability for Computing", Harchol-Balter '24
Definition of Estimator e.g. ? = ? ? , where ? is unknown ?: quantity we re trying to estimate think of these as i.i.d.instances of ? This is a r.v. because it s a function of r.v.s ?1,?2, ,??: sampled data Sometimes just write ? ? ?1,?2, ,??: estimator of the unknown ? This is a constant ? ?1= ?1,?2= ?2, ,??= ??: estimation of ? based on specific instantiation of the data 4 "Introduction to Probability for Computing", Harchol-Balter '24
Sample Mean Estimator Defn 15.2: Let ?1,?2, ,?? be i.i.d. samples of r.v. ? with unknown mean ? = ? ? . The sample mean is a point estimator of ?: ? ?1= ?1,?2= ?2, ,??= ?? = Mn= ? =?1+ ?2+ + ?? ? Alternate name for sample mean Alternate name for sample mean 5 "Introduction to Probability for Computing", Harchol-Balter '24
Choices of estimators Let ?1,?2, ,?? be i.i.d. samples of r.v. ? with unknown finite mean ? = ? ? and unknown finite variance ?2= ??? ? . Two possible estimators for ?: ??= ? =?1+ ?2+ + ?? ? ??= ?2 Q: How do we choose between estimators? 6 "Introduction to Probability for Computing", Harchol-Balter '24
Desirable properties of a point estimator Defn 15.3: Let ? ?1,?2, ,??be a point estimator for ?. The bias of ? is: unbiased estimator is desirable property #1 ? ? = ? ? ? If ? ? = 0, we say that ? is and unbiased estimator of ?. Which of ??= ? and ??= ?2are unbiased estimators? Both! So why do we prefer ??? 7 "Introduction to Probability for Computing", Harchol-Balter '24
Desirable properties of a point estimator Defn 15.4: The mean squared error (MSE) of estimator ? ?1,?2, ,??is: Low MSE is desirable property #2 2 ??? ? = ? ? ? What metric does this remind you of? 8 "Introduction to Probability for Computing", Harchol-Balter '24
Desirable properties of a point estimator Defn 15.4: The mean squared error (MSE) of estimator ? ?1,?2, ,??is: Low MSE is desirable property #2 2 ??? ? = ? ? ? Lemma 15.5: If ? ?1,?2, ,??is an unbiased estimator, then ??? ? = ??? ? 2 2 Proof:??? ? = ? = ??? ? ? ? ? ? ? =? 9 "Introduction to Probability for Computing", Harchol-Balter '24
Desirable properties of a point estimator 2= ??? ? ??? ? = ? ? ? if ? ?1,?2, ,??is unbiased Q: Which of ??= ? and ??= ?2have lower MSE? ??is way better! Both ??= ? and ??= ?2are unbiased estimators. 1 ?2 ? ??? ?=??? ? ??? ?? = ??? ?? = ? ??? ?? = ??? ?? = ??? ?2 = ??? ? 10 "Introduction to Probability for Computing", Harchol-Balter '24
Desirable properties of a point estimator We also want our estimator to become more accurate as the sample size increases. Defn 15.6: Let ?1?1, ?2?1,?2, ?3?1,?2,?3, be a sequence of point estimators of ?, where ?? ?1,?2, ,?? is a function of ? i.i.d. samples. consistency is desirable property #3 We say that r.v. ?? is a consistent estimator of ? if, ? > 0, lim ? ? ?? ? ? = 0 Seems hard to prove that our estimator is consistent . Fortunately, there s a helpful lemma! 11 "Introduction to Probability for Computing", Harchol-Balter '24
Desirable properties of a point estimator ? ??? ?? = 0, then ?? is a consistent estimator. Lemma 15.7: If lim Proof: 2 ?? ? ?2 ? 2 ?2 ? ?? ? ? = ? ?? ? 2 ?? ? ?2 ??? ?? ?2 ? = ??? ?? ?2 ?? ? ? = lim lim ? ? = 0 ? 12 "Introduction to Probability for Computing", Harchol-Balter '24
Desirable properties of a point estimator ? ??? ?? = 0, then ?? is a consistent estimator. Lemma 15.7: If lim Q: Is ??= ?= ?? a consistent estimator of ? ? ? Assuming ??? ? finite ???! ??? ?? = ??? ?? =??? ? ? ??? ? ? ? ??? ?? = lim ? ??? ?? = lim lim = 0 ? ??is a consistent estimator 13 "Introduction to Probability for Computing", Harchol-Balter '24
An estimator for Variance Call this unknown ? Goal: Estimate the variance of height of women Obviously can t measure heights of all women, so sample heights of ? women, say ? = 5. ?:Can we use sample variance as an estimation of ?? ?: And how do we define sample variance ? 14 "Introduction to Probability for Computing", Harchol-Balter '24
An estimator for Variance Two cases: 1. The case where know?[?] and want to estimate ??? ? 2. The case where don t know ?[?] and want to estimate ??? ? 15 "Introduction to Probability for Computing", Harchol-Balter '24
Estimating Variance when Mean is Known Assume ? = ?[?] is known. Let ? be an estimator of ? = ??? ? : ? ? ?1,?2, ,?? = ?2 1 ?? ?2 ? ?=1 mean squared distance from mean sample variance 16 "Introduction to Probability for Computing", Harchol-Balter '24
Estimating Variance when Mean is Known Assume ? = ?[?] is known. Let ? be an estimator of ? = ??? ? : ? ? ?1,?2, ,?? = ?2 1 ?? ?2 ? ?=1 Is ?2an unbiased estimator? ???! ? ? ? ?2=1 =1 ? ?? ?2 ? ?=1 ? ?=1 ??? ?? = ??? ? 17 "Introduction to Probability for Computing", Harchol-Balter '24
Estimating Variance when Mean is UNknown How about we replace by ?? Assume ? = ?[?] is not known. Let ? be an estimator of ? = ??? ? : ? ? ?1,?2, ,?? = ?2 1 ?? ?2 ? ?=1 ? ? ?1,?2, ,?? = ?2 1 ?? ?2 ? ?=1 ? =?1+ ?2+ + ?? ? 18 "Introduction to Probability for Computing", Harchol-Balter '24
Estimating Variance when Mean is UNknown ? =?1+ ?2+ + ?? Assume ? = ?[?] is not known. Let ? be an estimator of ? = ??? ? : ? ? ? ?1,?2, ,?? = ?2 1 ?? ?2 ? ?=1 Q: Is ?2an unbiased estimator? So need to multiply ?2by ? 1 to get unbiased estimator ? A: Not quite. In Exercise 15.4, you will prove that: ? ?2=? 1 ? ??? ? 19 "Introduction to Probability for Computing", Harchol-Balter '24
Estimating Variance when Mean is UNknown Defn 15.8: Let ?1,?2,?3, be i.i.d. samples of r.v ? with unknown mean and variance. The sample variance is a point estimator of ? = ??? ? . It is denoted by ?2and defined by: ? ? 1 ? ?1,?2, ,?? = ?? ?2 ?2 ? 1?2= ? 1 ?=1 sample variance 20 "Introduction to Probability for Computing", Harchol-Balter '24
More complex estimators The sample mean estimator is often a component of more complex estimators. Consider the following example: Goal: Estimate # German tanks in WW II. Each tank has serial number. Below are the tanks we ve seen so far Mathematically, we re trying to estimate a maximum, call it ?, based on seeing ? samples, ?1,?2, ,??, each randomly picked without replacement, from the integers 1,2,3, ,?. 21 "Introduction to Probability for Computing", Harchol-Balter '24
Estimating the max Mathematically, we re trying to estimate a maximum, call it ?, based on seeing ? samples, ?1,?2, ,??, each randomly picked without replacement, from the integers 1,2,3, ,?. We know how to estimate the mean. Can we write the max in terms of the sample mean? 22 "Introduction to Probability for Computing", Harchol-Balter '24
Estimating the max Mathematically, we re trying to estimate a maximum, call it ?, based on seeing ? samples, ?1,?2, ,??, each randomly picked without replacement, from the integers 1,2,3, ,?. ? =?1+ ?2+ + ?? ? = ? + 1 ? ? =1 ? ?1+ ? ?2+ + ? ?? 2 ? ? ?? = 1 1 ?+ 2 1 ?+ 3 1 ?+ + ? 1 ?=? + 1 2 23 "Introduction to Probability for Computing", Harchol-Balter '24
Estimating the max Mathematically, we re trying to estimate a maximum, call it ?, based on seeing ? samples, ?1,?2, ,??, each randomly picked without replacement, from the integers 1,2,3, ,?. = ? + 1 ? ? =1 ? ?1+ ? ?2+ + ? ?? 2 ? Hence a reasonable estimator for ? is: ? ?1,?2, ,?? = 2 ? 1 24 "Introduction to Probability for Computing", Harchol-Balter '24
Estimating the max Mathematically, we re trying to estimate a maximum, call it ?, based on seeing ? samples, ?1,?2, ,??, each randomly picked without replacement, from the integers 1,2,3, ,?. = ? + 1 ? ? =1 ? ?1+ ? ?2+ + ? ?? 2 ? Is ? unbiased? Hence a reasonable estimator for ? is: Yes! ? ?1,?2, ,?? = 2 ? 1 ? ? = 2? ? 1 = 0 25 "Introduction to Probability for Computing", Harchol-Balter '24
Estimating the max Mathematically, we re trying to estimate a maximum, call it ?, based on seeing ? samples, ?1,?2, ,??, each randomly picked without replacement, from the integers 1,2,3, ,?. = ? + 1 ? ? =1 ? ?1+ ? ?2+ + ? ?? Is ? good? 2 ? Not necessarily. ? = 2 ? 1 could be smaller than some samples Hence a reasonable estimator for ? is: ? ?1,?2, ,?? = 2 ? 1 26 "Introduction to Probability for Computing", Harchol-Balter '24
