15.075 Notes, Sampling Distributions of Statistics

Central Limit Theorem (Convergence of the sample mean’s distribution to the normal distribution) Let X1 , X2 , . . . , Xn be a random sample drawn from any distribution with a finite mean µ and variance σ 2 . As n → ∞, the distribution of: X̄ − µ √ σ/ n “converges” to the distribution N (0, 1). In other words, X̄ − µ √ ≈ N (0, 1). σ/ n X̄−µ √ ? Remember that we proved that E(X̄) = µ and Var(X̄) = σ 2 /n. Note 1: What is σ/ n ¯ subtracting its mean, and dividing by its That means we are taking the random variable X, standard deviation. It’s a z-score! Note 2: “converge” means “convergence in distribution:” lim P n→∞ X̄ − µ √ ≤z σ/ n = Φ(z) for all z. Don’t worry about this if you don’t understand (it’s beyond the scope of 15.075). Note 3: CLT is really useful because it characterizes large samples from any distribution. As long as you have a lot of independent samples (from any distribution), then the distribu­ tion of the sample mean is approximately normal. Let’s demonstrate the CLT. Pick n large. Draw n observations from U [0, 1] (or whatever distribution you like). Repeat 1000 times. t=1 : : : t = 1000 x 1 x2 x3 .21 .76 .57 xn .84 x̄ = ni=1 xi (.21+.76+. . . )/n Then, histogram the values in the rightmost column and it looks normal. 1 CLTdemo n=z; myrand=rand(n,500); mymeans= m ean[myrand); hist(mymeans) BO 70 m 40 n=2 n= 3 60 4D 0 n= 30 2 Sampling Distribution of the Sample Variance - Chi-Square Distribution From the central limit theorem (CLT), we know that the distribution of the sample mean is approximately normal. What about the sample variance? Unfortunately there is no CLT analog for variance... But there is an important special case, which is when X1 , X2 , . . . , Xn are from a normal distribution. (Recall that the CLT applies to arbitrary distributions.) If this is true, the distribution of the sample variance is related to the Chi-Square (χ2 ) dis­ tribution. Let Z1 , Z2 , . . . , Zν be N (0, 1) r.v.’s and let X = Z12 + Z22 + · · · + Zν2 . Then the pdf of X can be shown to be: 1 f (x) = ν/2 xν/2−1 e−x/2 for x ≥ 0. 2 Γ(ν/2) This is the χ2 distribution with ν degrees of freedom (ν adjustable quantities). (Note: the χ2 distribution is a special case of the Gamma distribution with parameters λ = 1/2 and r = ν/2.) Fact proved in book: If X1 , X2 , . . . , Xn are iid N (µ, σ) r.v.’s, then (n − 1) 2 S ∼ χ2n−1 . σ2 That is, the sample variance times a constant (n−1) σ2 has a χ2n−1 distribution. Technical Note: we lost a degree of freedom when we used the sample mean rather than the true mean. In other words, fixing n − 1 quantities completely determines s2 , since: s2 := 1 n−1 (xi − x) ¯ 2. i Let’s simulate a χ2n−1 distribution for n = 3. Draw 3 samples from N (0, 1). Repeat 1000 times. t=1 : : : t = 1000 z1 -0.3 z2 -1.1 z3 0.2 Then, histogram the values in the rightmost column. 3 3 2 i=1 zi 1.34 For the chi-square distribution, it turns out that the mean and variance are: E(χ2ν ) = ν Var(χ2ν ) = 2ν. We can use this to get the mean and variance of S 2 : � 2 2 � σ χn−1 σ2 2 E(S ) = E = (n − 1) = σ 2 , n−1 n−1 � 2 2 � σ χn−1 σ4 σ4 2σ 4 2 2 Var(S ) = Var = Var(χ ) = 2(n − 1) = . n−1 (n − 1)2 n−1 n−1 (n − 1)2 So we can well estimate S 2 when n is large, since Var(S) is small when n is large. Remember, the χ2 distribution characterizes normal r.v. with known variance. You need to know σ! Look below, you can’t get the distribution for S 2 unless you know σ. X1 , X2 , . . . , Xn ∼ N (µ, σ 2 ) 4 → (n − 1)S 2 ∼ χ2n−1 σ2 Student’s t-Distribution Let X1 , X2 , . . . , Xn ∼ N (µ, σ 2 ). William Sealy Gosset aka “Student” (1876-1937) was looking at the distribution of: X̄ − µ √ T = S/ n ¯ X−µ √ which we know is N (0, 1). Contrast T with σ/ n So why was Student looking at this? Because he had a small sample, he didn’t know the variance of the distribution and couldn’t estimate it well, and he wanted to determine how far x̄ was from µ. We are in the case of: • N (0, 1) r.v.’s ¯ to µ • comparing X • unknown variance σ 2 • small sample size (otherwise we can estimate σ 2 very well by s2 .) Rewrite X̄ − µ √ = T = S/ n X̄−µ √ σ/ n √ 2 1√ √S n σ/ n =_ Z S 2 /σ 2 . The numerator Z is N (0, 1), and the denominator is sort of the square root of a chi-square, because remember S 2 (n − 1)/σ 2 ∼ χ2n−1 . Note that when n is large, S 2 /σ 2 → 1 so the T-distribution → N (0, 1). Student showed that the pdf of T is: �   �−(ν+1)/2 Γ ν+1 t2 2 f (t) = √ 1+ −∞ χ2ν,α ) = α ↑ ↑ P (Tν > tν,α ) = α ↑ ↑ P (Fν1 ,ν2 > fν1 ,ν2 ,α ) = α ↑ ↑ Note that because F is a ratio, Fν1 ,ν2 = 1 Fν2 ,ν1 which you might need to use in order to look up the F-scores in a table in the book. Actually, you will need to know: 1 . fν1 ,ν2 ,1−α = fν2 ,ν1 ,α 6 7