Estimation of the Population Mean

1 Estimation of the Population Mean • As mentioned previously, a key theme of this course is that we want to say something about the population using a sample of that population. Now, we will introduce statistical methods that we can apply using the sample to reach conclusions about population. These statistical methods are: (1) estimation, (2) hypothesis testing, and (3) confidence intervals. We will use these three approaches throughout the course. • Suppose you want to know the population mean of Y (i.e., the value µY ). How can we use the sample to estimate this population parameter? One way is to use the sample mean Y . Another way is to just use the first observation Y1 . Still another way is to consider the weighted average in which the observations are alternately weighted by 1/2 and 3/2, namely, Ỹ = n1 ( 12 Y1 + 32 Y2 +. . .+ 12 Yn−1 + 32 Yn ). All of these (i.e., Y , Y1 , and Ỹ ) are called estimators. • An estimator is a procedure for using sample data to compute an educated guess of the value of a population parameter (such as the population mean). Note that estimators are random variables. Can you explain why? • A “good” estimator of the population mean should have the following three properties: ◦ Unbiasedness. Let µ̂Y denote some estimator of µY . Mathematically, unbiasedness is stated as E(µ̂Y ) = µY . In words, unbiasedness implies that the mean of the estimator must equal the true population parameter. For example, if Ỹ is an unbiased estimator for the population mean, then E(Ỹ ) = µY . ◦ Consistency. When the sample size is large, the uncertainty about the value of µY arising from random variations in the sample is very small. Stated more precisely, consistency means that the probability that µ̂Y is within a small interval of the true value of µY approaches one p as the sample size increases. Mathematically, µ̂Y −→ µY . ◦ Efficiency. Efficiency entails a comparison of estimators. Suppose you had two candidate estimators, both of which are unbiased. If the first estimator has a smaller variance than the second estimator, then the first is said to be more efficient. • Consider again the sample mean Y . Let’s see if this estimator satisfies the above three properties. ◦ It is unbiased because E(Y ) = µY . Can you show this on your own? ◦ It is consistent by the law of large numbers. ◦ What about efficiency? Note that efficiency is based on comparing the variance across estimators. You can show that var(Y ) is smaller than the var(Y1 ) and var(Ỹ ). Hence, Y is more efficient than the other two estimators. 1 • In fact, Y is the Best Linear Unbiased Estimator (BLUE) for the population mean. BLUE means that out of all the estimators that are unbiased and are linear functions of Y1 , . . . , Yn , Y has the smallest variance. In addition, Y is the least squares estimator of µY . We will return to these points later on in the course. • The table below summarizes the various unknown parameters related to Y that we will encounter later (see Section 3 of these notes) and the estimators that we use for them. Note that as usual we have Y1 , . . . , Yn i.i.d. with mean µY and variance σY2 . Notation What it represents Estimator µY E(Yi ) Y = σY2 var(Yi ) s2Y (sample variance) 1 n Pn i=1 Note: sY = σY √ n sY √ n sd(Y ) Yi (sample average) 1 n−1 Pn i=1 (Yi − Y )2 (standard error of Y ) Note that µY and σY2 are characteristics of the population distribution (i.e., the distribution from σY which we drew the sample Y1 , . . . , Yn ), while √ is a characteristic of the sampling distribution n (i.e., the distribution of the sample statistic Y ). The standard error of Y is used as estimator for the standard deviation of Y if σY is unknown. 2 Basic Concepts of Hypothesis Testing • Hypothesis testing refers to a set of procedures used to weigh the “strength” of statistical evidence. In particular, we use these tests to determine whether the evidence from our sample is “sufficiently strong” to support one hypothesis over another. ◦ For example, suppose you were interested in determining whether a coin is fair or biased. If you toss the coin 100 times and find 51 heads, you might conclude that the coin is probably fair. If you find 90 heads, that would provide much more evidence that the coin is probably biased. But if you find 58 heads, what would you conclude? What is the threshold that we need before we can conclude that the coin is probably biased? Hypothesis testing helps us to figure that out. • The approach requires a null hypothesis H0 and an alternative hypothesis H1 . ◦ The null hypothesis H0 is the starting point of statistical hypothesis testing. ◦ The alternative hypothesis H1 is the hypothesis that contradicts the assertion of the null hypothesis. ◦ An important feature to note about hypothesis testing is that H0 and H1 are not treated in the same way: H0 is given the benefit of the doubt, and H1 has the burden of proof. This framework is analogous to the presumption of innocence in the law: the defendant is presumed innocent until proven guilty beyond a reasonable doubt. Similarly, H0 is presumed to be true unless the data strongly suggest otherwise. • A hypothesis test can be either a one-sided or a two-sided test. ◦ A one-sided test involves > and < in the alternative hypothesis, e.g., H1 : µ > µ0 or H1 : µ < µ0 2 ◦ A two-sided test involves 6= in the alternative hypothesis, e.g., H1 : µ 6= µ0 • The conclusion of a hypothesis test is either: (1) we reject H0 , or (2) we fail to reject H0 . This wording is important for the following reasons. ◦ When we conduct a hypothesis test, we are NOT testing whether H0 or H1 is true; rather, we are asking whether the available data is significant evidence to cast doubt on H0 in favor of H1 . ◦ Rejecting H0 means that the data lend support to H1 . It does NOT mean that we have proven H1 is true; rather, it means that the data from our sample are inconsistent with H0 , so the evidence in our data favor H1 . ◦ Failing to reject H0 means that the data lend support to H0 . It does does NOT mean that we have proven H0 is true; rather it means that the data are consistent with H0 , so they lend support to H0 . Lending support is not the same as proving H0 ; this is why we say “fail to reject H0 ” instead of “accept H0 .” • Because every hypothesis test concludes by either rejecting or failing to reject the null, there are 2 ways of drawing incorrect conclusions. ◦ Type I error: rejecting H0 when H0 is true (false positive). ◦ Type II error: sticking with H0 when H1 is true (false negative). ◦ Ideally, we would like the probability of both types of errors to be small. However, in practice, we cannot simultaneously reduce the chances of both Type I and Type II errors. For example, if we make it harder to reject H0 , then we are less likely to commit Type I error, but more likely to commit a Type II error. • How do we conclude whether we reject or fail to reject the null hypothesis? We can use 3 methods: (1) t-statistic, (2) p-value, and (3) confidence intervals. All methods will arrive at the same conclusion. The next section discusses the procedures in more detail. • Why do we even care about hypothesis tests? Let’s tie this question in with what we’ve learned so far in this class. ◦ We want to answer questions about the population. ◦ But collecting data from the entire population is infeasible or impractical, we take a simple random sample (which as you recall, results in i.i.d observations). ◦ Using data from the sample, we can calculate a sample statistic, e.g., the sample average Y . ◦ But because there is only one population but many possible samples, the value of the sample statistic we obtain would differ from sample to sample. This variability is called sampling variation. ◦ Sampling variation also implies that even if we observed a sample average of, say, Y = 145 in our sample, this does not mean that the population average height µ is equal to 145 as well. ◦ Hence, we need to use statistical methods, in this case hypothesis testing, so that we can reach a conclusion about the population average µ. 3 3 Two-Sided Hypothesis Tests In this class, we will be using hypothesis tests for means and differences of means. For the remainder of these notes, I will focus on two-sided hypothesis tests, and please refer to the textbook for one-sided hypothesis tests. I will also assume that we have a large sample size, so that by the Central Limit Theorem, we can use the normal distribution. The following table summarizes the notation that we use and the different characteristics we need to know to be able to carry out a hypothesis test. Mean Population parameter Differences of Means µ µ1 , µ2 Note: The subscript 1 and 2 refer to groups, e.g., 1 could be men, and 2 could be women. Sample statistic Y1−Y2 Y Note: The order of subtraction follows the same order of the parameters in the null hypothesis. Standard Error (SE) of sample statistic t-statistic √ sY / n Note: sY = t= q q 1 n−1 Pn i=1 (Yi − Y )2 Y −µY,0 SE(Y ) s21 n1 s22 n2 Note: This assumes that sample group 1 and sample group 2 are from independent populations. t= Note: µY,0 is the hypothesized value in the null hypothesis, e.g., if H0 : µ = 4, then µY,0 = 4 + X 1 −X 2 −d0 SE(X 1 −X 2 ) Note: d0 is the hypothesized value in the null hypothesis Note that the population parameter is usually what we want to say something about but it is unknown. The sample statistic is what we observe in the data. 3.1 Two-Sided Hypothesis Test Using the t-statistic The t-statistic is the number of standard errors that separate the sample statistic (e.g., Y , or Y 1 − Y 2 ) from the hypothesized value in the null hypothesis (e.g., µY,0 or d0 ). Here is a general “recipe” for conducting a hypothesis test using the t-statistic. You do not have to use this recipe, but I provide it here in case it is helpful. • Step 1: Choose a level of significance α. This is our tolerance for a Type I error. Note: This α might also just be given to you. • Step 2: Formulate the null and alternative hypothesis. ◦ Important: The null/alternative hypothesis involve the population parameter µ, NOT the sample statistic Y . • Step 3: Calculate the t-statistic (see table above for the formulas). ◦ Note that the t-stat has the following format: sample statistic − value from hypothesis se(sample statistic) • Step 4: Find the “two-sided critical value” in the sampling distribution that corresponds to α. Assuming that our sample size is large enough, we can use the normal table to find this critical value. 4 ◦ The two-sided critical value is the number zα/2 from the normal table such that P (|Z| > zα/2 ) = α. ◦ In words, zα/2 is basically a positive number z > 0 such that the area to the right of z is α/2, and the area to the left if −z is α/2 (so the total sum of the two areas is α). • Step 5: Compare the t-statistic (from Step 3) with the two-sided critical value (from Step 4), and apply the following rejection rule. ◦ Reject H0 if |t-stat| > two-sided critical value. Otherwise, we fail to reject H0 . ◦ If we reject H0 , for example with α = 0.05, then we say that our result is statistically significant at the 5% level. ◦ Note: The rejection rule for the two-sided test takes the absolute value of the t-stat. This is because for a two-sided test, we are looking for evidence against the null in both the left and right tails, i.e., either t-stat < −1 ∗ two-sided critical value or t-stat > two-sided critical value. Taking the absolute value covers both cases. Tip: I would recommend memorizing the two-sided critical values from the normal table for α levels 1%, 5% 10%, since these are the most common values of α when carrying out a hypothesis test. • α = 0.10: z0.10/2 = 1.64 • α = 0.05: z0.05/2 = 1.96 • α = 0.01: z0.01/2 = 2.58 3.2 Two-Sided Hypothesis Test Using the p-value The p-value is the probability of drawing a value of the sample statistic that is at least as extreme as what we observe in our sample, if the null hypothesis is true. It is also the smallest α level at which H0 can be rejected.We can think of the p-value as the weight of evidence against H0 , that is, small p-values mean that H0 is not plausible. Here is a general “recipe” for conducting a hypothesis test using the p-value. You do not have to use this recipe, but I provide it here in case it is helpful. • Step 1 to Step 3: Follow the same steps as in the previous section. Make a note of the t-statistic that you calculated in Step 3, since you will be needing this in the next step. • Step 4: Find the following probabilities using the normal table, which give us the p-value. ◦ p-value = P (|Z| > |t-stat|) = 2 · P (Z > |t-stat|) = 2 · P (Z < −|t-stat|). In words, this means that we need to take the absolute value of the t-stat, find the area to the right of the absolute value of the t-stat, or equivalently the area to the left of −|t-stat|, then multiply that area by 2 to find the p-value. • Step 5: Compare the p-value obtained from Step 4 with the level of α from Step 1. We reject H0 if p-value < α. Otherwise, we fail to reject H0 . ◦ A potentially useful phrase to help you remember the rejection rule: “If the p-value is low, the null has to go.” ◦ If we reject H0 , for example with α = 0.05, then we say that our result is statistically significant at the 5%. 3.3 Two-Sided Hypothesis Test Using the Confidence Interval A confidence interval (CI) is a range of plausible values of a parameter (e.g., µ) based on a sample from the population (e.g., Y ). Here is a general “recipe” for conducting a hypothesis test using the confidence interval. You do not have to use this recipe, but I provide it here in case it is helpful. 5 • Step 1 to Step 3. As before, choose significance level α, formulate the null and alternative hypotheses, and find the two-sided critical value. The important ingredient you will need here is the two-sided critical value. • Step 4. Construct the (100 − α)% confidence interval using the formula: statistic) sample statistic ± critical {z value} ∗ SE(sample | {z } | | {z } e.g., Y or Y 1 − Y 2 zα/2 see formula in table above • Step 5. Check if the hypothesized value (e.g. µY,0 , d0 ) is contained in the CI you constructed in Step 4, then apply the following rejection rule: ◦ If the hypothesized value is outside the CI, we reject H0 . ◦ Otherwise, we fail to reject H0 . 3.4 Equivalent Inferences for t-statistic, p-value, and Confidence Interval • As mentioned above, all three methods will reach the same conclusion. • The difference between the three lies in the “types” of information they contain. ◦ The hypothesis test using t-stat tells us whether we reject or fail to reject only one particular hypothesized value. ◦ The p-value gives us the smallest significance level α for which we can reject the null. ◦ The CI gives a list of all hypothesized values we would reject (i.e., those outside the CI) or fail to reject (i.e., those within the CI). 6