18.05 S22 Class 23: Confidence Intervals: Three Views

Confidence Intervals: Three Views Class 23, 18.05 1 Learning Goals 1. Be able to find 𝑧, 𝑡 and 𝜒2 confidence intervals using the corresponding standardized statistics. 2. Be able to use a hypothesis test to find a confidence interval for an unknown parameter. 3. Refuse to answer questions that ask, in essence, ‘given a confidence interval what is the probability or odds that it contains the true value of the unknown parameter?’ 2 Introduction Our approach to confidence intervals in the previous reading was a combination of standardized statistics and hypothesis testing. Today we will consider each of these perspectives separately, as well as introduce a third formal viewpoint. Each provides its own insight. 1. Standardized statistic. Most confidence intervals are based on standardized statistics with known distributions like 𝑧, 𝑡 or 𝜒2 . This provides a straightforward way to construct and interpret confidence intervals as a point estimate plus or minus some error. 2. Hypothesis testing. Confidence intervals may also be constructed from hypothesis tests. In cases where we don’t have a standardized statistic this method will still work. It agrees with the standardized statistic approach in cases where they both apply. This view connects the notions of significance level 𝛼 for hypothesis testing and confidence level 1 − 𝛼 for confidence intervals; we will see that in both cases 𝛼 is the probability of making a ‘type 1’ error. This gives some insight into the use of the word confidence. This view also helps to emphasize the frequentist nature of confidence intervals. 3. Formal. The formal definition of confidence intervals is perfectly precise and general. In a mathematical sense it gives insight into the inner workings of confidence intervals. However, because it is so general it sometimes leads to confidence intervals without useful properties. We will not dwell on this approach. We offer it mainly for those who are interested. 3 Confidence intervals via standardized statistics The strategy here is essentially the same as in the previous reading. Assuming normal data we have what we called standardized statistics like the standardized mean, Studentized mean, and standardized variance. These statistics have well known distributions which depend on hypothesized values of 𝜇 and 𝜎. We then use algebra to produce confidence intervals for 𝜇 or 𝜎. 1 18.05 Class 23, Confidence Intervals: Three Views, Spring 2022 2 Don’t let the algebraic details distract you from the essentially simple idea underlying confidence intervals: we start with a standardized statistic (e.g., 𝑧, 𝑡 or 𝜒2 ) and use some algebra to get an interval that depends only on the data and known parameters. 3.1 z-confidence intervals for the mean: normal data with known standard deviation 𝑧-confidence intervals for the mean of normal data are based on the standardized mean, i.e. the 𝑧-statistic. We start with 𝑛 independent normal samples 𝑥1 , 𝑥2 , … , 𝑥𝑛 ∼ N(𝜇, 𝜎2 ). We assume that 𝜇 is the unknown parameter of interest and 𝜎 is known. Notationally, let’s write the (unknown) true value of 𝜇 as 𝜇0 We know that the standardized mean is standard normal: 𝑧= 𝑥 − 𝜇0 √ ∼ N(0, 1). 𝜎/ 𝑛 For the standard normal critical value 𝑧𝛼/2 we have: 𝑃 (−𝑧𝛼/2 < 𝑍 < 𝑧𝛼/2 ) = 1 − 𝛼. Thus, 𝑃 (−𝑧𝛼/2 < 𝑥−𝜇 √ < 𝑧𝛼/2 | 𝜇 = 𝜇0 ) = 1 − 𝛼 𝜎/ 𝑛 A little bit of algebra puts this in the form of an interval around 𝜇: 𝜎 𝜎 𝑃 (𝑥 − 𝑧𝛼/2 ⋅ √ < 𝜇 < 𝑥 + 𝑧𝛼/2 ⋅ √ | 𝜇 = 𝜇0 ) = 1 − 𝛼 𝑛 𝑛 We can emphasize that the interval depends only on the statistic 𝑥 and the known value 𝜎 by writing this as 𝜎 𝜎 𝑃 ([𝑥 − 𝑧𝛼/2 ⋅ √ , 𝑥 + 𝑧𝛼/2 ⋅ √ ] contains 𝜇 | 𝜇 = 𝜇0 ) = 1 − 𝛼. 𝑛 𝑛 This is the (1 − 𝛼) 𝑧-confidence interval for 𝜇. We often write it using the shorthand 𝜎 𝑥 ± 𝑧𝛼/2 ⋅ √ 𝑛 Think of it as 𝑥 ± error. Make sure you notice that the probabilities are conditioned on 𝜇 = 𝜇0 . As with all frequentist statistics, we have to fix hypothesized values of the parameters in order to compute probabilities. 3.2 t-confidence intervals for the mean: normal data with unknown mean and standard deviation 𝑡-confidence intervals for the mean of normal data are based on the Studentized mean, i.e. the 𝑡-statistic. 18.05 Class 23, Confidence Intervals: Three Views, Spring 2022 3 Again we have 𝑥1 , 𝑥2 , … , 𝑥𝑛 ∼ N(𝜇, 𝜎2 ), but now we assume both 𝜇 and 𝜎 are unknown. As we did above, let’s write the (unknown) true value of 𝜇 as 𝜇0 . We know that the Studentized mean follows a Student 𝑡 distribution with 𝑛 − 1 degrees of freedom. That is, 𝑡= 𝑥 − 𝜇0 √ ∼ 𝑡(𝑛 − 1), 𝑠/ 𝑛 where 𝑠2 is the sample variance. Now all we have to do is replace the standardized mean by the Studentized mean and the same logic we used for 𝑧 gives us the 𝑡-confidence interval: start with 𝑃 (−𝑡𝛼/2 < 𝑥−𝜇 √ < 𝑡𝛼/2 | 𝜇 = 𝜇0 ) = 1 − 𝛼. 𝑠/ 𝑛 A little bit of algebra isolates 𝜇 in the middle of an interval: 𝑠 𝑠 𝑃 (𝑥 − 𝑡𝛼/2 ⋅ √ < 𝜇 < 𝑥 + 𝑡𝛼/2 ⋅ √ | 𝜇 = 𝜇0 ) = 1 − 𝛼 𝑛 𝑛 We can emphasize that the interval depends only on the statistics 𝑥 and 𝑠 by writing this as 𝑠 𝑠 𝑃 ([𝑥 − 𝑡𝛼/2 ⋅ √ , 𝑥 + 𝑡𝛼/2 ⋅ √ ] contains 𝜇 | 𝜇 = 𝜇0 ) = 1 − 𝛼. 𝑛 𝑛 This is the (1 − 𝛼) 𝑡-confidence interval for 𝜇. We often write it using the shorthand 𝑠 𝑥 ± 𝑡𝛼/2 ⋅ √ 𝑛 Think of it as 𝑥 ± error. 3.3 Chi-square confidence intervals for variance: normal data with unknown mean and standard deviation You guessed it: 𝜒2 -confidence intervals for the variance of normal data are based on the standardized variance, i.e. the 𝜒2 -statistic. We follow the same logic as above to get a 𝜒2 -confidence interval for 𝜎2 . Because this is the third time through it we’ll move a little more quickly. We assume we have 𝑛 independent normal samples: 𝑥1 , 𝑥2 , … , 𝑥𝑛 ∼ N(𝜇, 𝜎2 ). We assume that 𝜇 and 𝜎 are both unknown and write the (unknown) true value of 𝜎 as 𝜎0 . The standardized variance is (𝑛 − 1)𝑠2 𝑋2 = ∼ 𝜒2 (𝑛 − 1). 𝜎02 We know that the 𝑋 2 statistic follows a 𝜒2 distribution with 𝑛 − 1 degrees of freedom. For 𝑍 and 𝑡 we used, without comment, the symmetry of the distributions to replace 𝑧1−𝛼/2 by −𝑧𝛼/2 and 𝑡1−𝛼/2 by −𝑡𝛼/2 . Because the 𝜒2 distribution is not symmetric we need to be explicit about the critical values on both the left and the right. That is, 𝑃 (𝑐1−𝛼/2 < 𝑋 2 < 𝑐𝛼/2 ) = 1 − 𝛼, 18.05 Class 23, Confidence Intervals: Three Views, Spring 2022 4 where 𝑐𝛼/2 and 𝑐1−𝛼/2 are right tail critical values. Thus, 𝑃 (𝑐1−𝛼/2 < (𝑛 − 1)𝑠2 < 𝑐𝛼/2 | 𝜎 = 𝜎0 ) = 1 − 𝛼 𝜎2 A little bit of algebra puts this in the form of an interval around 𝜎2 : 𝑃( (𝑛 − 1)𝑠2 (𝑛 − 1)𝑠2 < 𝜎2 < | 𝜎 = 𝜎0 ) = 1 − 𝛼 𝑐𝛼/2 𝑐1−𝛼/2 We can emphasize that the interval depends only on the statistic 𝑠2 by writing this as 𝑃 ([ (𝑛 − 1)𝑠2 (𝑛 − 1)𝑠2 , ] contains 𝜎2 | 𝜎 = 𝜎0 ) = 1 − 𝛼. 𝑐𝛼/2 𝑐1−𝛼/2 This is the (1 − 𝛼) 𝜒2 -confidence interval for 𝜎2 . 4 Confidence intervals via hypothesis testing Suppose we have data drawn from a distribution with a parameter 𝜃 whose value is unknown. A significance test for the value 𝜃 has the following short description. 1. Set the null hypothesis 𝐻0 ∶ 𝜃 = 𝜃0 for some special value 𝜃0 , e.g. we often have 𝐻0 ∶ 𝜃 = 0. 2. Use the data to compute the value of a test statistic, call it 𝑥. 3. If 𝑥 is far enough into the tail of the null distribution (the distribution assuming the null hypothesis) then we reject 𝐻0 . In the case where there is no special value to test we may still want to estimate 𝜃. This is the reverse of significance testing; rather than seeing if we should reject a specific value of 𝜃 because it doesn’t fit the data we want to find the range of values of 𝜃 that do, in some sense, fit the data. This gives us the following definitions. Definition. Given a value 𝑥 of the test statistic, the (1 − 𝛼) confidence interval contains all values 𝜃0 which are not rejected (at significance level 𝛼) when they are the null hypothesis. Definition. A type 1 CI error occurs when the confidence interval does not contain the true value of 𝜃. For a (1 − 𝛼) confidence interval the type 1 CI error rate is 𝛼. Example 1. Here is an example relating confidence intervals and hypothesis tests. Suppose data 𝑥 is drawn from a binomial(12, 𝜃) distribution with 𝜃 unknown. Let 𝛼 = 0.1 and create the (1 − 𝛼) = 90% confidence interval for each possible value of 𝑥. Solution: Our strategy is to look at one possible value of 𝜃 at a time and choose rejection regions for a significance test with 𝛼 = 0.1. Once this is done, we will know, for each value of 𝑥, which values of 𝜃 are not rejected, i.e. the confidence interval associated with 𝑥. To start we set up a likelihood table for binomial(12, 𝜃) in Table 1. Each row shows the probabilities 𝑝(𝑥|𝜃) for one value of 𝜃. To keep the size manageable we only show 𝜃 in increments of 0.1. 18.05 Class 23, Confidence Intervals: Three Views, Spring 2022 𝜃\𝑥 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.07 0.28 1.00 1 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.07 0.21 0.38 0.00 2 0.00 0.00 0.00 0.00 0.00 0.02 0.06 0.17 0.28 0.23 0.00 3 0.00 0.00 0.00 0.00 0.01 0.05 0.14 0.24 0.24 0.09 0.00 4 0.00 0.00 0.00 0.01 0.04 0.12 0.21 0.23 0.13 0.02 0.00 5 0.00 0.00 0.00 0.03 0.10 0.19 0.23 0.16 0.05 0.00 0.00 6 0.00 0.00 0.02 0.08 0.18 0.23 0.18 0.08 0.02 0.00 0.00 7 0.00 0.00 0.05 0.16 0.23 0.19 0.10 0.03 0.00 0.00 0.00 8 0.00 0.02 0.13 0.23 0.21 0.12 0.04 0.01 0.00 0.00 0.00 5 9 0.00 0.09 0.24 0.24 0.14 0.05 0.01 0.00 0.00 0.00 0.00 10 0.00 0.23 0.28 0.17 0.06 0.02 0.00 0.00 0.00 0.00 0.00 11 0.00 0.38 0.21 0.07 0.02 0.00 0.00 0.00 0.00 0.00 0.00 12 1.00 0.28 0.07 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Table 1. Likelihood table for Binomial(12, 𝜃) Tables 2-4 below show the rejection region (in orange) and non-rejection region (in blue) for the various values of 𝜃. To emphasize the row-by-row nature of the process the Table 2 just shows these regions for 𝜃 = 1.0, then Table 3 adds in regions for 𝜃 = 0.9 and Table 4 shows them for all the values of 𝜃. Immediately following the tables we give a detailed explanation of how the rejection/nonrejection regions were chosen. 𝜃\𝑥 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.07 0.28 1.00 1 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.07 0.21 0.38 0.00 2 0.00 0.00 0.00 0.00 0.00 0.02 0.06 0.17 0.28 0.23 0.00 3 0.00 0.00 0.00 0.00 0.01 0.05 0.14 0.24 0.24 0.09 0.00 4 0.00 0.00 0.00 0.01 0.04 0.12 0.21 0.23 0.13 0.02 0.00 5 0.00 0.00 0.00 0.03 0.10 0.19 0.23 0.16 0.05 0.00 0.00 6 0.00 0.00 0.02 0.08 0.18 0.23 0.18 0.08 0.02 0.00 0.00 7 0.00 0.00 0.05 0.16 0.23 0.19 0.10 0.03 0.00 0.00 0.00 8 0.00 0.02 0.13 0.23 0.21 0.12 0.04 0.01 0.00 0.00 0.00 9 0.00 0.09 0.24 0.24 0.14 0.05 0.01 0.00 0.00 0.00 0.00 10 0.00 0.23 0.28 0.17 0.06 0.02 0.00 0.00 0.00 0.00 0.00 11 0.00 0.38 0.21 0.07 0.02 0.00 0.00 0.00 0.00 0.00 0.00 12 1.00 0.28 0.07 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 significance 0.000 Table 2. Likelihood table for binomial(12, 𝜃) with rejection (orange)/non-rejection (blue) regions for 𝜃 = 1.0 𝜃\𝑥 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.07 0.28 1.00 1 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.07 0.21 0.38 0.00 2 0.00 0.00 0.00 0.00 0.00 0.02 0.06 0.17 0.28 0.23 0.00 3 0.00 0.00 0.00 0.00 0.01 0.05 0.14 0.24 0.24 0.09 0.00 4 0.00 0.00 0.00 0.01 0.04 0.12 0.21 0.23 0.13 0.02 0.00 5 0.00 0.00 0.00 0.03 0.10 0.19 0.23 0.16 0.05 0.00 0.00 6 0.00 0.00 0.02 0.08 0.18 0.23 0.18 0.08 0.02 0.00 0.00 7 0.00 0.00 0.05 0.16 0.23 0.19 0.10 0.03 0.00 0.00 0.00 8 0.00 0.02 0.13 0.23 0.21 0.12 0.04 0.01 0.00 0.00 0.00 9 0.00 0.09 0.24 0.24 0.14 0.05 0.01 0.00 0.00 0.00 0.00 10 0.00 0.23 0.28 0.17 0.06 0.02 0.00 0.00 0.00 0.00 0.00 11 0.00 0.38 0.21 0.07 0.02 0.00 0.00 0.00 0.00 0.00 0.00 12 1.00 0.28 0.07 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 significance 0.000 0.026 Table 3. Likelihood table: rejection (orange)/non-rejection (blue) regions for 𝜃 = 1.0 and 0.9 18.05 Class 23, Confidence Intervals: Three Views, Spring 2022 𝜃\𝑥 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.07 0.28 1.00 1 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.07 0.21 0.38 0.00 2 0.00 0.00 0.00 0.00 0.00 0.02 0.06 0.17 0.28 0.23 0.00 3 0.00 0.00 0.00 0.00 0.01 0.05 0.14 0.24 0.24 0.09 0.00 4 0.00 0.00 0.00 0.01 0.04 0.12 0.21 0.23 0.13 0.02 0.00 5 0.00 0.00 0.00 0.03 0.10 0.19 0.23 0.16 0.05 0.00 0.00 6 0.00 0.00 0.02 0.08 0.18 0.23 0.18 0.08 0.02 0.00 0.00 7 0.00 0.00 0.05 0.16 0.23 0.19 0.10 0.03 0.00 0.00 0.00 8 0.00 0.02 0.13 0.23 0.21 0.12 0.04 0.01 0.00 0.00 0.00 6 9 0.00 0.09 0.24 0.24 0.14 0.05 0.01 0.00 0.00 0.00 0.00 10 0.00 0.23 0.28 0.17 0.06 0.02 0.00 0.00 0.00 0.00 0.00 11 0.00 0.38 0.21 0.07 0.02 0.00 0.00 0.00 0.00 0.00 0.00 12 1.00 0.28 0.07 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 significance 0.000 0.026 0.073 0.052 0.077 0.092 0.077 0.052 0.073 0.026 0.000 Table 4. Likelihood table: rejection (orange)/non-rejection (blue) regions for 𝜃 = 0.0 to 1.0 Choosing the rejection and non-rejection regions in the tables The first problem we confront is how exactly to choose the rejection region. We used two rules: 1. The total probabilitiy of the rejection region, i.e. the significance, should be less than or equal to 0.1. (Since we have a discrete distribution it is impossible to make the significance exactly 0.1.) 2. We build the rejection region by choosing values of 𝑥 one at a time, always picking the unused value with the smallest probability. We stop when the next value would make the significance more that 0.1. There are other ways to choose the rejection region which would result in slight differences. Our method is one reasonable way. Table 2 shows the rejection (orange) and non-rejection (blue) regions for 𝜃 = 1.0. This is a special case because most of the probabilities in this row are 0.0. We’ll move right on to the next table and step through the process for that. In Table 3, let’s walk through the steps used to find these regions for 𝜃 = 0.9. • The smallest probability is when 𝑥 = 0, so 𝑥 = 0 is in the rejection region. • The next smallest is when 𝑥 = 1, so 𝑥 = 1 is in the rejection region. • We continue with 𝑥 = 2, … , 8. At this point the total probability in the rejection region is 0.026. • The next smallest probability is when 𝑥 = 9. Adding this probability (0.09) to 0.026 would put the total probability over 0.1. So we leave 𝑥 = 9 out of the rejection region and stop the process. Note three things for the 𝜃 = 0.9 row: 1. None of the probabilities in this row are truly zero, though some are small enough that they equal 0 to 2 decimal places. 2. We show the significance for this value of 𝜃 in the right hand margin. More precisely, we show the significance level of the NHST with null hypothesis 𝜃 = 0.9 and the given rejection region. 18.05 Class 23, Confidence Intervals: Three Views, Spring 2022 7 3. The rejection region consists of values of 𝑥. When we say the rejection region is shown in orange we really mean the rejection region contains the values of 𝑥 corresponding to the probabilities highlighted in orange. Think: Look back at the 𝜃 = 1.0 row and make sure you understand why the rejection region is 𝑥 = 0, … , 11 and the significance is 0.000. Example 2. Using Table 4 determine the 0.90 confidence interval when 𝑥 = 8. Solution: The 90% confidence interval consists of all those 𝜃 that would not be rejected by an 𝛼 = 0.1 hypothesis test when 𝑥 = 8. Looking at the table, the blue (non-rejected) entries in the column 𝑥 = 8 correspond to 0.5 ≤ 𝜃 ≤ 0.8: the confidence interval is [0.5, 0.8]. Remark: The point of this example is to show how confidence intervals and hypothesis tests are related. Since Table 4 has only finitely many values of 𝜃, our answer is close but not exact. Using a computer we could look at many more values of 𝜃. For this problem we used R to find that, correct to 2 decimal places, the confidence interval is [0.42, 0.85]. Example 3. Explain why the expected type one CI error rate will be at most 0.092, provided that the true value of 𝜃 is in the table. Solution: The short answer is that this is the maximum significance for any 𝜃 in Table 4. Expanding on that slightly: we make a type one CI error if the confidence interval does not contain the true value of 𝜃, call it 𝜃true . This happens exactly when the data 𝑥 is in the rejection region for 𝜃true . The probability of this happening is the significance for 𝜃true and this is at most 0.092. Remark: The point of this example is to show how confidence level, type one CI error rate and significance for each hypothesis are related. As in the previous example, we can use R to compute the significance for many more values of 𝜃. When we do this we find that the maximum significance for any 𝜃 is 0.1 ocurring when 𝜃 ≈ 0.0452. Summary notes: 1. We start with a test statistic 𝑥. The confidence interval is random because it depends on 𝑥. 2. For each hypothesized value of 𝜃 we make a significance test with significance level 𝛼 by choosing rejection regions. 3. For a specific value of 𝑥 the associated confidence interval for 𝜃 consists of all 𝜃 that aren’t rejected for that value, i.e. all 𝜃 that have 𝑥 in their non-rejection regions. 4. Because the distribution is discrete we can’t always achieve the exact significance level, so our confidence interval is really an ‘at least 90% confidence interval’. Example 4. Open the applet https://mathlets.org/mathlets/confidence-intervals/. We want you to play with the applet to understand the random nature of confidence intervals and the meaning of confidence as (1 - type I CI error rate). (a) Read the help. It is short and will help orient you in the applet. Play with different settings of the parameters to see how they affect the size of the confidence intervals. (b) Set the number of trials to 𝑁 = 1. Click the ‘Run N trials’ button repeatedly and see that each time data is generated the confidence intervals jump around. (c) Now set the confidence level to 𝑐 = 0.5. As you click the ‘Run N trials’ button you 18.05 Class 23, Confidence Intervals: Three Views, Spring 2022 8 should see that about 50% of the confidence intervals include the true value of 𝜇. The ‘Z correct’ and ‘t correct’ values should change accordingly. (d) Now set the number of trials to 𝑁 = 100. With 𝑐 = 0.8. The ‘Run N trials’ button will now run 100 trials at a time. Only the last confidence interval will be shown in the graph, but the trials all run and the ‘percent correct’ statistics will be updated based on all 100 trials. Click the run trials button repeatedly. Watch the correct rates start to converge to the confidence level. To converge even faster, set 𝑁 = 1000. 5 Formal view of confidence intervals Recall: An interval statistic is an interval 𝐼𝑥 computed from data 𝑥. An interval is determined by its lower and upper bounds, and these are random because 𝑥 is random. We suppose that 𝑥 is drawn from a distribution with pdf 𝑓(𝑥|𝜃) where the parameter 𝜃 is unknown. Definition: A (1 − 𝛼) confidence interval for 𝜃 is an interval statistic 𝐼𝑥 such that 𝑃 (𝐼𝑥 contains 𝜃0 | 𝜃 = 𝜃0 ) = 1 − 𝛼 for all possible values of 𝜃0 . We wish this was simpler, but a definition is a definition and this definition is one way to weigh the evidence provided by the data 𝑥. Let’s unpack it a bit. The confidence level of an interval statistic is a probability concerning a random interval and a hypothesized value 𝜃0 for the unknown parameter. Precisely, it is the probability that the random interval 𝐼𝑥 (computed from random data 𝑥) contains the value 𝜃0 , given that the model parameter truly is 𝜃0 . Since the true value of 𝜃 is unknown, the frequentist statistician defines 95% confidence intervals so that the 0.95 probability is valid no matter which hypothesized value of the parameter is actually true. 6 Comparison with Bayesian probability intervals Confidence intervals are a frequentist notion, and as we’ve repeated many times, frequentists don’t assign probabilities to hypotheses, e.g., to the value of an unknown parameter. Rather they compute likelihoods; that is, probabilities about data or associated statistics given a hypothesis (note the condition 𝜃 = 𝜃0 in the formal view of confidence intervals). Note that the construction of confidence intervals proceeds entirely from the full likelihood table. In contrast Bayesian posterior probability intervals are truly the probability that the value of the unknown parameter lies in the reported range. We add the usual caveat that this depends on the specific choice of a (possibly subjective) Bayesian prior. This distinction between the two is subtle because Bayesian posterior probability intervals and frequentist confidence intervals share the following properties: 1. They start from a model 𝑓(𝑥|𝜃) for observed data 𝑥 with unknown parameter 𝜃. 18.05 Class 23, Confidence Intervals: Three Views, Spring 2022 9 2. Given data 𝑥, they give an interval 𝐼(𝑥) specifying a range of values for 𝜃. 3. They come with a number (say 0.95) that is the probability of something. In practice, many people misinterpret confidence intervals as Bayesian probability intervals, forgetting that frequentists never place probabilities on hypotheses (this is analogous to mistaking the 𝑝-value in NHST for the probability that 𝐻0 is false). The next section explores this mistake in some detail. The harm of this misinterpretation is somewhat mitigated by that fact that, given enough data and a reasonable prior, Bayesian and frequentist intervals often work out to be quite similar. For an amusing example illustrating how they can be quite different, see the first answer in the link just below (involving chocolate chip cookies!). This example uses the formal definitions and is really about confidence sets instead of confidence intervals. https://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval 7 Misinterpreting confidence intervals It is very tempting to think that given a 95% confidence interval for, say, the mean, the probability that the true mean is in the confidence interval is 95%. We know this can’t be true because the value of the mean can only be hypothesized and Frequentists don’t assign probabilities to hypotheses. To be more concrete, if the mean is 𝜃 and the confidence interval is [45, 55] then the statement 45 ≤ 𝜃 ≤ 55 is a hypothesis, so asking for the probability that 45 ≤ 𝜃 ≤ 55, is asking for the probability of a hypothesis. The mistake is subtle and hard to wrap your mind around. It boils down to a question of what is being randomly sampled. Here is an attempt to explain the issue. First, consider a test for a disease. Assume a person is given the test. Let 𝑇 + be a positive test and 𝐷+ be that they have the disease. Assume the test is 95% accurate, i.e. 𝑃 (𝑇 + |𝐷+ ) = 0.95. We know (base rate fallacy) that this does not imply that 𝑃 (𝐷+ |𝑇 + ) = 0.95. Let’s look at this from a different angle: Implicit in 𝑃 (𝑇 + |𝐷+ ) = 0.95 is the following experiment: Draw a random person from the set of all people with the disease and give them the test. Then 95% will test positive. That is, the population sampled is all people with the disease and the event considered is that the chosen person in that population tests positive. For 𝑃 (𝐷+ |𝑇 + ), the experiment is to draw a random person from the set of all people who tested positive. The probability is the fraction who have the disease. That is, the population sampled is all people who test positive and the event considered is that the chosen person in that population has the disease. We can’t expect 𝑃 (𝑇 + |𝐷+ ) and 𝑃 (𝐷+ |𝑇 + ) to be the same, since we’re sampling from different populations and looking at different events. The probability 𝑃 (𝐷+ |𝑇 + ) can be computed using Bayes’ theorem from the (prior) probability 𝑃 (𝐷+ ) and the likelihoods 𝑃 (𝑇 + |𝐷+), 𝑃 (𝑇 + |𝐷− ). Confidence intervals are a little more abstract, but the analysis is similar. Just as in testing for a disease, the populations sampled will be different. One source of difficulty is that the 18.05 Class 23, Confidence Intervals: Three Views, Spring 2022 10 events are essentially the same in both cases. Let’s assume we have a distribution with unknown mean 𝜃0 . We generate some data and compute a 95% confidence interval for the mean. The value of 95% comes from the following implied experiment: imagine having run many trials and created a confidence interval for each one. Then, 95% of confidence intervals contain the true mean. In notation, 𝑃 (random interval contains 𝜃0 ) = 0.95. That is, the random sample is drawn from the set of all confidence intervals generated by our trials. The event in question is that the chosen interval contains the true mean. What if we run one experiment and generate the 95% confidence interval, call it 𝐼. To a Frequentist, the true mean is not random and we have a fixed interval. So, to the Frequentist, it makes no sense to ask about the probability 𝜃0 is in 𝐼. To a Bayesian, it is fine to consider 𝜃0 as randomly drawn from a probability distribution –they often interpret it as a description of the uncertainty of our knowledge. So, they can ask for the probability 𝑃 (random 𝜃0 is in a given 𝐼). So, here, the random sample is drawn from the set of possible means and the event considered is that the chosen mean is contained in the given interval. As in the disease testing example, what population is being randomly sampled is different in the two cases, i.e in the first we have a random interval, in the second we have a random value for the true mean. As noted above, in both cases the event is that the true mean is in the interval. We finish by noting that 𝑃 (random 𝜃0 is in 𝐼) can be computed using Bayes’ theorem and depends on the prior distribution for the true mean and the likelihoods that each mean will generate the given confidence interval. The formula is a little unwieldy. Here it is. • Call the interval 𝐼 and the true mean 𝜃0 . • Call the data 𝐼. This is a shorthand for the data that the interval 𝐼 is based on. • Let 𝑝(𝜃) be the prior probability that 𝜃0 = 𝜃. • The likelihood 𝑓(𝐼|𝜃) is the probability (or density) that the experiment would produce the interval 𝐼 given 𝜃0 = 𝜃. • Let 𝑝(𝜃|𝐼) be the posterior probability that 𝜃0 = 𝜃. This is the updated probability found using the Bayes’ theorem. Bayes’ theorem gives us 𝑝(𝜃|𝐼) = 𝑓(𝐼|𝜃)𝑝(𝜃) , where 𝑓(𝐼) = ∑ 𝑓(𝐼|𝜃)𝑝(𝜃). 𝑓(𝐼) 𝜃 So we have, 𝑃 (𝜃0 ∈ 𝐼|𝐼) = ∑ 𝑝(𝜃|𝐼). (As usual, if 𝜃 has a continuous range of values, 𝜃 in 𝐼 then the sums will be replaced by integrals.)