18.05 S22 Class 6: Central Limit Theorem and the Law of Large Numbers

Central Limit Theorem and the Law of Large Numbers Class 6, 18.05 1 Learning Goals 1. Understand the statement of the law of large numbers. 2. Understand the statement of the central limit theorem. 3. Be able to use the central limit theorem to approximate probabilities of averages and sums of independent identically-distributed random variables. 2 Introduction We all understand intuitively that the average of many measurements of the same unknown quantity tends to give a better estimate than a single measurement. Intuitively, this is because the random error of each measurement cancels out in the average. In these notes we will make this intuition precise in two ways: the law of large numbers (LoLN) and the central limit theorem (CLT). Briefly, both the law of large numbers and central limit theorem are about many independent samples from same distribution. The LoLN tells us two things: 1. The average of many independent samples is (with high probability) close to the mean of the underlying distribution. 2. The density histogram of many independent samples is (with high probability) close to the graph of the density of the underlying distribution. To be absolutely correct mathematically we need to make these statements more precise, but as stated they are a good way to think about the law of large numbers. The central limit theorem says that the sum or average of many independent copies of a random variable is approximately a normal random variable. The CLT goes on to give precise values for the mean and standard deviation of the normal variable. These are both remarkable facts. Perhaps just as remarkable is the fact that often in practice 𝑛 does not have to be all that large. 2.1 There is more to experimentation than mathematics The mathematics of the LoLN says that the average of a lot of independent samples from a random variable will almost certainly approach the mean of the variable. The mathematics cannot tell us if the tool or experiment is producing data worth averaging. For example, if the measuring device is defective or poorly calibrated then the average of many measurements will be a highly accurate estimate of the wrong thing! This is an example of 1 18.05 Class 6, Central Limit Theorem and the Law of Large Numbers, Spring 2022 2 systematic error or sampling bias, as opposed to the random error controlled by the law of large numbers. 3 The law of large numbers Suppose 𝑋1 , 𝑋2 , …, 𝑋𝑛 are independent random variables with the same underlying distribution. In this case, we say that the 𝑋𝑖 are independent and identically-distributed, or i.i.d. In particular, the 𝑋𝑖 all have the same mean 𝜇 and standard deviation 𝜎. Let 𝑋 𝑛 be the average of 𝑋1 , … , 𝑋𝑛 : 𝑋𝑛 = 𝑋1 + 𝑋 2 + ⋯ + 𝑋 𝑛 1 𝑛 = ∑ 𝑋𝑖 . 𝑛 𝑛 𝑖=1 Note that 𝑋 𝑛 is itself a random variable. The law of large numbers and central limit theorem tell us about the value and distribution of 𝑋 𝑛 , respectively. LoLN: As 𝑛 grows, the probability that 𝑋 𝑛 is close to 𝜇 goes to 1. CLT: As 𝑛 grows, the distribution of 𝑋 𝑛 converges to the normal distribution 𝑁 (𝜇, 𝜎2 /𝑛). Before giving a more formal statement of the LoLN, let’s unpack its meaning through a concrete example (we’ll return to the CLT later on). Example 1. Averages of Bernoulli random variables Suppose each 𝑋𝑖 is an independent flip of a fair coin, so 𝑋𝑖 ∼ Bernoulli(0.5) and 𝜇 = 0.5. Then 𝑋 𝑛 is the proportion of heads in 𝑛 flips, and we expect that this proportion is close to 0.5 for large 𝑛. Randomness being what it is, this is not guaranteed; for example we could get 1000 heads in 1000 flips, though the probability of this occurring is very small. So our intuition translates to: with high probability the sample average 𝑋 𝑛 is close to the mean 0.5 for large 𝑛. We’ll demonstrate by doing some calculations in R. You can find the code used for ‘class 6 prep’ in the usual place on our website. To start we’ll look at the probability of being within 0.1 of the mean. We can express this probability as 𝑃 (|𝑋 𝑛 − 0.5| < 0.1) or equivalently 𝑃 (0.4 ≤ 𝑋 𝑛 ≤ 0.6) The law of large numbers says that this probability goes to 1 as the number of flips 𝑛 gets large. Our R code produces the following values for 𝑃 (0.4 ≤ 𝑋 𝑛 ≤ 0.6). 𝑛 = 10: 𝑛 = 50: 𝑛 = 100: 𝑛 = 500: 𝑛 = 1000: pbinom(6, 10, 0.5) - pbinom(3, 10, 0.5) pbinom(30, 50, 0.5) - pbinom(19, 50, 0.5) pbinom(60, 100, 0.5) - pbinom(39, 100, 0.5) pbinom(300, 500, 0.5) - pbinom(199, 500, 0.5) pbinom(600, 1000, 0.5) - pbinom(399, 1000, 0.5) = = = = = 0.65625 0.8810795 0.9647998 0.9999941 1 As predicted by the LoLN the probability goes to 1 as 𝑛 grows. We redo the computations to see the probability of being within 0.01 of the mean. Our R code produces the following values for 𝑃 (0.49 ≤ 𝑋 𝑛 ≤ 0.51). 18.05 Class 6, Central Limit Theorem and the Law of Large Numbers, Spring 2022 𝑛 = 10: 𝑛 = 100: 𝑛 = 1000: 𝑛 = 10000: pbinom(5, 10, 0.5) - pbinom(4, 10, 0.5) pbinom(51, 100, 0.5) - pbinom(48, 100, 0.5) pbinom(510, 1000, 0.5) - pbinom(489, 1000, 0.5) pbinom(5100, 10000, 0.5) - pbinom(4899, 10000, 0.5) = = = = 3 0.2460937 0.2356466 0.49334 0.9555742 Again we see the probability of being close to the mean going to 1 as 𝑛 grows. Since 0.01 is smaller than 0.1 it takes larger values of 𝑛 to raise the probability to near 1. This convergence of the probability to 1 is the LoLN in action! Whenever you’re confused, it will help you to keep this example in mind. So we see that the LoLN says that with high probability the average of a large number of independent trials from the same distribution will be very close to the underlying mean of the distribution. Now we’re ready for the formal statement. 3.1 Formal statement of the law of large numbers Theorem (Law of Large Numbers): Suppose 𝑋1 , 𝑋2 , …, 𝑋𝑛 , … are i.i.d. random variables with mean 𝜇. For each 𝑛, let 𝑋 𝑛 be the average of the first 𝑛 variables. Then for any 𝑎 > 0, we have lim 𝑃 (|𝑋 𝑛 − 𝜇| < 𝑎) = 1. 𝑛→∞ This says precisely that as 𝑛 increases the probability of being within 𝑎 of the mean goes to 1. Think of 𝑎 as a small tolerance of error from the true mean 𝜇. Looking back at Example 1, we see that for tosses of a fair coin: If we choose the number of tosses 𝑛 = 500, then with probability 𝑝 = 0.99999, the experimental frequency of heads 𝑋 𝑛 will be within 𝑎 = 0.1 of 0.5. In words, this tells us that, on average, only 1 in 100,000 experiments will produce an experimental frequency outside this range. If we decrease the tolerance 𝑎 and/or increase the probability 𝑝, then 𝑛 will need to be larger. 4 Histograms We can summarize multiple samples 𝑥1 , … , 𝑥𝑛 of a random variable in a histogram. Here we want to carefully construct histograms so that they resemble the area under the pdf. We will then see how the LoLN applies to histograms. The step-by-step instructions for constructing a density or frequency histogram are as follows. 1. Pick an interval of the real line and divide it into 𝑚 intervals, with endpoints 𝑏0 , 𝑏1 , …, 𝑏𝑚 . Usually these are equally sized, so let’s assume this to start. 𝑥 𝑏0 𝑏1 𝑏2 𝑏3 𝑏4 𝑏5 𝑏6 Six equally-sized bins Each of the intervals is called a bin. For example, in the figure above the first bin is [𝑏0 , 𝑏1 ] and the last bin is [𝑏5 , 𝑏6 ]. Each bin has a bin width, e.g. 𝑏1 − 𝑏0 is the first bin width. Usually the bins all have the same width, called the bin width of the histogram. 18.05 Class 6, Central Limit Theorem and the Law of Large Numbers, Spring 2022 4 2. Place each 𝑥𝑖 into the bin that contains its value. If 𝑥𝑖 lies on the boundary of two bins, we’ll put it in the left bin (this is the R default, though it can be changed). 3. To draw a frequency histogram: put a vertical bar above each bin. The height of the bar should equal the number of 𝑥𝑖 in the bin. 4. To draw a density histogram: put a vertical bar above each bin. The area of the bar should equal the fraction of all data points that lie in the bin. Notes: 1. When all the bins have the same width, the frequency histogram bars have area proportional to the count. So the density histogram results from simply by dividing the height of each bar by the total area of the frequency histogram. Ignoring the vertical scale, the two histograms look identical. 2. Caution: if the bin widths differ, the frequency and density histograms may look very different. There is an example of this below. Don’t let anyone fool you by manipulating bin widths to produce a histogram that suits their mischievous purposes! In 18.05, we’ll stick with equally-sized bins. In general, we prefer the density histogram since its vertical scale is the same as that of the pdf. Examples. Here are some examples of histograms, all with the data [0.5,1,1,1.5,1.5,1.5,2,2,2,2]. The R code that drew them is in the file ’class6-prep-b.r’. You can find it in the usual place on our website. 1. Here the density and frequency plots look the same but have different vertical scales. Histogram of x 0 0.0 1 2 Frequency 0.4 0.2 Density 0.6 3 0.8 4 Histogram of x 0.5 1.0 1.5 2.0 0.5 x 1.0 1.5 2.0 x Bins centered at 0.5, 1, 1.5, 2, i.e. width 0.5, bounds at 0.25, 0.75, 1.25, 1.75, 2.25. 2. Note the values are all on the bin boundaries and are put into the left-hand bin. That is, the bins are right-closed, e.g the first bin is for values in the right-closed interval (0, 0.5]. 18.05 Class 6, Central Limit Theorem and the Law of Large Numbers, Spring 2022 Histogram of x 3 0 0.0 1 2 Frequency 0.2 0.4 Density 0.6 4 0.8 Histogram of x 5 0.0 0.5 1.0 1.5 2.0 0.0 0.5 x 1.0 1.5 2.0 x Bin bounds at 0, 0.5, 1, 1.5, 2. 3. Here we show density histograms based on different bin widths. Note that the scale keeps the total area equal to 1. The gaps are bins with zero counts. Histogram of x 1.0 0.0 0.5 Density 0.0 0.2 0.4 Density 0.6 1.5 Histogram of x 0.0 0.5 1.0 1.5 2.0 0.5 1.0 x 1.5 2.0 x Right: narrow bins. Left: wide bins; 4. Here we use unqual bin widths, so the density and frequency histograms look different Histogram of x 3 0 0.0 1 2 Frequency 0.4 0.2 Density 0.6 4 5 0.8 Histogram of x 0.0 0.5 1.0 x 1.5 2.0 0.0 0.5 1.0 1.5 2.0 x Don’t be fooled! These are based on the same data. The density histogram is the better choice with unequal bin widths. In fact, R will complain 18.05 Class 6, Central Limit Theorem and the Law of Large Numbers, Spring 2022 6 if you try to make a frequency histogram with unequal bin widths. Compare the frequency histogram with unequal bin widths with all the other histograms we drew for this data. It clearly looks different. What happened is that by combining the data in bins (0.5, 1] and (1, 1.5] into one bin (0.5, 1.5) we effectively made the height of both smaller bins greater. The reason the density histogram is nice is discussed in the next section. 4.1 The law of large numbers and histograms The law of large number has an important consequence for density histograms. LoLN for histograms: With high probability the density histogram of a large number of samples from a distribution is a good approximation of the graph of the underlying pdf 𝑓(𝑥) over the range of the histogram. Let’s illustrate this by generating a density histogram with bin width 0.1 from 100000 draws from a standard normal distribution. As you can see, the density histogram very closely tracks the graph of the standard normal pdf 𝜙(𝑧). 0.5 0.4 0.3 0.2 0.1 0 -4 -2 0 2 4 Density histogram of 10000 draws from a standard normal distribution, with 𝜙(𝑧) in blue. 5 The Central Limit Theorem We now prepare for the statement of the CLT. 5.1 Standardization Given a random variable 𝑋 with mean 𝜇 and standard deviation 𝜎, we define its standardization of 𝑋 as the new random variable 𝑍= 𝑋−𝜇 . 𝜎 18.05 Class 6, Central Limit Theorem and the Law of Large Numbers, Spring 2022 7 Note that 𝑍 has mean 0 and standard deviation 1. Note also that if 𝑋 has a normal distribution, then the standardization of 𝑋 is the standard normal distribution 𝑍 with mean 0 and variance 1. This explains the term ‘standardization’ and the notation of 𝑍 above. 5.2 Statement of the Central Limit Theorem Suppose 𝑋1 , 𝑋2 , …, 𝑋𝑛 , …are i.i.d. random variables each having mean 𝜇 and standard deviation 𝜎. For each 𝑛, let 𝑆𝑛 denote the sum and let 𝑋 𝑛 be the average of 𝑋1 , … , 𝑋𝑛 . 𝑛 𝑆𝑛 = 𝑋1 + 𝑋2 + … + 𝑋𝑛 = ∑ 𝑋𝑖 𝑖=1 𝑋 + 𝑋2 + … + 𝑋 𝑛 𝑆 𝑋𝑛 = 1 = 𝑛. 𝑛 𝑛 The properties of mean and variance show 𝐸[𝑆𝑛 ] = 𝑛𝜇, Var(𝑆𝑛 ) 𝐸[𝑋 𝑛 ] = 𝜇, = 𝑛𝜎2 , Var(𝑋 𝑛 ) = 𝜎2 , 𝑛 √ 𝑛𝜎 𝜎 𝑆𝑛 = 𝜎𝑋 𝑛 𝜎 =√ . 𝑛 Since they are multiples of each other, 𝑆𝑛 and 𝑋 𝑛 have the same standardization 𝑍𝑛 = 𝑆𝑛 − 𝑛𝜇 𝑋 −𝜇 √ = 𝑛√ 𝜎 𝑛 𝜎/ 𝑛 Central Limit Theorem: For large 𝑛, 𝑋 𝑛 ≈ N(𝜇, 𝜎2 /𝑛), 𝑆𝑛 ≈ N(𝑛𝜇, 𝑛𝜎2 ), 𝑍𝑛 ≈ N(0, 1). Notes: 1. In words: 𝑋 𝑛 is approximately a normal distribution with the same mean as 𝑋 but a smaller variance. 2. 𝑆𝑛 is approximately normal. 3. Standardized 𝑋 𝑛 and 𝑆𝑛 are approximately standard normal. The central limit theorem allows us to approximate a sum or average of i.i.d random variables by a normal random variable. This is extremely useful because it is usually easy to do computations with the normal distribution. A precise statement of the CLT is that the cdf’s of 𝑍𝑛 converge to Φ(𝑧): lim 𝐹𝑍𝑛 (𝑧) = Φ(𝑧). 𝑛→∞ The proof of the Central Limit Theorem is more technical than we want to get in 18.05. It is accessible to anyone with a decent calculus background. 18.05 Class 6, Central Limit Theorem and the Law of Large Numbers, Spring 2022 5.3 8 Standard Normal Probabilities To apply the CLT, we will want to have some normal probabilities at our fingertips. The following probabilities appeared in Class 5. Let 𝑍 ∼ 𝑁 (0, 1), a standard normal random variable. Then with rounding we have: 1. 𝑃 (|𝑍| < 1) ≈ 0.68 2. 𝑃 (|𝑍| < 2) ≈ 0.95; more precisely 𝑃 (|𝑍| < 1.96) ≈ 0.95. 3. 𝑃 (|𝑍| < 3) ≈ 0.997 These numbers are easily computed in R using pnorm. However, they are well worth remembering as rules of thumb. You should think of them as: 1. The probability that a normal random variable is within 1 standard deviation of its mean is 0.68. 2. The probability that a normal random variable is within 2 standard deviations of its mean is 0.95. 3. The probability that a normal random variable is within 3 standard deviations of its mean is 0.997. This is shown graphically in the following figure. within 1 ⋅ 𝜎 ≈ 68% within 2 ⋅ 𝜎 ≈ 95% within 3 ⋅ 𝜎 ≈ 99% Normal PDF 68% 95% 99% 𝜇 − 3𝜎 𝜇 − 2𝜎 𝜇−𝜎 𝜇 𝜇+𝜎 𝜇 + 2𝜎 𝜇 + 3𝜎 𝑧 Claim: From these numbers we can derive: 1. 𝑃 (𝑍 < 1) ≈ 0.84 2. 𝑃 (𝑍 < 2) ≈ 0.977 3. 𝑃 (𝑍 < 3) ≈ 0.999 Proof: We know 𝑃 (|𝑍| < 1) = 0.68. The remaining probability of 0.32 is in the two regions 𝑍 > 1 and 𝑍 < −1. These regions are referred to as the right-hand tail and the left-hand tail respectively. By symmetry each tail has area 0.16. Thus, 𝑃 (𝑍 < 1) = 𝑃 (|𝑍| < 1) + 𝑃 (left-hand tail) = 0.84 The other two cases are handled similarly. 5.4 Applications of the CLT Example 2. Flip a fair coin 100 times. Estimate the probability of more than 55 heads. 18.05 Class 6, Central Limit Theorem and the Law of Large Numbers, Spring 2022 9 Solution: Let 𝑋𝑗 be the result of the 𝑗th flip, so 𝑋𝑗 = 1 for heads and 𝑋𝑗 = 0 for tails. The total number of heads is 𝑆 = 𝑋1 + 𝑋2 + … + 𝑋100 . We know 𝐸[𝑋𝑗 ] = 0.5 and Var(𝑋𝑗 ) = 1/4. Since 𝑛 = 100, we have 𝐸[𝑆] = 50, Var(𝑆) = 25 and 𝜎𝑆 = 5. The central limit theorem says that the standardization of 𝑆 is approximately N(0, 1). The question asks for 𝑃 (𝑆 > 55). Standardizing and using the CLT we get 𝑃 (𝑆 > 55) = 𝑃 ( 𝑆 − 50 55 − 50 > ) ≈ 𝑃 (𝑍 > 1) = 0.16. 5 5 Here 𝑍 is a standard normal random variable and 𝑃 (𝑍 > 1) = 1 − 𝑃 (𝑍 < 1) ≈ 0.16. Example 3. Estimate the probability of more than 220 heads in 400 flips. Solution: This is nearly identical to the previous example. Now 𝜇𝑆 = 200 and 𝜎𝑆 = 10 and we want 𝑃 (𝑆 > 220). Standardizing and using the CLT we get: 𝑃 (𝑆 > 220) = 𝑃 ( 𝑆 − 𝜇𝑆 220 − 200 > ) ≈ 𝑃 (𝑍 > 2) = 0.025. 𝜎𝑆 10 Again, 𝑍 ∼ N(0, 1) and the rules of thumb show 𝑃 (𝑍 > 2) = 0.025. Note: Even though 55/100 = 220/400, the probability of more than 55 heads in 100 flips is larger than the probability of more than 220 heads in 400 flips. This is due to the LoLN and the larger value of 𝑛 in the latter case. Example 4. Estimate the probability of between 40 and 60 heads in 100 flips. Solution: As in the first example, 𝐸[𝑆] = 50, Var(𝑆) = 25 and 𝜎𝑆 = 5. So 𝑃 (40 ≤ 𝑆 ≤ 60) = 𝑃 ( 40 − 50 𝑆 − 50 60 − 50 ) ≈ 𝑃 (−2 ≤ 𝑍 ≤ 2) ≤ ≤ 5 5 5 We can compute the right-hand side using our rule of thumb. For a more accurate answer we use R: pnorm(2) - pnorm(-2) = 0.954 … Recall that in Section 3 we used the binomial distribution to compute an answer of 0.965…. So our approximate answer using CLT is off by about 1%. Think: Would you expect the CLT method to give a better or worse approximation of 𝑃 (200 < 𝑆 < 300) with 𝑛 = 500? We encourage you to check your answer using R. Example 5. Polling. When taking a political poll the results are often reported as a number with a margin of error. For example 52% ± 3% favor candidate A. The rule of √ thumb is that if you poll 𝑛 people then the margin of error is ±1/ 𝑛. We will now see exactly what this means and that it is an application of the central limit theorem. 18.05 Class 6, Central Limit Theorem and the Law of Large Numbers, Spring 2022 10 Suppose there are 2 candidates A and B. Suppose further that the fraction of the population who prefer A is 𝑝0 . That is, if you ask a random person who they prefer then the probability they’ll answer A is 𝑝𝑜 To run the poll a pollster selects 𝑛 people at random and asks ‘Do you support candidate A or candidate B?’ Thus we can view the poll as a sequence of 𝑛 independent Bernoulli(𝑝0 ) trials, 𝑋1 , 𝑋2 , …, 𝑋𝑛 , where 𝑋𝑖 is 1 if the ith person prefers A and 0 if they prefer B. The fraction of people polled that prefer A is just the average 𝑋. We know that each 𝑋𝑖 ∼ Bernoulli(𝑝0 ) so, 𝐸[𝑋𝑖 ] = 𝑝0 and 𝜎𝑋𝑖 = √𝑝0 (1 − 𝑝0 ). Therefore, the central limit theorem tells us that 𝑋 ≈ N(𝑝0 , 𝜎2 /𝑛), where 𝜎 = √𝑝0 (1 − 𝑝0 ). In a normal distribution 95% of the probability is within 2 standard deviations of the mean. √ This means that in 95% of polls of 𝑛 people the sample mean 𝑋 will be within 2𝜎/ 𝑛 of the true mean 𝑝0 . The final step is to note that for any value of 𝑝0 we have 𝜎 ≤ 1/2. (It is an easy calculus exercise to see that 1/4 is the maximum value of 𝜎2 = 𝑝0 (1 − 𝑝0 ).) This means that we can conservatively say that in 95% of polls of 𝑛 people the sample mean √ 𝑋 is within 1/ 𝑛 of the true mean. The frequentist statistician then takes the interval √ 𝑋 ± 1/ 𝑛 and calls it the 95% confidence interval for 𝑝0 . A word of caution: it is tempting and common, but wrong, to think that there is a 95% probability the true fraction 𝑝0 is in a particular confidence interval. This is subtle, but the error is the same one as thinking you have a disease if a test with a 95% true positive rate comes back positive. We will go into this in much more detail when we learn about confidence intervals. 5.5 Why use the CLT Since the probabilities in the above examples can be computed exactly using the binomial distribution, you may be wondering what is the point of finding an approximate answer using the CLT. In fact, we were only able to compute these probabilities exactly because the 𝑋𝑖 were Bernoulli and so the sum 𝑆 was binomial. In general, the distribution of the 𝑋𝑖 may not be familiar, or may not even be known, so you will not be able to compute the probabilities for 𝑆 exactly. It can also happen that the exact computation is possible in theory but too computationally intensive in practice, even for a computer. The power of the CLT is that it applies whenever 𝑋𝑖 has a mean and a variance. Though the CLT applies to many distributions, we will see in the next section that some distributions require larger 𝑛 for the approximation to be a good one. 5.6 How big does 𝑛 have to be to apply the CLT? Short answer: often, not that big. The following sequences of pictures show the convergence of averages to a normal distribution. 18.05 Class 6, Central Limit Theorem and the Law of Large Numbers, Spring 2022 11 First we show the standardized average of 𝑛 i.i.d. uniform random variables with 𝑛 = 1, 2, 4, 8, 12. The pdf of the average is in blue and the standard normal pdf is in red. By the time 𝑛 = 12 the fit between the standardized average and the true normal looks very good. −2 −1 0 x 1 2 3 0.2 −3 −2 −1 0 x 1 2 3 2 3 −3 −2 −1 0 x 1 2 3 0.2 0.4 Uniform: n = 12 0.0 0.0 0.2 0.4 Uniform: n = 8 0.0 0.0 0.2 0.2 0.0 −3 Uniform: n = 4 0.4 Uniform: n = 2 0.4 0.4 Uniform: n = 1 −3 −2 −1 0 x 1 2 3 −3 −2 −1 0 x 1 Next we show the standardized average of 𝑛 i.i.d. exponential random variables with 𝑛 = 1, 2, 4, 8, 16, 64. Notice that this asymmetric density takes more terms to converge to the normal density. Exponential: n = 4 −3 −2 −1 0 x 1 2 3 0.2 0.0 0.0 0.0 0.2 0.4 0.4 0.4 Exponential: n = 2 0.8 Exponential: n = 1 −3 −1 0 x 1 2 3 Exponential: n = 16 −1 0 x 1 2 3 −1 0 x 1 2 3 0.2 0.4 0.0 0.0 −2 −2 Exponential: n = 64 0.2 0.4 0.2 0.0 −3 −3 0.4 Exponential: n = 8 −2 −3 −2 −1 0 x 1 2 3 −3 −2 −1 0 x 1 2 3 Next we show the (non-standardized) average of 𝑛 exponential random variables with 𝑛 = 1, 2, 4, 16, 64. Notice how this standard deviation shrinks as 𝑛 grows, resulting in a spikier (more peaked) density. 18.05 Class 6, Central Limit Theorem and the Law of Large Numbers, Spring 2022 Exponential: n = 2 Exponential: n = 4 −2 −1 0 1 x 2 3 4 0.4 0.0 0.0 0.0 0.4 0.4 0.8 0.8 Exponential: n = 1 12 −2 0 1 x 2 3 4 −2 −1 0 1 x 2 3 4 Exponential: n = 64 0.0 1.0 2.0 0.0 0.5 1.0 1.5 3.0 Exponential: n = 16 −1 −2 −1 0 1 x 2 3 4 −2 −1 0 1 x 2 3 4 The central limit theorem works for discrete variables also. Here is the standardized average of 𝑛 i.i.d. Bernoulli(0.5) random variables with 𝑛 = 1, 2, 12, 64. Notice that as 𝑛 grows, the average can take more values, which allows the discrete distribution to ’fill in’ the normal density. −3 −2 −1 0 x 1 2 3 2 3 0.2 0.4 Bernoulli: n = 12 0.0 0.2 0.4 Bernoulli: n = 2 0.0 0.0 0.2 0.4 Bernoulli: n = 1 −3 −2 −1 0 x 1 2 3 −3 −2 −1 0 x 1 2 3 0.0 0.2 0.4 Bernoulli: n = 64 −3 −2 −1 0 x 1 Note. In order to put the binomial (sum of Bernoulli) and normal distribution on the same axes, we had to convert the binomial probability mass function to a density. We did this by making it a bar graph with bars centered on each value and with bar width equal to the distance between values. Then the height of each bar is chosen so that the area equals the probability of the corresponding value. Finally we show the (non-standardized) average of 𝑛 Bernoulli(0.5) random variables, with 𝑛 = 4, 12, 64. Notice how the standard deviation gets smaller resulting in a spikier (more peaked) density. (In these figures, rather than plotting colored bars, we made the bars white and only plotted a blue line at the center of each bar. 18.05 Class 6, Central Limit Theorem and the Law of Large Numbers, Spring 2022 Bernoulli: n = 12 Bernoulli: n = 64 −1.0 0.0 0.5 x 1.0 1.5 2.0 4 2 0 0.0 0.0 0.4 1.0 0.8 2.0 6 Bernoulli: n = 2 0.0 0.5 x 1.0 0.0 0.5 x 1.0 13