18.05 S22 Class 7: Covariance and Correlation

Covariance and Correlation Class 7, 18.05 1 Learning Goals 1. Understand the meaning of covariance and correlation. 2. Be able to compute the covariance and correlation of two random variables. 2 Covariance Covariance is a measure of how much two random variables vary together. For example, height and weight of giraffes have positive covariance because when one is big the other tends also to be big. Definition: Suppose 𝑋 and 𝑌 are random variables with means 𝜇𝑋 and 𝜇𝑌 . The covariance of 𝑋 and 𝑌 is defined as Cov(𝑋, 𝑌 ) = 𝐸[(𝑋 − 𝜇𝑋 )(𝑌 − 𝜇𝑌 )]. 2.1 Properties of covariance 1. Cov(𝑎𝑋 + 𝑏, 𝑐𝑌 + 𝑑) = 𝑎𝑐Cov(𝑋, 𝑌 ) for constants 𝑎, 𝑏, 𝑐, 𝑑. 2. Cov(𝑋1 + 𝑋2 , 𝑌 ) = Cov(𝑋1 , 𝑌 ) + Cov(𝑋2 , 𝑌 ). 3. Cov(𝑋, 𝑋) = Var(𝑋) 4. Cov(𝑋, 𝑌 ) = 𝐸[𝑋𝑌 ] − 𝜇𝑋 𝜇𝑌 . 5. Var(𝑋 + 𝑌 ) = Var(𝑋) + Var(𝑌 ) + 2Cov(𝑋, 𝑌 ) for any 𝑋 and 𝑌 . 6. If 𝑋 and 𝑌 are independent then Cov(𝑋, 𝑌 ) = 0. Warning: The converse is false: zero covariance does not always imply independence. Notes. 1. Property 4 is like the similar property for variance. Indeed, if 𝑋 = 𝑌 it is exactly that property: Var(𝑋) = 𝐸[𝑋 2 ] − 𝜇2𝑋 . By Property 5, the formula in Property 6 reduces to our earlier formula Var(𝑋 + 𝑌 ) = Var(𝑋) + Var(𝑌 ) when 𝑋 and 𝑌 are independent. We give the proofs below. However, understanding and using these properties is more important than memorizing their proofs. 1 18.05 Class 7, Covariance and Correlation, Spring 2022 2.2 2 Sums and integrals for computing covariance Since covariance is defined as an expected value we compute it in the usual way as a sum or integral. Discrete case: If 𝑋 and 𝑌 have joint pmf 𝑝(𝑥𝑖 , 𝑦𝑗 ) then 𝑛 𝑚 𝑛 𝑚 Cov(𝑋, 𝑌 ) = ∑ ∑ 𝑝(𝑥𝑖 , 𝑦𝑗 )(𝑥𝑖 − 𝜇𝑋 )(𝑦𝑗 − 𝜇𝑌 ) = (∑ ∑ 𝑝(𝑥𝑖 , 𝑦𝑗 )𝑥𝑖 𝑦𝑗 ) − 𝜇𝑋 𝜇𝑌 . 𝑖=1 𝑗=1 𝑖=1 𝑗=1 Continuous case: If 𝑋 and 𝑌 have joint pdf 𝑓(𝑥, 𝑦) over range [𝑎, 𝑏] × [𝑐, 𝑑] then 𝑑 𝑏 𝑑 𝑏 Cov(𝑋, 𝑌 ) = ∫ ∫ (𝑥 − 𝜇𝑥 )(𝑦 − 𝜇𝑦 )𝑓(𝑥, 𝑦) 𝑑𝑥 𝑑𝑦 = (∫ ∫ 𝑥𝑦𝑓(𝑥, 𝑦) 𝑑𝑥 𝑑𝑦) − 𝜇𝑥 𝜇𝑦 . 𝑐 2.3 𝑎 𝑐 𝑎 Examples Example 1. Flip a fair coin 3 times. Let 𝑋 be the number of heads in the first 2 flips and let 𝑌 be the number of heads on the last 2 flips (so there is overlap on the middle flip). Compute Cov(𝑋, 𝑌 ). Solution: We’ll do this twice, first using the joint probability table and the definition of covariance, and then using the properties of covariance. With 3 tosses there are only 8 outcomes {HHH, HHT,...}, so we can create the joint probability table directly. 𝑋\𝑌 0 1 2 𝑝(𝑥𝑖 ) 0 1/4 0 1/8 1/8 1 1/8 2/8 1/8 2 0 1/8 1/8 𝑝(𝑦𝑗 ) 1/4 1/2 1/4 1/2 1/4 1 From the marginals we compute 𝐸[𝑋] = 1 = 𝐸[𝑌 ]. Now we use use the definition: Cov(𝑋, 𝑌 ) = 𝐸[(𝑋 − 𝜇𝑥 )(𝑌 − 𝜇𝑌 )] = ∑ 𝑝(𝑥𝑖 , 𝑦𝑗 )(𝑥𝑖 − 1)(𝑦𝑗 − 1) 𝑖,𝑗 We write out the sum leaving out all the terms that are 0, i.e. all the terms where 𝑥𝑖 = 1 or 𝑦𝑖 = 1 or the probability is 0. 1 1 1 Cov(𝑋, 𝑌 ) = (0 − 1)(0 − 1) + (2 − 1)(2 − 1) = . 8 8 4 We could also have used property 4 to do the computation: From the full table we compute 𝐸[𝑋𝑌 ] = 1 ⋅ 1 1 1 5 2 +2 +2 +4 = . 8 8 8 8 4 18.05 Class 7, Covariance and Correlation, Spring 2022 3 5 1 −1= . 4 4 Next we redo the computation of Cov(𝑋, 𝑌 ) using the properties of covariance. As usual, let 𝑋𝑖 be the result of the 𝑖th flip, so 𝑋𝑖 ∼ Bernoulli(0.5). We have So Cov(𝑋𝑌 ) = 𝐸[𝑋𝑌 ] − 𝜇𝑋 𝜇𝑌 = 𝑋 = 𝑋 1 + 𝑋2 and 𝑌 = 𝑋 2 + 𝑋3 . We know 𝐸[𝑋𝑖 ] = 1/2 and Var(𝑋𝑖 ) = 1/4. Therefore using Property 2 of covariance, we have Cov(𝑋, 𝑌 ) = Cov(𝑋1 +𝑋2 , 𝑋2 +𝑋3 ) = Cov(𝑋1 , 𝑋2 )+Cov(𝑋1 , 𝑋3 )+Cov(𝑋2 , 𝑋2 )+Cov(𝑋2 , 𝑋3 ). Since the different tosses are independent we know Cov(𝑋1 , 𝑋2 ) = Cov(𝑋1 , 𝑋3 ) = Cov(𝑋2 , 𝑋3 ) = 0. Looking at the expression for Cov(𝑋, 𝑌 ) there is only one non-zero term Cov(𝑋, 𝑌 ) = Cov(𝑋2 , 𝑋2 ) = Var(𝑋2 ) = 1 . 4 Example 2. (Zero covariance does not imply independence.) Let 𝑋 be a random variable that takes values −2, −1, 0, 1, 2; each with probability 1/5. Let 𝑌 = 𝑋 2 . Show that Cov(𝑋, 𝑌 ) = 0 but 𝑋 and 𝑌 are not independent. Solution: We make a joint probability table: 𝑌 \𝑋 -2 -1 0 0 1 4 1 2 𝑝(𝑦𝑗 ) 0 0 0 1/5 1/5 0 0 1/5 0 1/5 0 2/5 1/5 0 0 0 1/5 2/5 𝑝(𝑥𝑖 ) 1/5 1/5 1/5 1/5 1/5 1 Using the marginals we compute means 𝐸[𝑋] = 0 and 𝐸[𝑌 ] = 2. Next we show that 𝑋 and 𝑌 are not independent. To do this all we have to do is find one place where the product rule fails, i.e. where 𝑝(𝑥𝑖 , 𝑦𝑗 ) ≠ 𝑝(𝑥𝑖 )𝑝(𝑥𝑗 ): 𝑃 (𝑋 = −2, 𝑌 = 0) = 0 but 𝑃 (𝑋 = −2) ⋅ 𝑃 (𝑌 = 0) = 1/25. Since these are not equal 𝑋 and 𝑌 are not independent. Finally we compute covariance using Property 4: 1 Cov(𝑋, 𝑌 ) = (−8 − 1 + 1 + 8) − 𝜇𝑋 𝜇𝑦 = 0. 5 Discussion: This example shows that Cov(𝑋, 𝑌 ) = 0 does not imply that 𝑋 and 𝑌 are independent. In fact, 𝑋 and 𝑋 2 are as dependent as random variables can be: if you know the value of 𝑋 then you know the value of 𝑋 2 with 100% certainty. 18.05 Class 7, Covariance and Correlation, Spring 2022 4 The key point is that Cov(𝑋, 𝑌 ) measures the linear relationship between 𝑋 and 𝑌 . In the above example 𝑋 and 𝑋 2 have a quadratic relationship that is completely missed by Cov(𝑋, 𝑌 ). Continuous covariance works the same way, except our computations are done with integrals instead of sums. Here is an example. Example 3. Continuous covariance. Suppose 𝑋 and 𝑌 are jointly distributed random variables, with range on the unit square [0, 1] × [0, 1] and joint pdf 𝑓(𝑥, 𝑦) = 2𝑥3 + 2𝑦3 . (i) Verify the 𝑓(𝑥, 𝑦) is a valid probability density. (ii) Compute 𝜇𝑋 and 𝜇𝑌 . (iii) Compute the covariance of Cov(𝑋, 𝑌 ) Solution: Part of the point of this example is to show how to set up and compute the integrals using a joint density function. Since the pdf here is a polynomial, these computations are relatively easy. (i) A valid pdf has two properties: it is nonnegative and the total integral over the entire joint range is 1. Nonnegativity is clear: 𝑓(𝑥, 𝑦) ≥ 0. The integral is not hard to compute 1 1 1 1 ∫ ∫ 𝑓(𝑥, 𝑦) 𝑑𝑥 𝑑𝑦 = ∫ ∫ 2𝑥3 + 2𝑦3 𝑑𝑥 𝑑𝑦 0 0 0 0 1 1 𝑥4 1 Inner integral: ∫ 2𝑥 + 2𝑦 𝑑𝑥 = + 2𝑥𝑦3 ∣ = + 2𝑦3 . 2 2 0 0 3 3 1 1 𝑦 𝑦4 1 + 2𝑦3 𝑑𝑦 = + ∣ = 1. Outer integral: ∫ 2 2 0 0 2 So, the integral over the entire joint range is 1. Thus, 𝑓(𝑥, 𝑦) = 𝑥 + 𝑦 is a valid probability density. (ii) We need to compute integrals to find the means. We will write down the integrals, but not show the details of their computation. (Also, by symmetry, we know the two means are the same.) 1 1 1 1 𝜇𝑋 = ∫ ∫ 𝑥𝑓(𝑥, 𝑦) 𝑑𝑥 𝑑𝑦 = ∫ ∫ 2𝑥4 + 2𝑥𝑦3 ) 𝑑𝑥 𝑑𝑦 = 0 0 1 0 0 1 1 1 𝜇𝑌 = ∫ ∫ 𝑦𝑓(𝑥, 𝑦) 𝑑𝑥 𝑑𝑦 = ∫ ∫ 2𝑦𝑥3 + 2𝑦4 𝑑𝑥 𝑑𝑦 = 0 0 0 0 13 20 13 20 (iii) We know Cov(𝑋, 𝑌 ) = 𝐸 [(𝑋 − 𝜇𝑥 )(𝑌 − 𝜇𝑌 )]. This is an integral. Again, we will write down the integral, but not show details of its computation, 1 1 Cov(𝑋, 𝑌 ) = 𝐸 [(𝑋 − 𝜇𝑥 )(𝑌 − 𝜇𝑌 )] = ∫ ∫ (𝑥 − 13/20)(𝑦 − 13/20)𝑓(𝑥, 𝑦) 𝑑𝑥 𝑑𝑦 0 1 0 1 = ∫ ∫ (𝑥 − 7/12)(𝑦 − 7/12)(2𝑥3 + 2𝑦3 ) 𝑑𝑥 𝑑𝑦 = − 0 0 9 400 18.05 Class 7, Covariance and Correlation, Spring 2022 5 (In fact, we wrote down the integral in the most straightforward way, but secretly we did the computation by computing 𝐸[𝑋𝑌 ] − 𝐸[𝑋]𝐸[𝑌 ].) y 0.0 0.0 0.2 0.2 0.4 0.4 y 0.6 0.6 0.8 0.8 1.0 1.0 Here’s a plot of the pseudo-random samples generated from this distribution. Because the R code could do it easily, we also include a plot with a more extreme density function. 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 x 0.4 0.6 0.8 1.0 x Samples from 𝑓(𝑥, 𝑦) = 2𝑥3 + 2𝑦3 . Samples from 𝑓(𝑥, 𝑦) = 10𝑥19 + 10𝑦19 . 3 Correlation The units of covariance Cov(𝑋, 𝑌 ) are ‘units of 𝑋 times units of 𝑌 ’. This makes it hard to compare covariances: if we change scales then the covariance changes as well. Correlation is a way to remove the scale from the covariance. Definition: The correlation coefficient between 𝑋 and 𝑌 is defined by Cor(𝑋, 𝑌 ) = 𝜌 = 3.1 Cov(𝑋, 𝑌 ) . 𝜎𝑋 𝜎𝑌 Properties of correlation 1. 𝜌 is the covariance of the standardizations of 𝑋 and 𝑌 . 2. 𝜌 is dimensionless (it’s a ratio!). 3. −1 ≤ 𝜌 ≤ 1. Furthermore, 𝜌 = +1 if and only if 𝑌 = 𝑎𝑋 + 𝑏 with 𝑎 > 0, 𝜌 = −1 if and only if 𝑌 = 𝑎𝑋 + 𝑏 with 𝑎 < 0. Property 3 shows that 𝜌 measures the linear relationship between variables. If the correlation is positive then when 𝑋 is large, 𝑌 will tend to large as well. If the correlation is negative then when 𝑋 is large, 𝑌 will tend to be small. Example 2 above shows that correlation can completely miss higher order relationships. Example 4. We continue Example 1. To compute the correlation we divide the covariance by the standard deviations. In Example 1 we found Cov(𝑋, 𝑌 ) = 1/4 and Var(𝑋) = 18.05 Class 7, Covariance and Correlation, Spring 2022 6 √ √ 2Var(𝑋𝑗 ) = 1/2. So, 𝜎𝑋 = 1/ 2. Likewise 𝜎𝑌 = 1/ 2. Thus Cor(𝑋, 𝑌 ) = 1/4 1 Cov(𝑋, 𝑌 ) = = . 𝜎𝑋 𝜎𝑌 1/2 2 We see a positive correlation, which means that larger 𝑋 tend to go with larger 𝑌 and smaller 𝑋 with smaller 𝑌 . In Example 1 this happens because toss 2 is included in both 𝑋 and 𝑌 , so it contributes to the size of both. Example 5. Look back at Example 3. See if you can compute the following. Var(𝑋) = 31/400, so 𝜎𝑋 = √31/400 ≈ 0.28 Var(𝑌 ) = Var(𝑋), so 𝜎𝑌 ≈ 0.28 Cov(𝑋, 𝑌 ) Cor(𝑋, 𝑌 ) = ≈ −0.29. 𝜎𝑋 𝜎𝑌 3.2 Bivariate normal distributions The bivariate normal distribution has density −1 𝑓(𝑥, 𝑦) = 2) [ e 2(1−𝜌 (𝑥−𝜇𝑋 )2 𝜎2 𝑋 + (𝑦−𝜇𝑌 )2 𝜎2 𝑌 − 2𝜌(𝑥−𝜇𝑥 )(𝑦−𝜇𝑦 ) ] 𝜎𝑥 𝜎𝑦 2𝜋𝜎𝑋 𝜎𝑌 √1 − 𝜌2 For this distribution, the marginal distributions for 𝑋 and 𝑌 are normal and the correlation between 𝑋 and 𝑌 is 𝜌. In the figures below we used R to simulate the distribution for various values of 𝜌. Individually 𝑋 and 𝑌 are standard normal, i.e. 𝜇𝑋 = 𝜇𝑌 = 0 and 𝜎𝑋 = 𝜎𝑌 = 1. The figures show scatter plots of the results. These plots and the next set show an important feature of correlation. We divide the data into quadrants by drawing a horizontal and a verticle line at the means of the 𝑦 data and 𝑥 data respectively. A positive correlation corresponds to the data tending to lie in the 1st and 3rd quadrants. A negative correlation corresponds to data tending to lie in the 2nd and 4th quadrants. You can see the data gathering about a line as 𝜌 becomes closer to ±1. 3 rho=0.00 rho=0.30 ● 3 ● ● ● ● ●● ● ●● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ●● ● ● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ● ●● ● ● ●●● ●● ●● ● ● ●● ● ● ●● ● ●● ● ● ● ● ●● ●● ● ● ● ●● ●●● ● ●● ●●● ● ●●● ●● ● ● ● ● ● ● ● ●●● ● ● ● ●●● ● ●● ● ● ●● ●●● ● ● ● ● ● ●● ●● ● ●● ● ●● ●●● ● ● ● ● ● ●● ● ●●● ●● ● ● ●●● ● ● ● ● ● ● ●● ●●●● ●● ● ● ●● ●●● ●● ●● ● ● ●● ●● ● ●● ●●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ●● ●● ●● ●● ● ● ●●● ● ● ● ●● ● ●●● ● ●●●● ● ●● ●● ● ● ● ●● ●● ● ●● ● ●● ● ● ● ● ● ●● ●● ●● ● ●● ● ● ●● ● ● ●● ●● ●● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ●● ● ●● ● ● ● ●● ● ● ●● ● ●● ● ● ●● ● ●● ● ● ● ●●● ● ●●●● ● ● ● ● ●● ●● ●●● ● ● ●● ● ●● ●●●● ●● ● ● ● ●●● ● ● ● ● ● ●● ●● ● ●● ● ●● ●●●● ● ● ●●● ● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ●●● ●●● ● ●●●● ●●● ● ● ● ● ●●● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ●●● ● ● ● ●● ● ● ●● ● ● ● ●● ●●●● ●●● ●● ● ● ●● ● ● ● ●●●● ●● ●● ●● ● ●●●● ● ●●●● ● ● ● ● ●● ● ● ● ● ● ● ●● ●●●● ● ● ●● ● ● ● ●●● ● ● ●● ● ●● ● ● ●● ●● ● ●● ●● ●● ● ● ●● ●●● ● ● ●●●● ●● ●● ● ● ●●● ●● ● ●● ● ● ● ●● ●● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ●●● ●●●● ● ● ●●● ●● ● ● ● ●● ● ●● ● ●● ● ● ●● ● ●● ● ●● ● ● ●●● ● ● ● ●●●● ● ● ● ● ●● ● ● ●● ●●● ● ●● ● ● ● ● ●● ●● ● ●● ●● ●●● ●● ●●●●●● ●● ●● ●●●● ● ● ● ● ● ● ●● ● ● ●●● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●●● ● ● ● ●●● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●●● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●●●● ● ●● ● ● ● ● ●● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● −3 −2 −1 0 1 x 2 3 2 1 ● −1 0 y ● ● −3 −1 0 −3 y 1 2 ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ● ●● ● ● ●● ●●● ● ● ●● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ● ● ●● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ●● ●●● ●●● ● ● ●● ● ●● ●● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ●● ●● ●● ●●●●● ●● ● ● ●● ● ● ●● ● ●● ● ●● ● ● ●● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ●●● ● ● ● ● ● ● ●● ●●●● ●● ● ● ●● ● ● ● ● ●●● ●●● ● ● ●● ● ●●● ●● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●●● ● ● ● ●● ●●●● ●●● ●● ● ● ● ●● ● ●● ● ● ● ●● ●●● ●● ●●●●● ● ● ● ● ● ● ● ●● ● ● ●●● ●● ● ● ● ●● ● ● ● ●●● ● ●●● ● ●● ● ●● ● ● ●● ● ●●● ● ● ●● ● ● ●●●● ● ●●● ●● ● ●● ●● ●● ●● ●● ●● ● ●● ● ●●● ●● ● ●● ● ●●● ● ● ●● ● ● ●● ●● ● ● ●● ● ● ●●● ● ● ● ● ● ●●●● ●●●● ●● ●● ●● ● ●● ● ● ● ● ● ● ●● ●● ● ● ●●● ● ●●●● ● ●●●● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ● ●● ●●●● ● ●● ● ● ●● ● ●●●●● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ●● ●●● ● ● ●● ● ● ● ●● ● ●●●● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ●● ● ●● ● ●● ●● ●● ● ●● ● ● ● ● ● ● ●● ●● ● ● ●● ●●● ●●● ● ● ●●●● ● ●● ● ● ● ● ●● ● ● ●●● ● ● ● ●● ●● ●● ● ●● ● ●● ●● ● ●● ● ● ● ● ●●● ● ●● ●● ● ● ● ●● ● ●●● ● ●● ● ● ●●● ● ● ● ●●●● ●● ● ●● ● ● ●●● ● ●● ● ●●●●● ● ●●● ● ● ●●● ●● ● ● ●● ●● ●●●● ● ●● ●● ● ●● ●●● ● ● ●● ● ● ● ●●●● ● ●● ● ● ● ● ● ●●●● ●● ● ● ●●●●● ● ● ●● ●● ●● ● ●● ● ● ● ●● ● ●●●● ●●● ● ● ● ● ● ● ● ● ● ● ● ●● ●●● ●●●●● ● ●● ● ● ● ● ● ●● ●● ● ● ● ● ●● ● ●● ● ● ● ● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −4 −3 −2 −1 0 x 1 2 3 18.05 Class 7, Covariance and Correlation, Spring 2022 7 rho=0.70 2 ●● ● ● ● ●● ● ● ●● ● ●● ●● ● ●● ● ●● ● ●●● ● ● ●● ● ● ●● ● ● ●● ●● ●●● ● ● ●● ● ● ● ● ● ●● ● ●● ● ●● ●● ● ● ● ● ● ● ●●● ● ● ●●●● ●●●● ● ● ●● ●●●● ● ● ● ●●● ●● ● ● ● ●●● ●● ● ●●●●●● ●● ● ● ● ●● ●● ● ●● ● ●● ● ●●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ●● ●●●●● ●● ●●● ● ●● ● ●● ● ● ●●●● ● ● ●● ● ● ●● ● ●● ●● ● ● ● ●● ●●●● ●● ● ●●● ●● ●● ●● ● ● ●● ● ● ● ●●● ● ● ●●● ● ●● ●● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ●● ● ●● ●● ● ● ● ●●●● ● ● ●●●●● ● ●● ●●●●● ● ●●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ●● ●● ● ●● ● ●●● ● ● ● ● ● ● ●● ● ●●● ● ●● ● ● ●●● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●●● ●●● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ●● ●●●●● ●●● ●● ● ● ●● ●● ● ● ●● ●● ●● ●● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ●● ● ● ● ●● ●●● ●● ●●● ●● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ●● ●●●● ●● ● ● ●● ● ● ● ● ● ●● ●● ● ●●● ● ●●● ● ● ●● ● ● ●● ● ●● ● ●● ●● ●●●● ● ●● ●● ●● ● ● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ●●● ● ● ● ●●●● ● ●● ●●● ● ●●● ● ●● ●●● ● ● ● ●●● ● ● ●● ● ●● ●● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ●●● ●● ●● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ●●● ● ● ● ●● ●● ● ●●●● ● ●● ● ● ● ● ●● ● ● ● ● ● ●● ●● ● ● ●● ●●●● ● ●● ● ● ● ● ● ●●● ● ● ●● ●● ●●● ● ●●● ● ● ● ●● ●●● ● ● ● ●● ● ●● ● ●●●●● ● ● ● ●● ● ● ●● ●●●● ● ● ● ●● ● ●● ● ● ●●●● ● ● ●●● ● ● ● ● ● ●●● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ●●● ●● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ● ● ● ● ●●● ●● ● ●●●●●●●●●● ●● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● 2 ●● ●● −3 −2 −4 ● ● 0 −2 ● ●● y 1 ● 0 −3 −1 y rho=1.00 −1 0 x 1 2 3 ● ●● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● −4 −2 3 rho=−0.50 1 0 y −2 ● 2 ●● ● ● −3 3.3 −2 −1 0 x 1 ● ● ● 2 ● ● ●● ● ●● ● ● ● ● ●● ● ● ●●● ● ● ●● ●●●● ● ● ● ●● ●● ● ● ● ●● ● ●● ●● ●●●● ●●● ●● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ●● ●●●●● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ●● ● ● ●● ● ●● ● ● ●● ●● ●●● ● ● ● ●●●● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●●●●● ● ● ● ●● ● ●●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ●●● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●●●● ● ●●● ●● ●● ●● ● ● ●● ● ● ● ● ● ●● ●● ● ● ●●● ● ●●● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●● ●●● ● ● ● ● ● ●● ● ● ● ●● ● ●● ●● ●● ● ● ●● ● ● ●●● ● ● ● ● ● ● ●● ●●● ● ●●● ●● ● ● ● ●●● ● ●● ● ● ●●● ● ● ●● ● ●● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●● ●● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●●● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ●● ● ● ●● ●● ● ● ● ●● ● ● ●● ● ● ● ●● ●● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ●● ● ● ● ●●●●● ● ●●● ● ● ●● ● ● ● ● ● ● ● ● ●● ●●● ● ● ● ●● ● ●●● ●● ●● ● ●●●●●● ●●●●● ●●● ● ● ●● ●●●●● ● ● ● ●● ● ● ●● ● ●● ●● ● ● ● ●●● ● ● ● ● ● ●● ● ● ●● ●● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● 0 ●● ● ● −2 ● ● ●● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ●● ●●● ● ● ● ●●●●● ●● ● ● ● ●●● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ● ●● ● ● ●●●● ● ●●●● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●●●●● ● ● ●●●●● ● ● ●●●●●● ● ●● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ●● ●● ● ● ● ●● ● ●● ● ●● ●● ● ● ● ●● ● ● ●●●● ● ● ●● ● ●●● ●●● ●●●●● ●●● ● ● ●● ● ●● ●●● ●●●●● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ●●● ● ● ● ●● ●● ● ● ●● ● ●● ●● ● ● ● ●● ●● ●● ● ●●● ●● ● ●● ●●● ●● ●● ● ● ● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ●●●●● ● ● ●● ●●● ● ●●●● ● ● ● ●● ●●● ● ● ●● ● ● ●● ●● ● ● ● ●●● ●● ●●●● ● ●● ● ● ●●●● ● ● ●● ●●● ●●●●● ●●● ● ●● ● ●● ●● ● ●● ● ●● ●●● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ● ●●●● ●●● ● ● ●●● ●● ● ● ●● ● ●●●● ●● ● ● ● ● ●● ● ● ● ●●●● ● ● ● ●●● ● ● ●● ● ●● ● ● ● ● ●● ● ● ●●● ● ●● ● ● ● ●●● ● ● ● ● ●●●● ● ●● ● ● ●●●●● ●● ● ● ● ● ● ●●●●● ●● ●●● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●●● ● ● ●● ●●● ●●● ●● ● ●● ●● ● ●● ●● ● ● ● ●● ●● ●●●●● ●● ●● ● ●● ● ● ●● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ●●● ●● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ●●● ● ● ●● ● ●●● ● ●●● ●●● ● ● ●●● ● ●● ● ● ●● ● ●● ●● ● ● ● ● ●●● ● ● ●● ●● ●● ● ●● ●● ● ● ● ● ●● ●●●● ● ●●●● ●●●●● ● ●● ● ●● ● ● ● ● ● ●●● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ●● ●● ●● ●● ● ●● ● ● ● ● ● ●● ● ●● ● ● ● ●●● ● ●●● ● ●●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ●●● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● −4 ● ● y 2 ● ● 2 ● ● ● ● 0 x rho=−0.90 ● ● ● ● 3 ● ● −3 −2 −1 0 1 2 3 4 x Overlapping uniform distributions We ran simulations in R of the following scenario. 𝑋1 , 𝑋2 , …, 𝑋20 are i.i.d and follow a 𝑈 (0, 1) distribution. 𝑋 and 𝑌 are both sums of the same number of 𝑋𝑖 . We call the number of 𝑋𝑖 common to both 𝑋 and 𝑌 the overlap. The notation in the figures below indicates the number of 𝑋𝑖 being summed and the number which overlap. For example, 5,3 indicates that 𝑋 and 𝑌 were each the sum of 5 of the 𝑋𝑖 and that 3 of the 𝑋𝑖 were common to both sums. (The data was generated using rand(1,1000);) Using the linearity of covariance it is easy to compute the theoretical correlation. For each plot we give both the theoretical correlation and the correlation of the data from the simulated sample. 18.05 Class 7, Covariance and Correlation, Spring 2022 ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ●●●● ● ● ● ● ● ●●● ●● ● ● ● ● ●● ●● ● ●● ● ● ●● ●● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ●●●● ●● ●●● ● ● ●● ● ● ●● ●● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●●● ● ●● ●● ●●● ● ●● ● ●●● ● ● ●● ● ●● ● ● ●● ●● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ●●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ● ● ●●●● ●● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●●● ● ●● ● ● ● ●● ●● ● ●● ●● ● ●● ● ● ● ●●● ● ● ●● ● ●● ●● ● ●● ● ● ● ●● ● ● ●● ● ●●● ●● ● ● ● ● ● ●● ● ●● ● ●● ●● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ●●● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ●● ● ● ● ●● ●● ● ● ●● ● ●● ● ● ●● ●● ● ●● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●●● ●● ● ●● ● ● ● ●●●● ●● ● ●● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ● ●●● ● ●● ●●●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ●● ● ● ●●● ● ● ●● ● ●● ●● ●● ● ●●● ● ● ● ● ● ●● ● ●●●● ● ●● ● ● ● ●● ●● ● ● ●● ● ●● ●● ● ● ●● ● ● ● ● ●● ● ● ●●● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ●●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● (2, 1) cor=0.50, sample_cor=0.48 2.0 (1, 0) cor=0.00, sample_cor=−0.07 0.2 0.4 0.6 0.8 1.0 0.5 y 1.5 ●● ●● 0.0 0.0 0.4 y 0.8 ●● 0.0 8 1.0 ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ●● ● ●● ●● ● ● ● ● ●● ● ●● ●● ●● ●● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ●●● ●● ● ●● ● ● ● ●●● ● ●● ●●● ● ● ● ●●● ● ● ● ●● ● ● ● ●● ● ● ● ●●● ● ● ● ●●● ● ●●● ● ●● ● ●● ●● ● ● ● ● ● ●●●●● ● ● ●●●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ●●●● ●● ●●● ● ●● ● ● ● ● ●●●● ●● ● ●●●● ●●● ● ●● ●● ●● ●● ● ●● ● ● ● ● ● ●● ●● ●● ●● ● ● ● ●● ● ● ●● ● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ●●● ● ●● ● ● ●● ● ● ● ● ● ●● ●● ●● ●● ● ●● ● ● ● ● ●●● ● ●● ●●● ● ● ● ●●●● ● ● ●●● ●● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ● ● ●● ●● ● ● ●● ● ●● ●● ● ●● ●● ● ● ● ●● ●● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ● ● ● ●● ●●●● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ●● ●●● ● ● ● ● ● ● ● ●●● ● ●● ●● ● ●● ● ●● ● ●●●● ●● ● ● ●●●● ●● ● ● ● ● ●● ●●●●● ● ● ● ●● ● ●● ● ● ● ● ● ● ●●● ● ● ● ● ●● ●●● ● ● ● ●●● ● ● ● ●● ● ● ●● ● ●●● ● ●● ● ● ● ● ● ●● ●● ●● ●● ● ●●● ●● ●● ●●●●● ●●●●● ●● ● ●●● ● ●●●● ● ●● ● ● ●● ● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ●● ● ●● ● ● ●● ● ●●●● ● ●● ●● ●● ●● ● ● ● ● ●● ● ● ●● ● ●● ● ●● ● ●●● ● ●● ● ●● ●● ●●● ● ●●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●● ● ●● ●● ●● ● ● ● ●● ●● ● ● ● ●●● ● ●● ●●●● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ●●●● ● ●● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●● ●● ●● ● ●● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ●● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● 0.0 0.5 (5, 3) cor=0.60, sample_cor=0.63 ● ● ● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●●●● ●●● ● ●●● ●●● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●●●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●● ●● ● ● ●● ● ● ● ●●● ● ● ●●●● ● ●● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ●●● ● ●● ● ● ● ● ● ● ●●●●● ● ● ●●● ● ●●● ● ●●● ● ● ●●● ● ●● ● ●●●●●● ● ● ●● ● ● ● ●● ●● ●● ●● ● ● ● ●● ●●●● ● ● ● ●● ● ●● ●●● ● ●● ●●●● ●●● ● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ●● ●● ●●●●●●● ● ● ●● ●● ●● ●● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ●● ●● ●● ● ● ● ● ● ● ● ● ●●●●● ● ● ●● ● ● ● ● ●● ●●●●●● ● ●● ● ●● ●● ●●● ● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ●● ●●● ● ● ●● ●● ●● ●● ●● ●● ● ● ●●● ● ●● ●● ● ●● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●●● ●● ● ● ●● ● ●● ●●● ●●● ● ●● ●● ● ● ● ●●●● ● ●● ● ● ● ● ● ● ●●●● ●●●●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●●●● ● ● ● ●● ● ● ● ●● ● ●● ●●●●● ●● ● ● ●●●● ● ● ● ● ●● ●● ●● ● ● ●● ●● ● ●●●●● ● ● ● ● ● ● ●●●● ● ● ● ● ● ● ●●● ●● ● ● ● ●● ● ● ● ● ●● ●● ●● ●● ● ● ● ● ● ● ●●● ● ●● ●● ● ● ●●● ● ● ●● ● ● ● ● ●● ● ●●● ●●●●● ●● ● ● ● ● ●●●● ●●●● ●● ●● ●●● ●●● ●● ● ●●● ● ● ● ● ●● ● ●● ●● ● ● ●● ●●● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●●● ●● ● ●●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ● ●● ● ● ●●● ●● ●● ●● ● ● ● ● ● ●●● ●● ● ● ● ● ● ●● ● ●● ● ● ●● ●● ● ●● ● ●● ● ● ●● ● ● ● ● ● ●●● ● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ●● ● ● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 2 3 4 ● ● ● ● ● ● 3 ● 2 ● ● 1 ● ● ● ● ● 2.0 ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●●● ●● ●● ● ● ● ●● ● ●●● ● ● ●● ● ●●● ● ●●●● ● ● ● ●● ●● ● ● ● ● ●● ●●● ● ● ●● ● ● ● ● ● ●● ● ●●● ● ● ● ● ● ●● ●●● ● ●● ● ● ●● ● ●● ● ●● ● ● ●● ● ●● ● ● ● ●● ●● ● ● ●● ● ● ● ●● ● ●● ● ●● ● ● ●● ●● ●● ●●● ●● ● ● ● ●● ●● ● ●● ●● ● ● ● ●● ●● ●●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●●● ● ● ● ● ●● ●● ● ●●● ●● ● ●●● ● ● ●● ●●● ●● ● ● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●● ● ●● ● ●● ● ● ● ●● ●● ● ● ● ● ●●● ● ● ● ● ●● ● ●● ●● ●●● ● ●● ● ●●●●● ● ●●● ●● ● ● ● ●● ● ●●● ●● ●● ● ● ●●● ● ● ● ● ●●● ●● ● ● ●●● ●● ● ●● ●● ●●● ● ●● ● ●●● ● ●●● ● ●●● ●● ● ● ● ● ● ● ●● ● ●● ●●●● ●● ● ●●●● ● ●●● ●● ● ● ● ● ●●● ● ● ● ●● ●●●●● ● ● ●●●● ● ● ●●● ● ●● ●● ● ●● ● ● ● ●●● ●●●● ●● ● ● ●● ● ● ● ●● ● ● ●● ●●●●●●● ●●● ● ●●● ● ● ● ●●●● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ●●● ● ● ●●● ●●●● ● ● ● ●● ● ●● ● ● ● ●● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●●● ●●● ● ●● ● ● ● ● ●● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ●●● ● ● ●● ●● ● ● ● ● ● ● ●● ● ● ●● ●●● ● ● ● ● ●●●● ●● ● ● ● ●●● ●● ● ● ● ● ● ●●● ●● ● ● ●● ● ●●● ● ● ● ● ● ●●●● ● ● ●● ● ●● ● ● ●● ● ●● ●● ● ● ●● ● ● ●● ● ● ● ●● ●● ●● ● ● ● ● ●●● ● ●●● ● ●● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ●● ● ●● ● ● ● ●● ●●●●●● ● ● ● ●● ● ●● ● ● ● ● ●●●● ●● ●●● ●● ●● ●● ● ●●●● ●●●● ●● ● ● ●● ● ●● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ●●●● ● ●● ●● ● ● ● ●● ●● ● ● ●● ● ● ● ●● ● ● ● ● ●●●● ● ● ● ● ●● ● ●● ●● ● ●●● ● ● ● ● 4 1 2 3 4 x x (10, 5) cor=0.50, sample_cor=0.53 (10, 8) cor=0.80, sample_cor=0.81 ● ● ● ●●● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ●● ●●● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ●● ● ● ●● ● ● ●● ● ● ● ● ●●●●● ● ●●● ● ● ● ● ● ●●●● ● ● ● ●●● ● ●● ●● ● ● ●● ● ● ● ● ●●● ● ● ●● ● ● ●● ● ● ● ●● ● ● ●●● ●● ● ● ●● ●●● ● ● ●● ●●●●● ●●● ●● ● ●● ●● ●●● ● ● ● ●●● ●●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●●● ● ● ●● ● ● ● ●●● ●● ● ● ● ●● ● ● ● ●● ● ● ●●●● ●● ● ●● ●● ● ●● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ● ●● ● ● ● ●●● ● ●● ●●● ● ●● ●● ● ●● ●●● ● ● ● ●● ●●● ● ●●● ● ●● ●● ●●●●● ● ● ● ●● ● ●● ● ●● ●● ●● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ●● ●● ● ● ● ● ●● ● ● ●● ● ●● ● ● ● ● ● ●● ●● ● ● ●● ● ● ●●● ● ● ● ●● ● ●● ● ●●● ●● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ●● ●● ● ●●● ● ●●● ● ● ● ● ●● ●●● ●●● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ●● ● ● ● ●●● ●● ●● ●● ●● ● ●● ●●● ●● ●● ●● ● ● ● ●● ●● ●●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●●● ●● ●●● ●● ●● ● ● ● ●● ●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●● ● ●● ●● ●● ●● ●●● ● ●● ●●● ● ● ●●● ● ●●● ●●● ●● ●●●● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ● ●● ●● ● ● ●●● ●● ●●● ●● ● ● ● ● ● ● ●●●●●●● ● ● ●● ● ● ●● ●● ● ●● ● ● ● ●● ● ●● ●● ●●● ●● ● ● ● ●●● ● ●● ● ●● ● ● ● ● ● ●● ● ● ●●● ●● ●● ● ● ●●● ●● ● ●●● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ●●●● ● ● ● ●● ● ●●●● ●●● ● ● ● ● ●●●● ● ● ●●● ● ● ● ● ●● ● ● ● ● ●●● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ●● ●● ●● ●● ● ● ● ● ●● ●● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ●● ● ● ●● ● ●● ●● ● ●● ● ● ● ●● ● ● ● ● ● 2 3 7 6 5 4 3 ● 4 5 x 6 7 ●● ● ●● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●●● ● ●● ●●● ●● ●● ● ● ●● ● ●● ● ● ●● ●●●● ● ● ●● ●● ● ● ● ●● ●● ●● ● ●● ● ● ●● ●●● ● ●● ● ● ●●● ●●● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ●● ● ●●● ● ● ●● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●●●●● ●●● ● ●● ●● ● ●●●● ● ●●● ●● ● ● ●● ● ●● ●● ● ●● ●● ● ●●● ● ●● ● ●●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●● ●● ● ●● ●● ● ● ●●●● ●● ● ●● ●● ● ●● ● ●●●● ● ● ● ● ●● ● ● ● ● ● ●● ● ●●● ● ●●● ● ● ● ● ●● ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ●● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ●● ● ●● ● ●● ● ● ● ●● ● ● ● ● ●●● ● ●● ● ●● ●● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ●● ●● ●●●●● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ●●● ● ●● ● ● ● ● ●● ● ●● ● ● ●● ●● ● ● ● ●● ●●● ●● ● ● ● ● ● ● ●● ● ●● ●● ● ● ●● ● ● ● ●●● ●● ●● ● ● ●● ● ●● ● ● ● ● ●● ● ● ●●● ●● ●● ● ● ●● ● ● ● ● ● ●●● ● ●●●● ●● ● ●● ●● ● ●●● ●● ●● ●● ● ● ●● ● ● ● ● ● ● ●●● ● ● ●●● ● ● ● ●● ●●● ●● ● ●● ● ● ● ● ●● ● ● ● ● ● ●●● ●● ● ● ●●●● ● ●● ● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ●● ●● ● ●●●● ●● ● ●● ●● ● ●● ● ●●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ●●● ● ● ● ● ● ● ●● ● ●● ● ● ● ●● ● ●● ●● ●● ● ●● ● ●●● ●● ●● ●● ● ● ●● ●● ● ● ● ● ●● ●● ● ● ● ● ●● ● ● ●●● ● ● ●●●● ●● ●●● ● ●● ● ● ● ● ● ●●●● ● ●●● ●● ● ● ●● ●● ● ● ● ●●● ● ●● ● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● 2 2 ● ● ● ● ● ● y 7 6 5 ● ● ● 8 3 2 1 ● ●● y 4 ● ● 1.5 ● 4 ● ● ● ● (5, 1) cor=0.20, sample_cor=0.21 1 3 ● ● 1.0 x ● y ● x ● y ● ● ● ● ● 3 4 5 x 6 7 8 18.05 Class 7, Covariance and Correlation, Spring 2022 9 4 Proof of the properties of covariance and correlation 4.1 Proofs of the properties of covariance 1 and 2 follow from similar properties for expected value. 3. This is the definition of variance: Cov(𝑋, 𝑋) = 𝐸[(𝑋 − 𝜇𝑋 )(𝑋 − 𝜇𝑋 )] = 𝐸[(𝑋 − 𝜇𝑋 )2 ] = Var(𝑋). 4. Recall that 𝐸[𝑋 − 𝜇𝑥 ] = 0. So Cov(𝑋, 𝑌 ) = 𝐸[(𝑋 − 𝜇𝑋 )(𝑌 − 𝜇𝑌 )] = 𝐸[𝑋𝑌 − 𝜇𝑋 𝑌 − 𝜇𝑌 𝑋 + 𝜇𝑋 𝜇𝑌 ] = 𝐸[𝑋𝑌 ] − 𝜇𝑋 𝐸[𝑌 ] − 𝜇𝑌 𝐸[𝑋] + 𝜇𝑋 𝜇𝑌 = 𝐸[𝑋𝑌 ] − 𝜇𝑋 𝜇𝑌 − 𝜇𝑋 𝜇𝑌 + 𝜇𝑋 𝜇𝑌 = 𝐸[𝑋𝑌 ] − 𝜇𝑋 𝜇𝑌 . 5. Using properties 3 and 2 we get Var(𝑋+𝑌 ) = Cov(𝑋+𝑌 , 𝑋+𝑌 ) = Cov(𝑋, 𝑋)+2Cov(𝑋, 𝑌 )+Cov(𝑌 , 𝑌 ) = Var(𝑋)+Var(𝑌 )+2Cov(𝑋, 𝑌 ). 6. If 𝑋 and 𝑌 are independent then 𝑓(𝑥, 𝑦) = 𝑓𝑋 (𝑥)𝑓𝑌 (𝑦). Therefore Cov(𝑋, 𝑌 ) = ∫ ∫(𝑥 − 𝜇𝑋 )(𝑦 − 𝜇𝑌 )𝑓𝑋 (𝑥)𝑓𝑌 (𝑦) 𝑑𝑥 𝑑𝑦 = ∫(𝑥 − 𝜇𝑋 )𝑓𝑋 (𝑥) 𝑑𝑥 ∫(𝑦 − 𝜇𝑌 )𝑓𝑌 (𝑦) 𝑑𝑦 = 𝐸[𝑋 − 𝜇𝑋 ]𝐸[𝑌 − 𝜇𝑌 ] = 0. 4.2 Proof of Property 3 of correlation (This is for the mathematically interested.) 0 ≤ Var ( 𝑌 𝑋 𝑌 𝑋 𝑌 𝑋 ) = Var ( ) + Var ( ) − 2Cov ( ) = 2 − 2𝜌 − , 𝜎𝑋 𝜎𝑌 𝜎𝑋 𝜎𝑌 𝜎𝑋 𝜎𝑌 This implies 𝜌 ≤ 1 Likewise 0 ≤ Var ( 𝑋 𝑌 + ), so −1 ≤ 𝜌. 𝜎𝑋 𝜎𝑌 If 𝜌 = 1 then 0 = Var ( 𝑋 𝑌 𝑋 𝑌 − ) ⇒ − = 𝑐. 𝜎𝑋 𝜎𝑌 𝜎𝑋 𝜎𝑌