Expectation and Variance, Covariance and Correlation, and Movement Generating Functions

9.07 Introduction to Probability and Statistics for Brain and Cognitive Sciences Emery N. Brown Lecture 6: Expectation and Variance, Covariance and Correlation, And Movement Generating Functions I. Objectives Understand how to compute expectations and variances for linear combinations of random variables. Understand how to compute the covariance between linear combinations of random variables. Understand the concept of moments of a probability density. Understand how the moments of a probability density or probability mass function can be derived from the moment generating function. Understand the basic properties of moment generating functions and their use in probability calculations. II. Expectations and Covariances A. Expectation In Lectures 2 and 3, we defined the expected value of a discrete and continuous random variable respectively as 1. Discrete If X : p x ( x) (6.1) then E( X ) = provided that ∑ xi px ( xi ) (6.) ∑| xi | px ( xi ) < ∞. 2. Continuous If X : f x ( x) (6.3) then E ( x) = provided that ∞ ∫−∞ x f x ( x)dx ∫ | x | f x ( x)dx < ∞. (6.4) page 2: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions Remark 6.1. The expected value has an important physical interpretation as the center of mass or the first moment. We also stated that if Y = g ( X ), we can obtain the expected value of Y directly without first computing p y ( y) or f y ( y). That is, given Y = g ( X ) 1. Discrete If X : px ( x) then E (Y ) = ∑ g ( x) p x ( x) (6.5) x provided that If ∑ | g ( x) | p( x) < ∞. 2. Continuous X : f x ( x) then ∞ E (Y ) = provided that ∫−∞ g (x) f x (x)dx (6.5) ∑ yi p y ( yi ). (6.6) ∫ | g ( x) | f x ( x)dx < ∞. Proof (Discrete Case) E (Y ) = i Let Ai be the x's mapped into by y by g. We have x ∈ Ai if g ( x) = yi . Hence, p y ( yi ) = ∑ px ( x) x∈Ai E (Y ) = ∑ yi ∑ px ( x) i = x∈Ai ∑ ∑ yi px ( x) (6.7) i x∈Ai = ∑ ∑ g ( x) px ( x) i x∈Ai = ∑ g ( x) px ( x). x Remark 6.2. A very important function g ( x) whose expected value we considered is g ( x) = ( x − µ x ) 2 which is the variance σ x2 = E ( g ( X )) = E ( X − µ x )2 = E[ X 2 − 2X µ x + µ x 2 ] = EX 2 − µ x2 . of X. We know that page 3: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions 1 The standard deviation of X is σ x = [EX 2 − µ x2 ] 2 . In general, if X1,..., X n are jointly distributed random variables and Y = g ( X1,..., X n ) then 1. Discrete If X1, X 2 ,..., X n : px ( X1, X 2 ,..., X n ) then E (Y ) = provided that ∑ g ( x1,..., xn ) px ( x1,..., xn ) (6.8) ∑ | g ( x1,..., xn ) | px ( x1,..., xn ) < ∞. 2. Continuous If X1, X 2 ,..., X n : f x ( x1,..., xn ) then ∫ E (Y ) = g ( x1,..., xn ) f (x1,..., xn )dx1,..., dxn (6.9) provided that E (| Y |) < ∞. Remark 6.3. If X and Y are independent and g and h are fixed functions, then E[ g ( X )h(Y )] = E[ g ( X )]E[h(Y )] provided that each of the expectations on the right is finite. B. Expectations of Linear Combinations of Random Variables In Lectures 2 and 3 we stated that if Y = aX + b, then E(Y ) = aE( X ) + b. That is, expectation is a linear operation. We now state a very important generalization of this result. Proposition 6.1. If X1,..., X n are jointly distributed random variables with expectations E( X i ) and Y is a linear function of the X i 's, n Y =a+ ∑ bi E( X i ). (6.10) i=1 Simply stated, the expectation of the linear combination of a set of random variables is the linear combination of the expectations provided that each expectation is finite. Proof: (see Rice, p. 125). Example 2.1 (continued). The rat executed 40 trials of a learning experiment with a probability p of a correct response on each trial. What is the expected number of correct responses? Each trial is a Bernoulli experiment with probability p of the animal executing it correctly. Let X i be 40 the outcome on trial i, then Y = ∑ Xi i=1 Hence, is the total number of correct responses on each trial. page 4: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions 40 E (Y ) = E( ∑ Xi ) i=1 40 = ∑ E( X i ) (6.11) i=1 40 = ∑ p = 40 p. i=1 In general, if there were n trials E(Y ) = np. We showed this result before in Lecture 2 using the definition of expectation directly and the binomial theorem. Example 5.4 (continued). Suppose that the number of channel openings in a 500 msec time interval at each of n sites at the frog neuromuscular junction is a Poisson random variable. Each site has an expected number of openings λi per millisecond for i = 1,..., n. What is the expected number of openings in 500 msec across all n sites? We have X i : P(λi ) (6.12) E ( X i ) = λi and in 500 msec we have Zi = 500X i (6.13) E ( Zi ) = 500λi and n Y= ∑ Zi i=1 N E (Y ) = E( (6.14) ∑ Zi ) i=1 n = ∑ i=1 n ∑ λi . E ( Zi ) = 500 i=1 Example 5.4 (continued). We assumed that the length of time that each of two channels is open is an exponential random variable with parameters λ1 and λ2 respectively. What is the expected sum of the opening times at the two sites? We have X i : E(λi ) i = 1, 2 and E ( X i ) = λi−1. Hence, Z = X1 + X 2 and E ( Z ) = E ( X1 + X 2 ) = λ1−1 + λ2−1. Notice that when λ1 = λ2 = λ we have E (Z ) = 2λ −1. This is completely consistent with our observation that when λ1 = λ2 = λ, Z : Γ(2, λ ), which we showed in Lecture 5. page 5: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions III. Covariance and Correlation A. Covariance Example 4.6 (continued). In this example, we saw that positive values on one sensor were associated with positive values on the other sensor and negative values on one sensor were associated with negative values on the other sensor. The values seemed to be related. We formalize the characterization of this relationship in terms of the concept of covariance and correlation. The latter we have already discussed in terms of the correlation coefficient. Definition 6.1. If X and Y are jointly distributed random variables with E( X ) = µx and E(Y ) = µ y , then the covariance of X and Y is Cov( X , Y ) = E ( X − µ x )(Y − µ y ) = ∞ ∞ ∫−∞ ∫−∞ (X − µ x )(Y −µ y ) f xy ( x, y)dxdy. (6.15) Remark 6.4. The covariance of X and Y measures the strength of the association between X and Y in the sense mentioned above in Example 4.6. It is the first cross moment of X and Y. It has units of the random variables squared, assuming X and Y measure the same thing. Remark 6.5. If X = Y , we have Cov( X , Y ) = E ( X − µ x )(Y − µ y ) = E ( X − µ x )2 (6.16) = σ x2 . Therefore, the covariance of X with itself is simply the variance of X . Remark 6.6. Variance is the second central moment, and E ( X 2 ) is the second moment. The variance tells us about how mass is distributed away from the mean. We report the standard deviation because σ x is on the same scale (units) as X . Remark 6.7. We have on expanding the terms in Definition 6.1 Cov( X , Y ) = E ( X − µ x )(Y − µ y ) = E ( XY − µ xY − µ y X + µ x µ y ) = E ( XY ) − µ x E (Y ) − µ y E ( X ) + µ x µ y (6.17) = E ( XY ) − µ x µ y − µ y µ x + µ x µ y = E ( XY ) − E ( X ) E (Y ). Example 6.1 (Rice p. 138). Suppose X and Y have the joint probability density f ( x, y ) = (2x + 2 y − 4 xy ) where x ∈(0,1) and y ∈ (0,1). (6.18) page 6: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions Find Cov( X , Y ). First it is easy to show that E ( X ) = E (Y ) = E( X ) = 1 1 1 2 1 ∫0 ∫0 x(2 x + 2 y − 4 xy)dydx = 2 (6.19) E (Y ) = 1 1 1 ∫0 ∫0 y(2 x + 2 y − 4 xy)dxdy = 2 We need to compute E ( XY ) = 1 1 2 ∫0 ∫0 xy(2x + 2 y − 4xy)dxdy = 9 . (6.20) Hence, Cov( X , Y ) = E ( XY ) − E ( X ) E (Y ) = 2 1 2 1 −( ) = − . 9 2 36 (6.21) Here are 4 useful results. Result 6.1 Cov(a + X , Y ) Cov(a + X , Y ) = E ((a + X )Y ) − E (a + X ) E (Y ) = E (aY ) + E ( XY ) − [ E (a ) + E ( X )]E (Y ) = aE (Y ) + E ( XY ) − E (a ) E (Y ) − E ( X ) E (Y ) (6.22) = E ( XY ) − E ( X ) E (Y ) + aE (Y ) − aE (Y ) = cov( X , Y ) Result 6.2 Cov(aX , bY ) Cov(aX , bY ) = E (aXbY ) − E (aX ) E (bY ) = abE ( XY ) − abE ( X ) E (Y ) = ab cov( X , Y ) (6.23) Result 6.3 Cov( X , Y + Z) Cov( X , Y + Z ) = E ( XY + Z ) − E ( X ) E (Y + Z ) = E ( XY + XZ ) − E ( X )[ E (Y ) + E ( Z )] = E ( XY ) + E ( XZ ) − E ( X ) E (Y ) − E ( X ) E ( Z ) = E ( XY ) − E ( X ) E (Y ) + E ( XZ ) − E ( X ) E ( Z ) = cov( X , Y ) + cov( X , Z ) (6.24) page 7: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions Result 6.4 Cov(aW + bX + cY + dZ) Cov(aW + bX , cY + dZ) = Cov(aW + bX , cY ) + Cov(aW + bX , dZ ) = Cov(aW , cY ) + cov(bX , cY ) + Cov(aW , dZ ) + Cov(bX , dZ ) (6,25) = acCov(W , Y ) + bcCov( X , Y ) + adCov(WZ ) + bdCov( X , Z ). We can state the following general result I Proposition 6.2. Let U = a0 + J ∑ ai X i V = b0 + i=1 ∑b j X j j=1 Then I Cov(U ,V ) = J ∑∑ aib j Cov( X i , X j ). (6.26) i=1 j=1 We will be more interested in the following special cases i) Cov( X + Y , X + Y ) = Cov( X , X ) + Cov(Y ,Y ) + 2Cov( X ,Y ) (6.27) = Var( X ) + Var(Y ) + 2Cov( X ,Y ) ii) Cov( X − Y , X − Y ) = Var( X ) + Var(Y ) − 2Cov( X ,Y) (6.28) I iii) If U = a + ∑ ai X i and X i are independent then i=1 I Var(U ) = Cov(U ,U ) = Cov(a + ∑ I ai X i , a + i=1 I = ∑a j X j ) i=1 I (6.29) ∑∑ ai a j cov( X i X j ). i=1 j=1 If i ≠ j Cov( X i , X j ) = 0 because the X i are independent. If i = j Cov( X i , X j ) = Var( X i ). Thus, I Var(U ) = I ∑ ai2 Var( X i ) = ∑ ai2σ i2 . i=1 (6.30) i=1 This result is to some extent the analog for variances of the result for expectations established in Proposition 6.1. We required independence in the variance calculations. page 8: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions Example 2.1 (continued). Here X : B(n, p) where n = 40. What is the variance of X ? We can write 40 X = ∑ Yi (6.31) i=1 where Yi : Bernoulli with probability of a correct response p. If we assume (as we do) that the Yi 's are independent, then by Proposition 6.2 iii 40 Var( X ) = ∑ 40 Var(Yi ) = i=1 ∑ p(1 − p) = 40 p(1 − p). (6.32) i=1 In general, for n we have Var( X ) = np(1 − p). We did not show this before because it is very tedious to do directly from the definition of the variance. Example 6.2. Integrate and Fire Model as a Random Walk + Drift As we discussed in Lecture 3, an elementary stochastic model for the subthreshold membrane voltage of a neuron is the random walk with drift model proposed by Gerstein and Mandelbrot (1964). In continuous time, it is V (t ) = µ t + σ t ∫t dW (u) (6.33) 0 where W (t ) is a Weiner process, i.e. a continuous time Gaussian random variable with mean 0 and variance 1. We assume E(W (s)W (t )) = 0 for s ≠ t. To simulate this on the computer we rewrite it in discrete time as V (ti ) = µti + σ i ∑Wi (6.34) j=0 where Wi : N(0,1). Find the variance of V (ti ). Applying Proposition 6.2 iii we get Var(V (ti )) = Var( µ ti + σ i ∑Wi ) j=0 =σ2 i ∑ Var(Wi ) (6.35) j=0 = σ 2 (i + 1). This is a restatement of the well-known classic result that the variance of a random walk is proportional to the length of the walk. Uncertainty increases with time. Notice however, that page 9: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions E (V (ti )) = E( µti + σ i ∑ Wi ) = E ( µti ) + σ j=0 i ∑ E(Wi ) = µti + 0 j=0 (6.36) = µti . On average the random walk with drift will be on the course of linear increase in membrane voltage. (see Gerstein GL, Mandelbrot M. Random walk models for the spiking activity of a single neuron. Biophysical Journal 4:41-68, 1964). B. Correlation Definition 6.2. If X and Y are jointly distributed random variables and the variances and covariances of X and Y are non-zero, then the correlation of X and Y is ρ= Cov( X , Y ) 1 2 [Var( X )Var(Y )] = Cov( X , Y ) σ xσ y (6.37) . 1 1 and we computed Cov( X , Y ) = − . 2 36 1 Because X and Y are both uniform on (0,1) it is easy to show that Var( X ) = Var(Y ) = . Now, 12 Example 6.1 (continued). Find ρ . We have E ( X ) = E (Y ) = we have 1 1 − 1 36 36 ρ= = =− . 1 1 12 1 3 ( × ) 12 12 12 − (6.38) We now establish an important property of the correlation coefficient. Proposition 6.3. −1 ≤ ρ ≤ 1. ρ = ±1 if and only if Pr(Y = a + bX ) = 1 for some constants a and b. Proof: 0 ≤ Var( = = X + σx Var( X ) + σx Var( X ) + σ x2 1 Y σy ) Var(Y ) σy Var(Y ) σ 2y + 1 + 2 Cov( +2 + X , Y σx σx Cov( X ,Y ) σ xσ y ) (6.39) 2ρ = 2(1 + ρ ) Or 2(1 + ρ ) ≥ 0 ⇒1+ ρ ≥ 0 or ρ ≥ −1. Also 0 ≤ Var( X σx − X σy ) = 2(1 − ρ ) (6.40) page 10: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions Or 2(1 − ρ ) ≥ 0 ⇒ (1 − ρ ) ≥ 0 1 ≥ ρ. To prove the last statement of the proposition requires Chebyshev’s inequality which we will prove in Lecture 7. Remark 6.8. Correlation provides a normalized (scale-free) measure of covariance. Example 4.6 (continued). We previously stated in Lecture 4 that ρ was the correlation coefficient for a bivariate Gaussian probability density. Now we demonstrate this by showing that cov( X ,Y ) = ρσ xσ y . Cov( X , Y ) = ∞ ∞ ∫−∞ ∫−∞ (x − µx )( y −µ y ) f ( x, y)dxdy. (6.41) Make the change of variables u = (x − µx ) / σ x , v = (y − µ y ) / σ y and we have = σ xσ y ⎧⎪ 1 1 ⎫⎪ 2 2 uv exp ⎨− (u + v − 2 ρ uv) dudv ⎬ 2 −∞ ⎪⎩ 2 (1− ρ ) ⎭⎪ ∞ 2π (1 − ρ 2 ) 1 2 ∫−∞ ∫ ∞ (6.42) completing the square = σ xσ y 1 2π (1 − ρ 2 ) 2 ⎧⎪ v 2 ⎫⎛ ⎧⎪ ⎫⎪ ⎞ 1 ⎪ ∞ v exp ⎨ ⎬ ⎜ u exp ⎨− (u − ρ v) 2 ⎬ du ⎟ du , 2 ⎟ −∞ ⎪⎩ 2(1 − ρ ) ⎪⎩ 2 ⎪⎭⎝⎜ −∞ ⎭⎪ ⎠ ∫ ∞ ∫ (6.43) 1 1 The inner integral is E (u ) × [2π (1 − ρ 2 )] 2 = ρ v[2π (1− ρ 2 )] 2 1 = ρσ xσ y [2π (1 − ρ 2 )] 2 1 2π (1 − ρ 2 ) 2 × 2 ⎪⎧ v ⎫⎪ v 2 exp ⎨ ⎬dv −∞ ⎪⎩ 2 ⎪⎭ ∫ ∞ 1 = ρσ xσ y (2π ) 2 2π = ρσ xσ y . 1 (2π ) 2 E (v) (6.44) IV. Moment Generating Functions “Learning to use moment generating functions in probability calculations is like going from using a PC to using a supercomputer.” -- E.N. Brown As we will see, the moment generating function simplifies many of the calculations we have already done and will enable many more, the most important of which is the proof of the Central Limit Theorem. page 11: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions Definition 6.3. The r th moment of a random variable X is E( X r ) = ∞ ∫−∞ x r (6.45) f x ( x)dx provided the expectation exists. The r th central moment of a random variable X is E{[ X − E ( X )]r } = ∞ ∫−∞ (x − E( x)) r f x ( x)dx (6.46) provided that the expectation exists. Remark 6.9. Some key factoids about moments E( X ) = µ 1st moment E( X 2 ) 2nd moment E ( X − µ ) 2 = σ 2 2nd central moment (variance) E ( X − µ )3 3rd central moment E ( X − µ )4 4 th central moment The skewness measures the asymmetry of the distribution Figure 6.A. Examples of Negative (left panel) and positive (right) skew in a probability density. It is defined is page 12: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions E ( X − µ )3 (6.47) σ3 The skewness of standard Gaussian distribution is 0. The kurtosis measures how “fat” the tails of a distribution is often reported as E ( X − µ )4 (6.48) σ4 or to make a comparison relative to a standard Gaussian distribution as E ( X − µ )4 σ4 −3 (6.49) because for kurtosis for the standard Gaussian distribution is 3. Definition 6.4. The moment generating function (mgf) of a random variable X is E (etx ) which is in the Discrete Case M x (t ) = ∑ etx px ( x) (6.50) x and is in the Continuous Case M x (t ) = ∞ ∫−∞ e tx f x ( x)dx (6.51) provided the expectations exist. This depends critically on how fast the tails in the probability density “die out”. Result 6.5. If M x (t ) exists in an open interval containing zero it uniquely determines f x ( x). This is an important, deep result which allows us to perform computations on M x (t ) to establish properties of X and f x ( x). For example, if we differentiate M (t ) we obtain M '(t ) = and d dt ∞ tx d tx ∫−∞ e ∞ = ∫−∞ dt e = ∫−∞ xe ∞ tx f x ( x)dx f x ( x)dx f x ( x )dx (6.52) page 13: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions ∞ ∫−∞ xf x ( x)dx = E( X ) M '(0) = (6.53) Differentiating r times and evaluating the derivative at zero gives M (r ) (0) = E ( X r ). (6.54) Result 6.6. If M (t ) exists in an open interval containing 0, then M (r ) (0) = E ( X r ). In this sense M (t ) can be used to “generate” the moments of X . Example 6.3. Poisson MGF M (t ) = = ∞ ∑ etk λ k −λ e k! ∑ ( λ et )t − λ e k! k =0 ∞ k =0 = e−λ ∞ ∑ k =0 (6.55) t t ( λ et ) k = e−λ eλ e = eλ (e −1) . k! Then sum averages for all t. Differentiating, we have M '(t ) = λ et eλ (e −1) t M ''(t ) = λ et eλ (e t−1 ) + λ 2e2t (eλ (e t−1 ) ) (6.56) M '(0) and λ ⇒ E (k ) = λ M ''(0) = λ + λ 2 and E (k 2 ) = λ 2 + λ or E (k 2 ) −[ E (k )]2 = λ 2 + λ − λ 2 = λ. Example 6.4 Gamma MGF M (t ) = ∞ etx β α ∫0 Γ(α ) xα −1e− β x dx = βα Γ(α ) = β α Γ(α ) βα = Γ(α ) ( β − t)α ( β − t)α ∞ ∫0 xα −1e(t− β )x dx where we require t < β for the integral to exist (6.57) page 14: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions M '(t ) = αβ α ( β − t) −(α +1) ⇒ M '(0) = E ( X ) = α β (6.58) M ''(t ) = α (α +1) β α ( β − t) −(α +2) ⇒ M ''(0) = E ( X 2 ) = and Var( X ) = E ( X 2 ) −[ E ( X )]2 = α (α +1) β2 2 α (α +1) ⎛ α ⎞ α −⎜ ⎟ = . 2 β2 β ⎝β ⎠ Example 6.5. Standard Gaussian MGF M (t ) = (2π )− 2 1 ∞ ∫−∞ e 2 tx − x2 e (6.59) dx Complete the square to rewrite 1 1 x2 − tx = ( x 2 − 2tx + t 2 ) − t 2 2 2 2 (6.60) and hence, t2 M (t ) = ∞ e2 1 (2π ) 2 ∫−∞ e − ( x −t ) 2 2 dx. (6.61) Let u = x − t then dx = du t2 = ∞ e2 (2π ) 1 2 ∫−∞ u2 2 e du = t2 e2 (2π ) 1 1 2 t2 (2π ) 2 = e 2 . (6.62) Result 6.7 (Linear Transformation). If X has mgf M x (t ) and Y = a + bX , then Y has mgf M y (t ) = eat M x (bt ). Proof: M Y (t ) = E (etY ) = E (eat +btX ) (6.63) = E (eat ebtX ) = eat E (ebtX ) = eat M x (bt ) Example 6.5 (continued). Let Y = µ + σ X where X : N (0,1) then M Y (t ) = e µt M x (σ t ) = e µt eσ 2 t2 2 . Result 6.8 (Convolution of Independent Random Variables) If X and Y are independent with mgf’s M x (t ) and M y (t ) respectively, and Z = X + Y , then M z (t ) = M x (t )M y (t ) on the common interval where both mgf’s exist. page 15: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions M z (t ) = E (etZ ) = E (et ( X +Y ) ) = E (etX etY ) tX (6.64) tY = E (e ) E (e ) = M x (t ) M y (t ) by independence. Example 5.4 (continued). We had X : Exponential(λ1 ) Y : Exponential (λ2 ) and Z = X + Y . We showed that if λ1 = λ2 = λ and X and Y are independent, then Z : Γ(α = 2, β = λ ). We now establish the same result using mgf’s (Example 6.4 and Result 6.8). M z (t ) = M x (t )M y (t ) λ =( λ )( λ −t λ −t λ 2 =( ) λ −t (6.65) ) which is the mgf of a gamma random variable with α = 2 and β = λ. Example 6.6. X : N (µ x , σ x2 ) Y : N (µ y , σ 2y ) and X and Y are independent. Find the distribution of Z = X + Y. By Example 6.5, M z (t ) = M x (t )M y (t ) t2 t2 2 σ x2 µ σy = eµx e 2 e y e 2 (6.66) 2 =e Hence, Z : N(µ x + µ y , σ x2 2 2 t ( µ x + µ y ) (σ x +σ y ) 2 e + σ y2 ). Remark 6.10. There is an obvious generalization of Result 6.8, namely if X1,..., X n are i =1, ..., n, and Z = independent random variables with mgf’s M x (t ) i n ∑ X i , then i=1 n M z (t ) = ∏ M x (t ). i (6.67) i=1 The X i 's do not have to come from the same distribution. Remark 6.11. In general, if two random variables follow the same distribution the sum will not necessarily follow the same distribution. See Example 5.4, 5.5 and consider the case of two independent gamma random variables with β1 = β 2 . Remark 6.12. Result 6.8 is the special case of the well-known mathematical fact that the convolution of two functions in this case probability density functions in the probability (time) page 16: Lecture 6: Expectations and Variances, Covariances and Correlation, Moment Generating Functions domain corresponds to multiplying the transforms in the t (frequency) domain. This result is exploited a lot in linear systems theory and Fourier analysis. Remark 6.13. There is a multivariate generalization of the mgf which we will not consider. Remark 6.14. The limitation that the mgf may not exist is rectified by working instead with the characteristic function φ x (t ) = E (eitX ) = ∞ ∫−∞ eitx f x ( x)dx (6.68) 1 where i = (−1) 2 . The integral converges for all t because | eitx |≤ 1. The characteristic function has the same properties of the mgf without the existence limitation. Its use requires familiarity with complex numbers and their manipulations. V. Summary We can now perform basic probability calculations on linear combinations of random variables. These results have now established the essential groundwork for proving the Central Limit Theorem in Lecture 7. Acknowledgments I am grateful to Julie Scott for technical assistance in preparing this lecture. Reference Rice JA. Mathematical Statistics and Data Analysis, 3rd edition. Boston, MA, 2007.