Multivariate Normal Distributions

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Lecture 14 Fall 2018 MULTIVARIATE NORMAL DISTRIBUTIONS Contents 1. Background on positive definite matrices 2. Definition of the multivariate normal distribution 3. Means and covariances of vector random variables 4. Key properties of the multivariate normal In an earlier lecture, we worked through the bivariate normal distribution and its properties, relying mostly on algebraic manipulation and integration of normal PDFs. Here, we revisit the subject in more generality (n dimensions), while using more elegant tools. First, some background. 1 BACKGROUND ON POSITIVE DEFINITE MATRICES. Definition 1. Let A be a square (n × n) symmetric matrix. (a) We say that A is positive definite, and write A > 0, if xT Ax > 0, for every nonzero x ∈ Rn . (b) We say that A is nonnegative definite, and write A ≥ 0, if xT Ax ≥ 0, for every x ∈ Rn . It is known (e.g., see any basic linear algebra text) that: (a) A symmetric matrix has n real eigenvalues. (b) A positive definite matrix has n real and positive eigenvalues. (c) A nonnegative definite matrix has n real and nonnegative eigenvalues. 1 (d) To each eigenvalue of a symmetric matrix, we can associate a real eigenvector. Eigenvectors associated with distinct eigenvalues are orthogonal; eigenvectors associated with repeated eigenvalues can always be taken to be orthogonal. Without loss of generality, all these eigenvectors can be normalized so that they have unit length, resulting in an orthonormal basis. (e) The above essentially states that a symmetric definite matrix becomes diagonal after a suitable orthogonal change of basis. A concise summary of the above discussion is the following spectral decomposition formula: Every symmetric matrix A can be expressed in the form A= n X i zi zTi , i=1 where 1 , . . . , n are the eigenvalues of A, and z1 , . . . , zn is an associated collection of orthonormal eigenvectors. (Note here that zi zTi is a n × n matrix, of rank 1.) For nonnegative definite matrices, we have i ≥ 0, which allows us to take square roots and define n p X B= i zi zTi . i=1 We then observe that: (a) The matrix B is symmetric. (b) We have B 2 = A (this is an easy calculation). Thus B is a symmetric square root of A. √ (c) The matrix B has eigenvalues i . Therefore, it is positive (respectively, nonnegative) definite if and only if A is positive (respectively, nonnegative) definite. Finally, if A is positive definite, then each i is positive, and we can define the matrix n X 1 C= zi zT . i i i=1 An easy calculation shows that CA = AC = I, so that C = A−1 . 2 2 DEFINITION OF THE MULTIVARIATE NORMAL DISTRIBUTION Our interest in positive definite matrices stems for the following. When A is positive definite, the quadratic form q(x) = xT Ax goes to infinity as kxk → ∞, so that e−q(x) decays to zero, as kxk → ∞, and therefore can be used to define a multivariate PDF. There are multiple ways of defining multivariate normal distributions. We will present three, and will eventually show that they are consistent with each other. The first generalizes our definition of the bivariate normal. It is the most explicit and transparent; on the downside it can lead to unpleasant algebraic manipulations. Recall that |V | stands for the absolute value of the determinant of a square matrix V . Definition 2. A random vector X has a nondegenerate (multivariate) normal distribution if it has a joint PDF of the form fX (x) = p n (x − µ)T V −1 (x − µ) 1 exp − , 2 (2ˇ)n |V | for some real vector µ and some positive definite matrix V . The second definition is constructive, which makes it operationally useful. Definition 3. A random vector X has a (multivariate) normal distribution if it can be expressed in the form X = DW + µ, for some matrix D and some real vector µ, where W is a random vector whose components are independent N (0, 1) random variables. The last definition is possibly the hardest to penetrate, but in the eyes of some, it is the most elegant. Definition 4. A random vector X has a (multivariate) normal distribution if for every real vector a, the random variable aT X is normal. 3 A brief remark on the use of the word “nondegenerate” in Definition 2. Under Definition 2, fX (x) > 0 for all x. On the other hand, consider the following example. Let X1 ∼ N (0, 1) and let X2 = 0. The random vector X = (X1 , X2 ) is normal according to Definitions 3 or 4, but cannot be described by a joint PDF (all of the probability is concentrated on the horizontal axis, a set of zero area). This is an example of a degenerate normal distribution: the distribution is concentrated on a proper subspace of Rn . The most extreme example is a one-dimensional random variable, which is identically equal to zero. This qualifies as normal under Definitions 3 and 4. One may question the wisdom of calling the number “zero” a “normal random variable;” the reason for doing so is that it allows us to state results such as “a linear function of a normal random variable is normal”, etc., without having to worry about exceptions and special conditions that will prevent degeneracy. 3 MEANS AND COVARIANCES OF VECTOR RANDOM VARIABLES Let us first introduce a bit more notation. If X = (X1 , . . . , Xn ) is a random vector, we define E[X] = (E[X1 ], . . . , E[Xn ]), which we treat as a column vector. Similarly, If A is a random matrix (a matrix with each entry being a random variable Aij ), we use the notation E[A], to denote the matrix whose entries are E[Aij ]. Given two random vectors X = (X1 , . . . , Xn ) and Y = (Y1 , . . . , Ym ), we can consider all the possible covariances Cov(Xi , Yj ) = E[(Xi − E[Xi ])(Yi − E[Yj ])], and we can arrange them in a n × m covariance matrix Cov(X, Y) whose (i, j)th entry is Cov(Xi , Yj ). It is easily checked that h i Cov(X, Y) = E (X − E[X])(Y − E[Y])T . Notice also that Cov(X, X) is a n × n symmetric matrix. Exercise 1. Prove that Cov(X, X) is nonnegative definite. 4 4 KEY PROPERTIES OF THE MULTIVARIATE NORMAL The theorem below includes almost everything useful there is to know about multivariate normals. We will prove and state the theorem, while working mostly with Definition 3. The proof of equivalence of the three definitions will be completed in the next lecture, together with some additional observations. Theorem 1. Suppose that X = (X1 , . . . , Xn ) is multivariate normal, in the sense of Definition 3, and let µi be the ith component of µ. (a) For every i, Xi is normal, with mean µi . (b) We have Cov(X, X) = DD T . (c) If C is a m × n matrix and d is a vector in Rm , then Y = CX + d is multivariate normal in the sense of Definition 3, with mean Cµ + d and covariance matrix CDD T C T . (d) If |D| 6= 0, then X is a nondegenerate multivariate normal in the sense of Definition 2, with V = DD T = Cov(X, X). (e) The joint CDF FX of X is completely determined by the mean and covariance of X. (f) The components of X are uncorrelated (i.e., the covariance matrix is diagonal) if and only if they are independent. (g) If  X Y  ∼N  µX µY   VXX , VY X VXY VY Y  , and VY Y > 0, then: (i) E[X | Y] = µX + VXY VY−1 Y (Y − µY ). (ii) Let X̃ = X − E[X | Y]. Then, X̃ is independent of Y , and independent of E[X | Y]. ˜ | Y) = Cov(X̃, X) ˜ = VXX − VXY V −1 VY X . (iii) Cov(X̃, X YY Proof: (a) Under definition 3, Xi is a linear function of independent normal random variables, hence normal. Since E[W] = 0, we have E[Xi ] = µi . (b) For simplicity, let us just consider the zero mean case. We have Cov(X, X) = E[XXT ] = E[DWWT D T ] = DE[WWT ]D T = DD T , 5 where the last equality follows because the components of W are independent (hence the covariance matrix is diagonal), with unit variance (hence the diagonal entries are all equal to 1). (c) We have Y = CX + d = C(DW + µ) + d, which is itself a linear function of independent standard normal random variables. Thus, Y is multivariate normal. The formula for E[Y] is immediate. The formula for the covariance matrix follows from part (b), with D being replaced by (CD). (d) This is an exercise in derived distributions. Let us again just consider the case of µ = 0. We already know (Lecture 10) that when X = DW , with D invertible, then fW (D −1 x) fX (x) = . | det D| In our case, since the Wi are i.i.d. N(0,1), we have n 1 o 1 fW (w) = p exp − wT w , 2 (2ˇ)n leading to fX (x) = p 1 (2ˇ)n |DD T | exp n o 1 − xT (D −1 )T D −1 x . 2 This is of the form given in Definition 2, with V = DD T . In conjunction with part (b), we also have Cov(X, X) = V . The argument for the nonzero mean case is essentially the same. (e) Using part (d), the joint PDF of X is completely determined by the matrix V , which happens to be equal to Cov(X, X), together with the vector µ. The degenerate case is a little harder, because of the absence of a convenient closed form formula. One could think of a limiting argument that involves injecting a tiny bit of noise in all directions, to make the distribution nondegenerate, and then taking the limit. This type of argument can be made to work, but will involve tedious technicalities. Instead, we will take a shortcut, based on the inversion property of transforms. This argument is simpler, but relies on the heavy machinery behind the proof of the inversion property. T Let us find the multivariate transform MX (s) = E[es x ]. We note that sT X is normal with mean sT µ. Letting X̃ = X − µ, the variance of sT X 6 is ˜ T ]s = sT Cov(X, X)s = sT V s. var(sT X) = E[sT X̃X̃T s] = sT E[X̃X Using the formula for the transform of a single normal random variable (sT X in this case), we have Tx MX (s) = E[es Tµ ] = MsT X (1) = es TV es s/2 . Thus, µ and V completely determine the transform of X. By the inversion property of transforms, µ and V completely determine the distribution (e.g., the CDF) of X. (f) If the components of X are independent they are of course uncorrelated. For the converse, suppose that the components of X are uncorrelated, i.e., the matrix V is a diagonal. Consider another random vector Y that has the same mean and as X, whose components are independent normal, and such that the variance of Yi is the same as the variance of Xi . Then, X and Y have the same mean and covariance. By part (e), X and Y have the same distribution. Since the components of Y are independent, it follows that the components of X are also independent. For the special case when V is invertible, we could alternatively use part (d) which provides an explicit formula for the joint PDF of X. When V is diagonal we see that the joint PDF is the product of its marginal PDF. Namely fX (x) = fX1 (x1 ) · · · fXn (xn ) and thus the components of X are independent. (g) Once more, to simplify notation, let us just deal with the zero-mean case. Let us define X̂ = VXY VY−1 Y Y. We then have ˆ T ] = VXY V −1 E[YY T ] = VXY = E[XY T ]. E[XY YY This proves that X − X̂ is uncorrelated with Y. Note that (X − X̂, Y) is a linear function of (X, Y), so, by part (c), it is also multivariate normal. Using an argument similar to the one in the proof of part (f), we conclude that X − X̂ is independent of Y, and therefore independent from any function of Y. Recall now the abstract definition of conditional expectations. The relation E[(X − X̂)g(Y)] = 0, for every function g, implies that X̂ = E[X | Y], which proves part (i). 7 For part (ii), note that we already proved that X̃ = X − X̂ = X − E[X | Y] is independent of Y. Since E[X | Y] is a function of Y, it follows that X̃ is independent of E[X | Y]. ˜ | For part (iii), note that X̃ is independent of Y, which implies that Cov(X̃, X Y) = Cov(X̃, X̃). Finally, ˜ = E[X̃X̃T ] = E[(X − X̂)(X − X̂)T ] = E[(X − X̂)XT ] Cov(X̃, X) T −1 T = VXX − E[VXY VY−1 Y YX ] = VXX − VXY VY Y E[YX ] = VXX − VXY VY−1 Y VY X . Note that in the case of the bivariate normal, we have Cov(X, Y ) = ˆ˙X ˙Y , 2 ,V 2 VXX = ˙X Y Y = ˙Y . Then, part (g) of the preceding theorem, for the zeromean case, reduces to E[X | Y ] = ˆ ˙X Y, ˙Y 2 var(X̃) = ˙X (1 − ˆ2 ), which agrees with the formula we derived through elementary means in Lecture 9, for the special case of unit variances. 8