Multivariate Normal Distribution and Multivariate CLT

Lecture 4 Multivariate normal distribution and multivariate CLT. We start with several simple observations. If X = (x1 , . . . , xk )T is a k × 1 random vector then its expectation is EX = (Ex1 , . . . , Exk )T and its covariance matrix is Cov(X) = E(X − EX)(X − EX)T . Notice that a covariance matrix is always symmetric Cov(X)T = Cov(X) and nonnegative definite, i.e. for any k × 1 vector a, aT Cov(X)a = EaT (X − EX)(X − EX)T aT = E|aT (X − EX)|2 � 0. We will often use that for any vector X its squared length can be written as |X|2 = X T X. If we multiply a random k × 1 vector X by a n × k matrix A then the covariance of Y = AX is a n × n matrix Cov(Y ) = EA(X − EX)(X − EX)T AT = ACov(X)AT . Multivariate normal distribution. Let us consider a k × 1 vector g = (g1 , . . . , gk )T of i.i.d. standard normal random variables. The covariance of g is, obviously, a k × k identity matrix, Cov(g) = I. Given a n × k matrix A, the covariance of Ag is a n × n matrix � := Cov(Ag) = AIAT = AAT . Definition. The distribution of a vector Ag is called a (multivariate) normal distribution with covariance � and is denoted N (0, �). One can also shift this disrtibution, the distribution of Ag + a is called a normal distri­ bution with mean a and covariance � and is denoted N (a, �). There is one potential problem with the above definition - we assume that the distribution depends only on covariance ma­ trix � and does not depend on the construction, i.e. the choice of g and a matrix A. For example, if we take a m × 1 vector g → of i.i.d. standard normal random variables and a n × m matrix B then the covariance of Bg → is a n × n matrix Cov(Bg → ) = BB T . It is possible that � = AAT = BB T so both constructions should give a normal distribution N (0, �). This is, indeed, true - the distribution of Ag and Bg → is the same, so the definition of normal distribution N (0, �) does not depend on the construction. It is not very difficult to prove that Ag and Bg → have the same distribution, but we will only show the simplest case. Invertible case. Suppose that A and B are both square n × n invertible matrices. In this case, vectors Ag and Bg → have density which we will now compute. Since the density of g is � 1 � � 1 �n � 1 � � 1 2 → exp − x2i = → exp − |x| , 2 2 2� 2� i�n for any set � ∞ Rn we can write −1 P(Ag ∞ �) = P(g ∞ A �) = � � 1 �n � 1 � → exp − |x|2 dx. 2 2� A−1 � Let us now make the change of variables y = Ax or x = A−1 y. Then � � � 1 � 1 �n 1 −1 2 → P(Ag ∞ �) = exp − |A y| dy. 2 | det(A)| 2� � But since det(�) = det(AAT ) = det(A) det(AT ) = det(A)2 � we have | det(A)| = det(�). Also |A−1 y|2 = (A−1 y)T (A−1 y) = y T (AT )−1 A−1 y = y T (AAT )−1 y = y T �−1 y. Therefore, we get � � � 1 � 1 1 �n T −1 � → P(Ag ∞ �) = exp − y � y dy. 2 2� det(�) � This means that a vector Ag has the density � 1 � 1 � exp − y T �−1 y 2 det(�) which depends only on � and not on A. This means that Ag and Bg → have the same distri­ butions. It is not difficult to show that in a general case the distribution of Ag depends only on the covariance �, but we will omit this here. Many times in these lectures whenever we want to represent a normal distribution N(0, �) constructively, we will find a matrix A (not necessarily square) such that � = AAT and use the fact that a vector Ag for i.i.d. vector g has normal distribution N(0, �). One way to find such A is to take a matrix square-root of �. Since � is a symmetric nonnegative definite matrix, its eigenvalue decomposition is � = QDQT for an orthogonal matrix Q and a diagonal matrix D with eigenvalues �1 , . . . , �n of � on the diagonal. In Matlab, ’[Q,D]=eig(Sigma);’ will produce this decomposition. Then if D 1/2 1/2 represents a diagonal matrix with �i on the diagonal then one can take A = QD 1/2 or A = QD 1/2 QT . It is easy to check that in both cases AAT = QDQT = �. In Matlab QD 1/2 QT is given by ’sqrtm(Sigma)’. Let us take, for example, a vector X = QD 1/2 g for i.i.d. standard normal vector g which by definition has normal distribution N (0, �). If q1 , . . . , qn are the column vectors of Q then 1/2 X = QD 1/2 g = (�1 g1 )q1 + . . . + (�1n/2 gn )qn . Therefore, in the orthonormal coordinate basis q1 , . . . , qn a random vector X has coordi­ 1/2 1/2 nates �1 g1 , . . . , �n gn . These coordinates are independent with normal distributions with variances �1 , . . . , �n correspondingly. When det � = 0, i.e. � is not invertible, some of its eigenvalues will be zero, say, �k+1 = . . . = �n = 0. Then the random vector will be concen­ trated on the subspace spanned by vectors q1 , . . . , qk but it will not have density on the entire space Rn . On the subspace spanned by vectors q1 , . . . , qk a vector X will have a density f (x1 , . . . , xk ) = k � i=1 → � x2 � 1 exp − i . 2�i 2��i Linear transformation of a normal random vector. Suppose that Y is a n × 1 random vector with normal distribution N(0, �). Then given a m × n matrix M, a m × 1 vector MY will also have normal distribution N (0, M�M T ). To show this, find any matrix A and i.i.d. standard normal vector g such that Ag has normal distribution N (0, �). Then, by definition, M(Ag) = (MA)g also has normal distribution with covariance (M A)(MA)T = MAAT M T = M�M T . Orthogonal transformation of an i.i.d. standard normal sample. Throughout the lectures we will often use the following simple fact. Consider a vector X = (X1 , . . . , Xn )T of i.i.d. random variables with standard normal distribution N(0, 1). If V is an orthogonal n × n matrix then the vector Y := V X also consists of i.i.d. random variables Y1 , . . . , Yn with standard normal distribution. A matrix V is orthogonal when one of the following equivalent properties hold: 1. 2. 3. 4. V −1 = V T . The rows of V form an orthonormal basis in Rn . The columns of V form an orthonormal basis in Rn . For any x ∞ Rn we have |V x| = |x|, i.e. V preserves the lengths of vectors. Below we will use that | det(V )| = 1. Basically, orthogonal transformations represent linear transformations that preserve distances between points, such as rotations and reflections. The joint p.d.f of a vector X is given by n � 1 1 2 2 → e−xi /2 = → f (x) = f (x1 , . . . , xn ) = e−|x| /2 , n 2� ( 2�) i=1 where |x|2 = x21 + . . . + x2n . To find the p.d.f. of a vector Y = V X, which is an linear transformation of X, we can use the change of density formula from probability or the change of variables formula from calculus as follows. For any set � ≥ Rn , P(Y ∞ �) = P(V X ∞ �) = P(X ∞ V −1 �) � � f (V −1 y) = f (x)dx = dy. V −1 � � | det(V )| where we made the change of variables y = V x. We know that | det(V )| = 1 and, since |V −1 y| = |y|, we have 1 1 −1 2 2 f (V −1 y) = → e−|V y| /2 = → e−|y| /2 = f (y). ( 2�)n ( 2�)n Therefore, we finally get that P(Y ∞ �) = � f (y)dy � which proves that a vector Y has the same joint p.d.f. as X. Multivariate CLT. We will state a multivariate Central Limit Theorem without a proof. Suppose that X = (x1 , . . . , xk )T is a random vector with covariance �. We assumed that Ex2i < �. If X1 , X2 , . . . is a sequence of i.i.d. copies of X then n 1 � Sn := → (Xi − EXi ) �d N(0, �), n i=1 where convergence in distribution �d means that for any set � ∞ Rk , lim P(Sn ∞ �) = P(Y ∞ �) n�� for a random vector Y with normal distribution N(0, �).