Lecture 1 Notes: Distributions and Normal Random Variables

Lecture 1 Distributions and Normal Random Variables 1 Random variables 1.1 Basic De˝nitions Given a random variable X , we de˝ne a cumulative distribution function (cdf ), FX : R → [0, 1], such that FX (t) = P {X ≤ t} for all t ∈ R. Here P {X ≤ t} denotes the probability that X ≤ t. To emphasize that random variable X has cdf FX , we write X ∼ FX . Note that FX (t) is a nondecreasing function of t. There are 3 types of random variables: discrete, continuous, and mixed. Discrete random variable, X , is characterized by a list of possible values, X = {x1 , ..., xn }, and their probabilities, p = {p1 , ..., pn }, where pi denotes the probability that X will take value xi , i.e. pi = P {X = xi } for all i = 1, ..., n. Note that p1 + ... + pn = 1 and pi ≥ 0 for all i = 1, ..., n by de˝nition of probability. Then ∑ the cdf of X is given by FX (t) = j=1,...,n: xj ≤t pj . Continuous random variable, Y , is characterized by its probability density function (pdf), fY : R → R, ∫ ∫ +∞ such that P {a < Y ≤ b} = ab fY (s)ds. Note that −∞ fY (s)ds = 1 and fY (s) ≥ 0 for all s ∈ R by ∫t de˝nition of probability. Then the cdf of Y is given by FY (t) = −∞ fY (s)ds. By the Fundamental Theorem of Calculus, fY (t) = dFY (t)/dt. A random variable is referred to as mixed if it is not discrete and not continuous. If cdf F of some random variable X is strictly increasing and continuous then it has inverse, q(x) = −1 F (x). It is de˝ned for all x ∈ (0, 1). Note that P {X ≤ q(x)} = P {X ≤ F −1 (x)} = F (F −1 (x)) = x for all x ∈ (0, 1). Therefore q(x) is called the x-quantile of X . It is such a number that random variable X takes a value smaller or equal to this number with probability x. If F is not strictly increasing or continuous, then we de˝ne q(x) as a generalized inverse of F , i.e. q(x) = inf{t ∈ R : F (t) ≥ x} for all x ∈ (0, 1). In other words, q(x) is a number such that F (q(x) + ε) ≥ x and F (q(x) − ε) < x for any ε > 0. As an exercise, check that P {X ≤ q(x)} ≥ x. 1 1.2 Functions of Random Variables Suppose we have random variable X and function g : R → R. Then we can de˝ne another random variable Y = g(X). The cdf of Y can be calculated as follows FY (t) = P {Y ≤ t} = P {g(X) ≤ t} = P {X ∈ g −1 (−∞, t]}, where g −1 may be the set-valued inverse of g . The set g −1 (−∞, t] consists of all s ∈ R such that g(s) ∈ (−∞, t], i.e. g(s) ≤ t. If g is strictly increasing and continuously di˙erentiable then it has strictly increasing and continuously di˙erentiable inverse g −1 de˝ned on set g(R). In this case P {X ∈ g −1 (−∞, t]} = P {X ≤ g −1 (t)} = FX (g −1 (t)) for all t ∈ g(R). If, in addition, X is a continuous random variable, then dFY (t) dFX (g −1 (t)) fY (t) = = = dt dt ( dFX (s) ds ) ( s=g −1 (t) dg(s) ds )−1 = fX (g −1 ( (t)) s=g −1 (t) dg(s) ds )−1 s=g −1 (t) for all t ∈ g(R) . If t ∈/ g(R), then fY (t) = 0. One important type of function is a linear transformation. If Y = X − a for some a ∈ R, then FY (t) = P {Y ≤ t} = P {X − a ≤ t} = P {X ≤ t + a} = FX (t + a). In particular, if X is continuous, then Y is also continuous with fY (t) = fX (t + a). If Y = bX with b > 0, then FY (t) = P {bX ≤ t} = P {X ≤ t/b} = FX (t/b). In particular, if X is continuous, then Y is also continuous with fY (t) = fX (t/b)/b. 1.3 Expected Value Informally, the expected value of some random variable can be interpreted as its average. Formally, if X is a random variable and g : R → R is some function, then, by de˝nition, E[g(X)] = ∑ g(xi )pi i for discrete random variables and ∫ +∞ E[g(X)] = −∞ g(x)fX (x)dx for continuous random variables. Expected values for some functions g deserve special names: • mean: g(x) = x, E[X] • second moment: g(x) = x2 , E[X 2 ] • variance: g(x) = (x − E[X])2 , E[(X − E[X])2 ] 2 • k -th moment: g(x) = xk , E[X k ] • k -th central moment: E[(X − EX)k ] The variance of random variable X is commonly denoted by V (X). 1.3.1 Properties of expectation 1) For any constant a (non-random), E[a] = a. 2) The most useful property of an expectation is its linearity: if X and Y are two random variables and a and b are two constants, then E[aX + bY ] = aE[X] + bE[Y ]. 3)If X is a random variable, then V (X) = E[X 2 ] − (E[X])2 . Indeed, V (X) = E[(X − E[X])2 ] = E[X 2 − 2XE[X] + (E[X])2 ] = E[X 2 ] − E[2XE[X]] + E[(E[X])2 ] = E[X 2 ] − 2E[X]E[X] + (E[X])2 = E[X 2 ] − (E[X])2 . 4) If X is a random variable and a is a constant, then V (aX) = a2 V (X) and V (X + a) = V (X). 1.4 Examples of Random Variables Discrete random variables: • Bernoulli (p): random variable X has Bernoully(p) distribution if it takes values from X = {0, 1}, P {X = 0} = 1 − p and P {X = 1} = p. Its expectation E[X] = 1 · p + 0 · (1 − p) = p. Its second moment E[X 2 ] = 12 · p + 02 · (1 − p) = p. Thus, its variance V (X) = E[X 2 ] − (E[X])2 = p − p2 = p(1 − p). Notation: X ∼ Bernoulli(p). • Poisson (λ): random variable X has a Poisson(λ) distribution if it takes values from X = {0, 1, 2, ...} and P {X = j} = e−λ λj /j!. As an exercise, check that E[X] = λ and V (X) = λ. Notation: X ∼ Poisson(λ}. Continuous random variables: • Uniform (a, b): random variable X has a Uniform(a, b) distribution if its density fX (x) = 1/(b − a) for x ∈ (a, b) and fX (x) = 0 otherwise. Notation: X ∼ U (a, b). • Normal (µ, σ 2 ): random variable X has a Normal(µ, σ 2 ) distribution if its density fX (x) = exp(−(x − √ µ)2 /(2σ 2 ))/( 2πσ) for all x ∈ R. Its expectation E[X] = µ and its variance V (X) = σ 2 . Notation: X ∼ N (µ, σ 2 ). As an exercise, check that if X ∼ N (µ, σ 2 ), then Y = (X − µ)/σ ∼ N (0, 1). Y is said to have a standard normal distribution. It is known that the cdf of N (µ, σ 2 ) is not analytical, i.e. it can not be written as a composition of simple functions. However, there exist tables that give 3 its approximate values. The cdf of a standard normal distribution is commonly denoted by Φ, i.e. if Y ∼ N (0, 1), then FY (t) = P {Y ≤ t} = Φ(t). 2 Bivariate (multivariate) distributions 2.1 Joint, marginal, conditional If X and Y are two random variables, then FX,Y (x, y) = P {X ≤ x, Y ≤ y} denotes their joint cdf. X and Y ∫x ∫y are said to have joint pdf fX,Y if fX,Y (x, y) ≥ 0 for all x, y ∈ R and FX,Y (x, y) = −∞ f (s, t)dtds. −∞ X,Y Under some mild regularity conditions (for example, if fX,Y (x, y) is continuous), ∂ 2 FX,Y (x, y) ∂x∂y fX,Y (x, y) = From the joint pdf fX,Y one can calculate the pdf of, say, X . Indeed, ∫ FX (x) = P {X ≤ x} = x ∫ +∞ f (s, t)dtds −∞ −∞ ∫ +∞ Therefore fX (s) = −∞ f (s, t)dt. The pdf of X is called marginal to emphasize that it comes from a joint pdf of X and Y . If X and Y have a joint pdf, then we can de˝ne a conditional pdf of Y given X = x (for x such that fX (x) > 0): fY |X (y|x) = fX,Y (x, y)/fX (x). Conditional probability is a full characterization of how Y is distributed for any given given X = x. The probability that Y ∈ A for some set A given that X = x can ∫ be calculated as P {Y ∈ A|X = x} = A fY |X (y|x)dy . In a similar manner we can calculate the conditional ∫ +∞ expectation of Y given X = x: E[Y |X = x] = −∞ yfY |X (y|x)dy . As an exercise, think how we can de˝ne the conditional distribution of Y given X = x if X and Y are discrete random variables. Two extremely useful properties of a conditional expectation are: for any random variables X and Y , • E[f (X)Y |X = x] = f (x)E[Y |X = x]; • the law of iterated expectations : E[E[Y |X = x]] = E[Y ]. 2.2 Independence Random variables X and Y are said to be independent if fY |X (y|x) = fY (y) for all x ∈ R, i.e. if the marginal pdf of Y equals conditional pdf Y given X = x for all x ∈ R. Note that fY |X (y|x) = fY (y) if and only if fX,Y (x, y) = fX (x)fY (y). If X and Y are independent, then g(X) and f (Y ) are also independent for any functions g : R → R and f : R → R. In addition, if X and Y are independent, then E[XY ] = E[X]E[Y ]. 4 Indeed, ∫ E[XY ] +∞ ∫ +∞ = −∞ ∫ +∞ −∞ ∫ +∞ −∞ ∫ +∞ −∞ = = −∞ = xyfX,Y (x, y)dxdy xyfX (x)fY (y)dxdy xfX (x)dx ∫ +∞ −∞ yfY (y)dy E[X]E[Y ] 2.3 Covariance For any two random variables X and Y we can de˝ne covariance as cov(X, Y ) = E[(X − E[X])(Y − E[Y ])]. As an exercise, check that cov(X, Y ) = E[XY ] − E[X]E[Y ]. Covariances have several useful properties: 1. cov(X, Y ) = 0 whenever X and Y are independent 2. cov(aX, bY ) = abcov(X, Y ) for any random variables X and Y and any constants a and b 3. cov(X + a, Y ) = cov(X, Y ) for any random variables X and Y and any constant a 4. cov(X, Y ) = cov(Y, X) for any random variables X and Y 5. |cov(X, Y )| ≤ √ V (X)V (Y ) for any random variables X and Y 6. V (X + Y ) = V (X) + V (Y ) + 2cov(X, Y ) for any random variables X and Y 7. V ( ∑n i=1 Xi ) = ∑n i=1 V (Xi ) whenever X1 , ..., Xn are independent To prove property 5, consider random variable X − aY with a = cov(X, Y )/V (Y ). On the one hand, its variance V (X − aY ) ≥ 0. On the other hand, V (X − aY ) = V (X) − 2acov(X, Y ) + a2 V (Y ) = V (X) − 2(cov(X, Y ))2 /V (Y ) + (cov(X, Y )2 /V (Y ) Thus, the last expression is nonnegative as well. Multiplying it by V (Y ) yields the result. √ The correlation of two random variables X and Y is de˝ned by corr(X, Y ) = cov(X, Y )/ V (X)V (Y ). By property 5 as before, |corr(X, Y )| ≤ 1. If |corr(X, Y )| = 1, then X and Y are linearly dependent, i.e. there exist constants a and b such that X = a + bY . 5 3 Normal Random Variables Let us begin with the de˝nition of a multivariate normal distribution. Let Σ be a positive de˝nite n × n matrix. Remember that the n × n matrix Σ is positive de˝nite if aT Σa > 0 for any non-zero n × 1 vector a. Here superindex T denotes transposition. Let µ be n × 1 vector. Then X ∼ N (µ, Σ) if X is continuous and its pdf is given by fX (x) = exp(−(x − µ)T Σ−1 (x − µ)/2) √ (2π)n/2 det(Σ) for any n × 1 vector x. A normal distribution has several useful properties: 1. if X ∼ N (µ, Σ), then Σij = cov(Xi , Xj ) for any i, j = 1, ..., n where X = (X1 , ..., Xn )T 2. if X ∼ N (µ, Σ), then µi = E[Xi ] for any i = 1, ..., n 3. if X ∼ N (µ, Σ), then any subset of components of X is normal as well. In particular, Xi ∼ N (µi , Σii ) 4. if X and Y are uncorrelated normal random variables, then X and Y are independent. As an exercise, check this statement 2 2 ), Y ∼ N (µY , σY2 ), and X and Y are independent, then X +Y ∼ N (µX +µY , σX +σY2 ) 5. if X ∼ N (µX , σX 6. Any linear combination of normals is normal. That is, if X ∼ N (µ, Σ) is an n × 1 dimensional normal vector, and A is a ˝xed k × n full-rank matrix with k ≤ n, then Y = AX is a normal k × 1 vector: Y ∼ N (Aµ, AΣAT ). 3.1 Conditional distribution Another useful property of a normal distribution is that its conditional distribution is normal as well. If [ X= X1 ] X2 ([ ∼N µ1 ] [ , µ2 Σ11 Σ12 Σ21 Σ22 ]) ˜ with µ̃ = µ1 +Σ12 Σ−1 (x2 −µ2 ) and Σ ˜ = Σ11 −Σ12 Σ−1 Σ21 . If X1 and X2 are both then X1 |X2 = x2 ∼ N (µ̃, Σ) 22 22 random variables (as opposed to random vectors), then E[X1 |X2 = x2 ] = µ1 + cov(X1 , X2 )(x2 − µ2 )/V (X2 ). Let us prove the last statement. Let [ ] Σ= σ11 σ12 σ12 σ22 be the covariance matrix of 2 × 1 normal random vector X = (X1 , X2 )T with mean µ = (µ1 , µ2 )T . Note that 2 Σ12 = Σ21 = σ12 since cov(X1 , X2 ) = cov(X1 , X2 ). From linear algebra, we know that det(Σ) = σ11 σ22 − σ12 and [ ] Σ−1 = 1 det(Σ) 6 σ22 −σ12 −σ12 σ11 . Thus the pdf of X is fX (x1 , x2 ) = exp{−[(x1 − µ1 )2 σ22 + (x2 − µ2 )2 σ11 − 2(x1 − µ1 )(x2 − µ2 )σ12 ]/(2 det(Σ)} √ , 2π det(Σ) and the pdf of X2 is fX2 (x2 ) = Note that exp{−(x2 − µ2 )2 /(2σ22 )} √ . 2πσ22 2 2 σ12 σ11 1 σ11 σ22 − (σ11 σ22 − σ12 ) − = = . det(Σ)σ22 det(Σ)σ22 det(Σ) σ22 Therefore the conditional pdf of X1 , given X2 = x2 , is fX1 |X2 (x1 |X2 = x2 ) = = fX (x1 , x2 ) fX2 (x2 ) 2 exp{−[(x1 − µ1 )2 σ22 + (x2 − µ2 )2 σ12 /σ22 − 2(x1 − µ1 )(x2 − µ2 )σ12 ]/(2 det(Σ))} √ √ 2π det(Σ)/σ22 = 2 2 exp{−[(x1 − µ1 )2 + (x2 − µ2 )2 σ12 /σ22 − 2(x1 − µ1 )(x2 − µ2 )σ12 /σ22 ]/(2 det(Σ)/σ22 )} √ √ 2π det(Σ)/σ22 = exp{−[x1 − µ1 − (x2 − µ2 )σ12 /σ22 ]2 /(2 det(Σ)/σ22 )} √ √ 2π det(Σ)/σ22 = exp{−(x1 − µ̃)2 /(2σ̃)} √ √ , 2π σ̃ where µ̃ = µ1 + (x2 − µ2 )σ12 /σ22 and σ̃ = det(Σ)/σ22 . Note, that the last expression equals the pdf of a normal random variable with mean µ̃ and variance σ̃ yields the result. 7