Derived Distributions

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Lecture 12 Fall 2018 DERIVED DISTRIBUTIONS Contents 1. Functions of a single random variable 2. Multivariate transformations 3. A single function of multiple random variables 4. Maximum and minimum of random variables 5. Sum of independent random variables – Convolution Given a random variable X with density fX , and a measurable function g, we are often interested in the distribution (CDF, PDF, or PMF) of the random variable Y = g(X). For the case of a discrete random variable X, this is straightforward: X pY (y) = pX (x). {x | g(x)=y} However, the case of continuous random variables is more complicated. Note that even if X is continuous, g(X) is not necessarily a continuous random variable, e.g., if the range of the function g is discrete. However, in many cases, Y is continuous and its PDF can be found by following a systematic procedure. 1 FUNCTIONS OF A SINGLE RANDOM VARIABLE The principal method for deriving the PDF of g(X) is the following two-step approach. 1 Calculation of the PDF of a Function Y = g(X) of a Continuous Random Variable X (a) Calculate the CDF FY of Y using the formula Z �
FY (y) = P g(X) ≤ y = fX (x) dx. {x | g(x)≤y} (b) Differentiate to obtain the PDF of Y : fY (y) = dFY (y). dy Example. Let Y = g(X) = X 2 , where X is a continuous random variable with known PDF. For any y > 0, we have FY (y) = P(Y ≤ y) = P(X 2 ≤ y) √ √ = P(− y ≤ X ≤ y) √ √ = FX ( y) − FX (− y), and therefore, by differentiating and using the chain rule, 1 1 √ √ fY (y) = √ fX ( y) + √ fX (− y), 2 y 2 y y > 0. Example. Suppose that X is a nonnegative random variable and that Y = exp(X 2 ). Note that FY (y) = 0 for y < 1. For y ≥ 1, we have p � 2
FY (y) = P eX ≤ y = P(X 2 ≤ log y) = P(X ≤ log y). By differentiating and using the chain rule, we obtain p fY (y) = fX ( log y) 1.1 1 √ , 2y log y y > 1. The case of monotonic functions The calculation in the last example can be generalized as follows. Assume that the range of the random variable X contains an open interval A. Suppose that g is strictly monotone (say increasing), and also differentiable on the interval 2 A. Let B the set of values of g(x), as x ranges over A. Let g−1 be the inverse function of g, so that g(g −1 (y)) = y, for y ∈ B. Then, for y ∈ B, and using the chain rule in the last step, we have fY (y) = d d dg−1 (y) P(g(X) ≤ y) = P(X ≤ g −1 (y)) = fX (g −1 (y)) . dy dy dy Recall from calculus that the derivative of an inverse function satisfies 1 dg−1 (y) = ′ −1 , dy g (g (y)) where g′ is the derivative of g. Therefore, fY (y) = fX (g −1 (y)) 1 g′ (g −1 (y)) . When g is strictly monotone decreasing, the only change is a minus sign in front of g ′ (g−1 (y)). Thus, the two cases can be summarized in the single formula: fY (y) = fX (g−1 (y)) 1 |g ′ (g−1 (y))| , y ∈ B. (1) An easy mnemonic for remembering (and also understanding this formula) is fY (y) |dy| = fX (x) |dx|, where x and y are related by y = g(x), and since dy = |g′ (x)| · |dx|, fY (y) |g ′ (x)| = fX (x). 1.2 Linear functions Consider now the special case where g(x) = ax + b, i.e., Y = aX + b. We assume that a = 6 0. Then, g ′ (x) = a and g−1 (y) = (y − b)/a. We obtain fY (y) = y − b 1 fX . |a| a Example. (A linear function of a normal random variable is normal) Suppose that d X = N (0, 1) and Y = aX + b. Then, (y−b)2 1 fY (y) = √ e− 2a2 , 2ˇ|a| 3 d so that Y is N (b, a2 ). More generally, if X = N (µ, ˙), then the same argument shows d that Y = ax + b = N (aµ + b, a2 ˙ 2 ). We conclude that a linear (more precisely, affine) function of a normal random variable is normal. 1.3 The general case Definition 1. Consider two measurable spaces (Ω1 , F1 ) and (Ω 2 , F2 ) and a measurable function g : Ω1 → Ω 2 . The pushforward g∗ µ of measure µ on (Ω1 , F1 ) along g is a measure on (Ω2 , F2 ) defined as g∗ µ(E) , µ(g−1 (E)) ∀E ∈ F2 . To relate this general notion to the discussion above, we notice that Y = g(X) ⇒ PY = g∗ PX To emphasize the fact that g “carries” µ into ˆ = g∗ µ we may schematically denote g dµ −→ dˆ (2) We establish some simple facts of pushforwards: Proposition 1. Let ˆ = g∗ µ. Then (i) (Change of variable formula) For any measurable f : Ω 2 → R we have Z Z f (!2 )dˆ = f (g(!1 ))dµ (3) Ω2 Ω1 and both integrals exist or do not exist simultaneously. (ii) For arbitrary non-negative °(!2 ) measure °(g(!1 ))dµ pushes forward to °(!2 )dˆ. Schematically: g ° ◦ g dµ −→ ° dˆ (4) Proof: (i) is easy: for f = 1E this follows from the definition of g∗ µ. Linearity of integration implies the statement for simple functions. Finally, splitting f = f + − f − and approximating both f + and f − by simple functions we conclude by invoking the MCT. 4 (ii) follows from the application of (3) with f = 1E · °: Z Z ° ◦ g(!1 )dµ °(!2 )dˆ = E g −1 E Note: Consider a special case with Ω1 = Ω 2 = R and dµ = dx – Lebesgue measure. Then whenever g – continuously differentiable with non-vanishing derivative on an open set U we have1 g 1U (x)|g′ (x)| dx −→ 1g(U ) (y) dy (5) This result will be established in multiple dimensions below. For now we apfX ◦g −1 ply (4) with °(x) = |g ′ ◦g −1 | to get: g fX (x) dx −→ fX (g−1 (y)) dy |g′ (g−1 (y)| which exactly means the formula for fY given in (1). 2 MULTIVARIATE TRANSFORMATIONS Suppose now that X = (X1 , . . . , Xn ) is a vector of random variables that are jointly continuous, with joint PDF fX (x) = fX (x1 , . . . , xn ). Consider a function g : Rn → Rn , and the random vector Y = (Y1 , . . . , Yn ) = g(X). Let gi be the components of g, so that Yi = gi (X) = gi (X1 , . . . , Xn ). Suppose that the function g is continuously differentiable on some open set A ⊂ Rn . Let B = g(A) be the image of A under g. Assume that the inverse function g−1 is well-defined on B; that is, for every y ∈ B, there is a unique x ∈ Rn such that g(x) = y. The formula that we develop here is an extension of the formula fY (y) |g ′ (x)| = fX (x) that we derived for the one-dimensional case, with the derivative g′ (x) being replaced by a matrix of partial derivatives, and with the absolute value being replaced by the absolute value of the determinant. It can be justified by appealing to the change of variables theorem from multivariate calculus, but we provide here a more transparent argument. 1 Here and below we abuse notation and write dx instead of Leb(dx). 5 2.1 Linear functions Let us first assume that g is a linear function, of the form g(x) = M x, for some n × n matrix M . Fix some x ∈ A and some  > 0. Consider the cube C = [x, x + ]n , and assume that  is small enough so that C ⊂ A. The image D = {M x | x ∈ C} of the cube C under the mapping g is a parallelepiped. Furthermore, the volume of D is known to be equal to |M | · n , where we use | · | to denote the absolute value of the determinant of a matrix. Having fixed x, let us also fix y = M x. Assuming that fX (x) is continuous at x, we have Z P(X ∈ C) = fX (t) dt = fX (x)n + o(n ) ≈ fX (x)n , C where o(n ) stands for a function such that lim↓0 o(n )/n = 0, and where the symbol ≈ indicates that the difference between the two sides is o(n ). Thus, fX (x) · n ≈ P(X ∈ C) = P(g(X) ∈ g(C)) = P(Y ∈ D) ≈ fY (y) · vol(D) = fY (y) · |M | · n . Dividing by n , and taking the limit as  ↓ 0, we obtain fX (x) = fY (y) · |M |. Let us now assume that the matrix M is invertible, so that its determinant is nonzero. Using the relation y = M x and the fact det(M −1 ) = 1/ det(M ), fY (y) = fX (M −1 y) = fX (M −1 y) · |M −1 |. |M | Note that if M is not invertible, the random variable Y takes values in a proper subspace S of Rn . Then, Y is not jointly continuous (cannot be described by a joint PDF). On the other hand, if we restrict our attention to S, and since S is isomorphic to Rm for some m < n, we could describe the distribution of Y in terms of a joint PDF on Rm . 2.2 The general case: heuristic Let us now generalize to the case where g is continuously differentiable at x. We define M (x) as the Jacobian matrix (@g/@x)(x), with entries (@gi /@xj )(x). 6 The image D = g(C) of the cube C is not a parallelepiped. However, from a first order Taylor series expansion, g is approximately linear in the vicinity of x. It can then be shown that the set D has volume |M (x)| · n + o(n ). It then follows, as in the linear case, that fY (y) = fX (g −1 (y)) = fX (g−1 (y)) · |M −1 (g −1 (y))|. |M (g−1 (y))| We note a useful fact from calculus that sometimes simplifies the application of the above formula. If we define J(y) as the Jacobian (the matrix of partial derivatives) of the mapping g−1 (y), and if some particular x and y are related by y = g(x), then J(y) = M −1 (x). Therefore, fY (y) = fX (g−1 (y)) · |J(y)|. 2.3 (6) The general case: formal results Our goal is to prove (6). First we state assumptions. Assumption A. Let U be an open set in Rn and g : U → Rn be continuously @g differentiable injection with non-vanishin Jacobian @x 6= 0. It is well known that if g : U → g(U ) satisfies Assumption A, then g(U ) is open and the inverse map g−1 satisfies Assumption A on g(U ). We first remind the following important result from calculus: For any continuous f : g(U ) → R with compact support we have Z Z @g dx (7) R f (y) dy = R f (g(x)) @x g(U ) U R where R denotes the Riemann integral. We now use this fact to establish the following: Theorem 1 (Jacobian formula). If U and g : U 7→ g(U ) satisfy Assumption A then @g g 1U (x) dx −→ 1g(U ) (y) dy @x 7 Applying Theorem 1 with (3) and (4) we can establish a number of consequences (for simplicity we assume U = Rn ): @g g dx −→ dy @x @g −1 g dx −→ dy @y g fX (x) dx −→ fX (g −1 (y)) (8) (9) @g −1 dy @y (10) −1 −1 @g where we also used the fact that @g@y = @x for y = g(x). In particular (10) implies (6). When g satisfies Assumption A on disjoint U1 and U2 then we have @g g  dx −→ 1g(U1 ) (y) + 1g(U2 ) (y) } dy 1U1 ∪U2 (x) @x which may be useful to find pushforwards along many-to-one maps. Proof (optional). Let µ be a measure defined via Z @g dx µ(E) = 1U (x) @x E and ˆ = g∗ µ. We need to show that ˆ(E) = Leb(E ∩ g(U )) for every Borel set E. Or, equivalently, that ˆ(E) = Leb(E) for every Borel E ⊂ g(U ). Under conditions of the theorem it is easy to see that g(U ) is an open set (since g is continuously invertible and hence every point y ∈ g(U ) has an open ball around itself contained in g(U )). First, we have indicated before (and proved in dimension 1) that Riemann integral, when it exists, coincides with Lebesgue integral over Lebesgue measure. Thus from (7) we conclude that for every continuous function f with compact support contained inside g(U ) we have Z Z f (y) dy = f (g(y))dµ Rn Rn where this time integrals are Lebesgue, and as before dy stands for Leb(dy). By (3) this implies that for all such functions f we have Z Z f (y) dy = f (y) dˆ Rn Rn 8 Now consider an arbitrary open ball B contained in g(U ). Although the indicator 1B (y) is not a continuous function, it can be easily approximated by a sequence continuous functions from below: 0 ≤ fn (y) ր 1B (y) Thus by the MCT we conclude that ˆ(B) = Leb(B) for every open ball contained in g(U ). Now every open set V ⊂ g(U ) can be written as a union of countably many open balls. Thus by continuity of measure ˆ(V ) = Leb(V ). But two ˙-finite measures coinciding on every open set must be identical (since open sets are a generating p-system). 2.4 The bivariate normal in polar coordinates Let X and Y be independent standard normal random variables. Let g be the mapping that transforms Cartesian coordinates to polar coordinates, and let (R, ) = g(X, Y ). The mapping g is either undefined or discontinuous at (x, y) = (0, 0). So, strictly speaking, in order to apply the multivariate transformation formula (6), we should work with A = R2 \ {(0, 0)}. The inverse mapping g −1 is given by (x, y) = g−1 (r, ) = (r cos , r sin ). Its Jacobian matrix is of the form   cos  sin  , −r sin  r cos  and therefore, |J(r, )| = r cos2  + r sin2  = r. From the multivariate transformation formula, we obtain fR, (r, ) = 1 −r2 /2 r −(r2 cos2 +r2 sin2 )/2 e = re , 2ˇ 2ˇ r > 0. We observe that the joint PDF of R and  is of the form fR, = fR f , where fR (r) = re−r and 2 /2 , 1 ,  = [0, 2ˇ]. 2ˇ In particular, R and  are independent. The random variable R is said to have a Rayleigh distribution. f () = 9 We can also find the density of Z = R2 . For the mapping g defined by √ g(r) = r 2 , we have g−1 (z) = z, and g′ (r) = 2r, which leads to 1/g ′ (g−1 (z)) = √ 1/2 z. We conclude that fZ (z) = √ −z/2 1 1 √ = e−z/2 , ze 2 z 2 z>0 which we recognize as an exponential PDF with parameter 1/2. An interesting consequence of the above results is that in order to simulate normal random variables, it suffices to generate two independent random variables, one uniform and one exponential. Furthermore, an exponential random variable is easy to generate using a nonlinear transformation of another independent uniform random variable. 3 A SINGLE FUNCTION OF MULTIPLE RANDOM VARIABLES Suppose that X = (X1 , . . . , Xn ) is a vector of jointly continuous random variables with known PDF fX . Consider a function g1 : Rn → R and the random variable Y1 = g1 (X). Note that the formulas from Section 2 cannot be used directly. In order to find the PDF of Y , one possibility is to calculate the multidimensional integral Z FY (y) = P(g(X) ≤ y) = fX (x) dx, {x | g(x)≤y} and then differentiate. Another possibility is to introduce additional functions g2 , . . . , gn : Rn → R, and define Yi = gi (X), for i ≥ 2. As long as the resulting function g = (g1 , . . . , gn ) is invertible, we can appeal to our earlier formula to find the joint PDF of Y , and then integrate to find the marginal PDF of Y1 . 6 The simplest choice in the above described method is to let Yi = Xi , i = 1, so that g(x) = (g1 (x), x2 , . . . , xn ). Let h : Rn → R be a function that corresponds to the first component of g −1 . That is, if y = g(x), then x1 = h(y). Then, the inverse mapping g −1 is of the form g −1 (y) = (h(y), y2 , . . . , yn ), and its Jacobian matrix is of the form  @h @h @y1 (y) @y2 (y) · · ·  0 1 ···  J(y) =  . . .. .. ..  . 0 0 ··· 10 @h @yn (y) 0 .. . 1    .  @h It follows that |J(y)| = | @y (y)|, and 1 fY (y) = fX (h(y), y2 , . . . , yn ) @h (y) . @y1 Integrating, we obtain fY1 (y1 ) = Z fX (h(y), y2 , . . . , yn ) @h (y) dy2 · · · dyn . @y1 Example. Let X1 and X2 be positive, jointly continuous, random variables, and suppose that we wish to derive the PDF of Y1 = g(X1 , X2 ) = X1 X2 . We define Y2 = X2 . From the relation x1 = y1 /x2 we see that h(y1 , y2 ) = y1 /y2 . The partial derivative @h/@y1 is 1/y2 . We obtain Z Z 1 1 fY1 (y1 ) = fX (y1 /y2 , y2 ) dy2 = fX (y1 /x2 , x2 ) dx2 . x2 y2 d For a special case, suppose that X1 , X2 = U (0, 1) are independent. Their common PDF is fXi (xi ) = 1, for xi ∈ [0, 1]. Note that fY1 (y1 ) = 0 for y ∈ / (0, 1). Furthermore, fX1 (y1 /x2 ) is positive (and equal to 1) only in the range x2 ≥ y1 . Also fX2 (x2 ) is positive, and equal to 1, iff x2 ∈ (0, 1). In particular, fX (y1 /x2 , x2 ) = fX1 (y1 /x2 )fX2 (x2 ) = 1, for x2 ≥ y1 . We then obtain fY1 (y1 ) = Z 1 y1 1 dx2 = − log y, x2 y1 ∈ (0, 1). The direct approach to this problem would first involve the calculation of FY1 (y1 ) = P(X1 X2 ≤ y1 ). It is actually easier to calculate Z 1Z 1 dx2 dx1 1 − FY1 (y1 ) = P(X1 X2 ≥ y1 ) = y1 y1 /x1 Z  y1  = 1− dx1 x1 y1 = (x1 − y1 log x1 ) 1 y1 = (1 − y1 ) + y1 log y1 . Thus, FY1 (y1 ) = y1 − y1 log y1 . Differentiating, we find that fY1 (y1 ) = − log y1 . An even easier solution for this particular problem (along the lines of the stick example in Lecture 9) is to realize that conditioned on X1 = x1 , the random variable Y1 = X1 X2 is uniform on [0, x1 ], and using the total probability theorem, Z 1 Z 1 1 fX1 (x1 )fY1 |X1 (y1 | x1 ) dx1 = dx1 = − log y1 . fY1 (y1 ) = x y1 y1 1 11 4 MAXIMUM AND MINIMUM OF RANDOM VARIABLES Let X1 , . . . , Xn be independent random variables, and let X (1) ≤ X (2) ≤ · · · ≤ X (1) denote the corresponding order statistics. Namely, X (1) = minj Xj , X (2) is the second smallest of the values X1 , . . . , Xn , and X (n) = maxj Xj . We would like to find the joint distribution of the order statistics and specifically the distribution of minj Xj and maxj Xj . Note that P(max Xj ≤ x) = P(X1 , . . . , Xn ≤ x) = P(X1 ≤ x) · · · P(Xn ≤ x) j = FX1 (x) · · · FXn (x). For the minimum, we have P(min Xj ≤ x) = 1 − P(min Xj > x) j j = 1 − P(X1 , . . . , Xn > x) = 1 − (1 − FX1 (x)) · · · (1 − FXn (x)). Let us consider the special case where the X1 , . . . , Xn are i.i.d., with common CDF F and PDF f . For simplicity, assume that F is differentiable everywhere. Then, P(max Xj ≤ x) = F n (x), j P(min Xj ≤ x) = 1 − (1 − F (x))n , j implying that fmaxj Xj (x) = nF n−1 (x)f (x), fminj Xj (x) = n(1 − F (x))n−1 f (x). Exercise 1. Assuming that X1 , . . . , Xn are independent with common density function f , establish that the joint distribution of X (1) , . . . , X (n) is given by fX (1) ,...,X (n) (x1 , . . . , xn ) = n!f (x1 ) · · · f (xn ), x1 < x2 < · · · < xn , and fX (1) ,...,X (n) (x1 , . . . , xn ) = 0, otherwise. Use this to derive the densities for maxj Xj and minj Xj . 12 5 SUM OF INDEPENDENT RANDOM VARIABLES – CONVOLUTION If X and Y are independent discrete random variables, the PMF of X + Y is easy to find: pX+Y (z) = P(X + Y = z) X = P(X = x, Y = y) {(x,y)|x+y=z} = X P(X = x, Y = z − x) = X pX (x)pY (z − x). x x When X and Y are independent and jointly continuous, an analogous formula can be expected to hold. We derive it in two different ways. A first derivation involves plain calculus. Let fX,Y be the joint PDF of X and Y . Then. Z Z ∞ Z z−x P(X + Y ≤ z) = fX,Y (x, y) dx dy = fX,Y (x, y) dy dx. {x,y | x+y≤z} −∞ −∞ Introduce the change of variables t = x + y. Then, Z ∞Z z P(X + Y ≤ z) = fX,Y (x, t − x) dt dx, −∞ −∞ which gives fX+Y (z) = Z ∞ −∞ fX,Y (x, z − x) dx. In the special case where X and Y are independent, we have fX,Y (x, y) = fX (x)fY (y), resulting in Z ∞ fX+Y (z) = fX (x)fY (z − x) dx. −∞ If we were to use instead our general tools, we could proceed as follows. Consider the linear function g that maps (X, Y ) to (X, X + Y ). It is easily seen that the associated determinant is equal to 1. Thus, with Z = X + Y , we have fX,Z (x, z) = fX,Y (x, z − x) = fX (x)fY (z − x). We then integrate over all x, to obtain the marginal PDF of Z. 13 d d Exercise 2. Suppose that X = N (µ1 , ˙12 ), X2 = N (µ2 , ˙22 ) and independent. Estabd lish that X1 + X2 = N (µ1 + µ2 , ˙12 + ˙22 ). Exercise 3. Establish the semigroup properties for Gamma and Cauchy distributions mentioned in Lecture 10. 14