Transformations of Random Variables, Joint Distributions of Random Variables

9.07 Introduction to Probability and Statistics for Brain and Cognitive Sciences Emery N. Brown Lecture 4: Transformations of Random Variables, Joint Distributions of Random Variables I. Objectives A. Understand the basic rules for computing the distribution of a function of a random variable. B. Understand how some important probability densities are derived using this method. C. Understand the concept of the joint distribution of random variables. D. Understand the bivariate Gaussian distribution. II. Transformations of Random Variables If we assume X is a continuous random variable with pdf f ( x) we consider the following transformations of X . They are 1. Y = aX + b (Linear) 2. Y =| X | (Absolute Value) 3. Y = X 2 (Quadratic) 4. Y = g ( x) (Monotonic) (4.1) In each case, we will want to find f y ( y). Example 4.1 Linear Transformation. Take Y = aX + b. Find f y ( y) (Rice: pp 64-67, Problem 62). Find the cdf and the pdf of Y . Fy ( y ) = Pr(Y ≤ y ) = Pr(aX + b ≤ y ) = Pr( X ≤ = Fx ( y −b ) a (4.2) y −b y −b ) = a f x ( x)dx −∞ a ∫ which is the cdf of Y . We find the pdf f y ( y) by differentiating Fy ( y) and considering two cases Case i a > 0 : page 2: Lecture 4: Transformations of Random Variables, Joint Distributions of Random Variables d y −b ( Fx ( )) dy a y −b 1 = fx ( )× a a f y ( y) = (4.3) Case ii a < 0 : d y −b ( Fx ( )) dy a y −b 1 )× < 0 = fx ( a a f y ( y) = for all y and hence, it is not a probability density. Thus, we multiply by two cases together we have f y ( y) = f x ( y −b 1 ) a |a| 1 if a < 0. Putting these −a (4.4) Remark 4.1. Example 4.1 illustrates a special case of the general result we establish in Proposition 4.1. Figure 4A. Fifteen-second time course of raw electromyogram (EMG) signals (panel 1) and the rectified EMG signals recorded from the semitendonosis muscle of a frog executing swimming motions. page 3: Lecture 4: Transformations of Random Variables, Joint Distributions of Random Variables Example 4.2. Probability Density of a Rectified Electromyogram Signal. X : N ( µ , σ 2 ) Y =| X | Find f y ( y). This is the problem that must be considered in analyzing electromyographic (EMG) data. An EMG is a recording of the electrical impulses transmitted through a group of muscle fibers. The impulses and the electrical potentials are both positive and negative (Figure 4A, panel 1). Because the force that is generated by a muscle is only positive, the data are analyzed after rectification. That is taking the absolute value of each observation (Figure 4B, panel 2). If we assume that the data are Gaussian, then we need to compute the probability density of the absolute value of a Gaussian random variable. Pr(Y ≤ y ) = Pr(| X | ≤ y ) = Pr(− y ≤ X ≤ y ) −y − µ X − µ y − µ ) = Pr( ≤ ≤ (4.5) σ σ σ y−µ −y − µ ) − Φ( ) = Φ( σ σ =( Hence, 1 12 ) [ 2π y −µ ∫−∞ σ 1 exp{− x 2 }dx − 2 fY ( y ) = φ ( − y −µ ∫−∞ σ 1 exp{− x 2 }dx] 2 −y − µ 1 y−µ 1 ) +φ( ) σ σ σ σ 1 y−µ −y − µ ) +φ( )]. = [φ ( σ σ σ If µ = 0 then, the probability density is f y ( y) = 2 y (4.6) φ ( ) y > 0. This is not the probability density of σ σ the sum of two Gaussian random variables. This we will study in Lectures 5 and 6. page 4: Lecture 4: Transformations of Random Variables, Joint Distributions of Random Variables Figure 4B. Histograms of the raw EMG signals (panel 1) and the rectified EMG signals (panel 2) from Figure 4A. Example 4.2 (continued). Figure 4B shows the histogram of the raw EMG signal (panel 1) and the rectified EMG signal (panel 2). How can we evaluate if the Gaussian assumption is reasonable? Example 4.3. If X : N (0,1) and Y = X 2 Find f y ( y). We have Pr(Y ≤ y ) = Pr( X 2 ≤ y ) 1 1 = Pr(− y 2 ≤ X ≤ y 2 ) 1 2 (4.7) 1 2 = Φ ( y ) − Φ (− y ) On differentiating we obtain f y ( y) = 1 1 1 − 12 y [φ ( y 2 ) + φ (− y 2 )] 2 = y − 2φ ( y 2 ) 1 1 1 1 = ( ) 2 y − 2 exp{− y} 2π 2 1 1 (4.8) 1 y− 2 exp{− y} 1 2 Γ( ) 2 1 = 1 1 and β = . This density is called a chi-squared density 2 2 2 2 with 1 degree of freedom written χ1 . The χ distribution with n degrees of freedom will be which is a gamma density with α = important in our data analyses. We can state a generalization of what we have done in computing the distribution of transformations of random variables. Proposition 4.1. If X is a continuous random variable with pdf f x ( x) and let Y = g ( X ) where g is a differentiable, strictly monotonic function on some interval I . Suppose f ( x) = 0 if x ∉ I . Then Y has the density function d (4.9) f y ( y) = f x (g −1 ( y))⋅ | g −1 ( y) | dy Proof: g is strictly monotonic means that if x1 < x2 then g ( x1 ) < g ( x2 ) if g is increasing and g ( x2 ) < g ( x1 ) if g is decreasing. On the interval I the mapping from X to Y is one-to-one. Hence, g −1 exists, and page 5: Lecture 4: Transformations of Random Variables, Joint Distributions of Random Variables Pr(Y ≤ y ) = Pr(g( X ) ≤ y) = Pr( X ≤ g −1 ( y)) = Fx ( g −1 ( y)) (4.10) d −1 (g ( y )) dy d = f x ( g −1 ( y)) ⋅ ( g −1 ( y)) dy f y ( y ) = Fx' ( g −1 ( y )) ⋅ d −1 d ( g ( y)) < 0, then we multiply by − ( g −1 ( y )) to insure that f y ( y) ≥ 0 for all y. Because g is dy dy a one-to-one mapping, we can write a more intuitive derivation because we must have, working with differentials, If f y ( y )dy = f x ( x)dx f y ( y ) = f x ( x) dx dy (4.11) Now x = g −1 ( y ) so that f y ( y) = f x (g −1 ( y)) ⋅ d −1 (g ( y)) dy (4.12) Again, we note that we take the absolute value of the derivative to make sure that f y ( y) ≥ 0 for all y. Figure 4C. Mapping and inverse mapping between x and y. II. Joint Distributions of Random Variables We consider now joint distributions of random variables. Being able to analyze the joint distribution of n random variables that represent a collection of data will be essential for our page 6: Lecture 4: Transformations of Random Variables, Joint Distributions of Random Variables statistical modeling. In general, the behavior of two random variables X and Y is given by the joint cumulative distribution function F ( x, y) = Pr( X ≤ x, Y ≤ y) (4.13) whether X and Y are continuous or discrete. A. Discrete Random Variables If X and Y are discrete random variables defined on the same outcome space and they assume values respectively x1, x2 ,..., and y1, y2 ,..., then their joint probability mass function is p( xi , y j ) = Pr( X = xi ,Y = y j ). We have that (4.14) ∑i ∑ j p(xi y j ) = 1 so that the joint pmf is well defined. We illustrate this idea with a simple example. Example 2.0 (continued). One roll of two fair dice yields Y 1,1 1, 2 1,3 1, 4 1,5 1, 6 2,1 2, 2 2,3 2, 4 2,5 2, 6 X 3,1 3, 2 3,3 3, 4 3,5 3, 6 4,1 4, 2 4,3 4, 4 4,5 4, 6 5,1 5, 2 5,3 5, 4 5,5 5, 6 6,1 6, 2 6,3 6, 4 6,5 6, 6 where X is the value on the face of the first die and Y is the value on the face of the second die. We have p( xi , yi ) = 1 36 (4.15) for xi = 1, 2,3, 4,5,6 and yi = 1, 2,3, 4,5,6 and 0 otherwise. For example p(3,1) = p(2, 2) = p(1,3) = 1 36 (4.15) and 6 Pr( X = 1) = ∑ p( X = 1, Yi ) = p( X = 1, Y = 1) + p( X = 1, Y = 2) + p( X = 1, Y = 3) + p( X = 1, Y = 4) + p( X = 1, Y = 5) i =1 1 + p( X = 1, Y = 6) = . 6 (4.17) page 7: Lecture 4: Transformations of Random Variables, Joint Distributions of Random Variables To find the pmf of X we sum across the columns, whereas to find the pmf of Y , we sum across the rows. This defines the marginal probability mass functions of X and Y respectively as ∑i p( x, yi ) p y ( y) = ∑ p( xi , y ) i p x ( x) = (4.18) B. Continuous Random Variables If X and Y are continuous random variables with a joint cdf F ( x, y), their joint probability density function is the piecewise continuous function of two variables f ( x, y) defined as f ( x, y ) ≥ 0 ∞ ∞ ∫−∞ ∫−∞ f ( x, y)dxdy = 1. (4.19) For any “reasonable” set A Pr( X , Y ) ∈ A) = ∫∫A f ( x, y)dxdy (4.20) If A = {( x, y) | X ≤ x, Y ≤ y}, F ( x, y) = x y ∫−∞ ∫−∞ f (u, v)dudv. (4.21) From the fundamental theorem of multivariate calculus, it follows that f ( x, y ) = ∂ 2 F ( x, y ) ∂x∂y (4.22) wherever the derivative is defined. For small δ x and δ y, if f is continuous, we have Pr( x ≤ X ≤ x + δ x, y ≤ Y ≤ y + δ y ) = x +δ x y +δ y ∫x ∫y f (u , v)dvdu ≈ f ( x, y )dxdy. (4.23) Roughly speaking for a 1-dimensional continuous random variable computing probabilities corresponds to computing the area under a curve, whereas for 2-dimensional continuous random variables, computing probabilities corresponds to computing the volume under a surface. page 8: Lecture 4: Transformations of Random Variables, Joint Distributions of Random Variables Figure 4.D. The domain of integration for Example 4.4 is the region {(x, y) | 0 ≤ y ≤ x ≤ 1}. Example 4.4 (Rice, pp. 75-76). Consider the bivariate density f ( x, y) = 12 2 (x + xy) 7 0 ≤ x ≤1 0 ≤ y ≤1 (4.24) Compute Pr( X > Y ) (4.25) {( x, y) | 0 ≤ y ≤ x ≤ 1} (4.26) This is the set Pr( X > Y ) = 12 7 12 = 7 36 = 14 1 x ∫0 ∫0 (x 2 + xy)dydx x xy 2 12 (x y + )dx = 0 2 7 ∫ 1 2 0 36 x 4 x dx = 0 14 4 ∫ 1 3 1 = 0 1 ∫0 (x3 + x3 )dx 2 (4.27) 9 14 The marginal cumulative distribution function of X is Fx ( x) = Pr(X ≤ x) = x ∞ ∫−∞ ∫−∞ f (u, y)dydu (4.28) page 9: Lecture 4: Transformations of Random Variables, Joint Distributions of Random Variables Figure 4.E. The domain of integration for Example 4.5 is the region 0 ≤ x ≤ y . Example 4.5 (Rice, pp. 79-80). Consider the joint probability density ⎧⎪λ 2 e−λ y f ( x, y) = ⎨ ⎪⎩ 0 0 ≤ x ≤ y, λ > 0 (4.29) Find the marginal pdf of X and the marginal pdf of Y . For f x ( x) we have f x ( x) = = ∞ ∫−∞ f xy (x, y)dy ∞ ∫x λ 2 e−λ y dy = lim − λ e −λb + λ e −λ x = λ e −λ x b→∞ (4.30) Hence, X has an exponential pdf. Now f xy ( x, y) = 0 for x ≤ 0 and x > y fY ( y) = ∞ y 2 −λ y ∫−∞ f xy (x)dx = ∫0 λ e dx = λ 2e−λ y x 0 = λ 2 ye−λ y . Hence, Y has a gamma pdf with α = 2 and β = λ. C. Bivariate Gaussian Probability Density An important bivariate probability density is given by y (4.31) page 10: Lecture 4: Transformations of Random Variables, Joint Distributions of Random Variables f ( x, y) = 1 2 2πσ xσ y (1− ρ ) 1 2 × exp{− 1 2 2(1− ρ ) [ ( x − µ x )2 σ x2 + ( y − µ y )2 2 ρ ( x − µ x )( y − µ y ) − ]} σ xσ y σ y2 (4.32) where x ∈ (−∞, ∞), y ∈ (−∞, ∞), µx ∈ (−∞, ∞), µ y ∈ (−∞, ∞), σ x > 0,σ y > 0 and −1 < ρ < 1. We have that µ x is the mean of X , µ y is the mean of Y , σ x is the standard deviation of X , σ y is the standard deviation of y and ρ is the correlation coefficient. We have that f ( x, y) is constant if ( x − µ x )2 σ x2 + ( y − µ y )2 σ y2 − 2 ρ ( x − µ x )( y − µ y ) σ xσ y =c (4.33) The locus of these points is an ellipse centered at (µ x , µ y ). If ρ = 0 the axes are parallel to the x and y axes. If ρ ≠ 0, then axes are tilted. If ρ > 0, then the tilt is positive. If ρ < 0, then the tilt is negative. Figure 4.F. Bivariate Gaussian probability density. The marginal distributions of x and y are respectively X N( µ x , σ x2 ) and Y N(µ y , σ y2 ). To show this we write f x (x) = Making the change of variables u = f x (x) = (x − µx ) σx 2 2πσ x (1 − ρ ) 1 2 ∫−∞ f xy ( x, y)dy and v = ∞ 1 ∞ (y − µy ) σy 1 ∫−∞ exp{− 2(1− ρ 2 ) (u (4.34) gives 2 + v 2 _ 2 ρ uv)}dv − (4.35) page 11: Lecture 4: Transformations of Random Variables, Joint Distributions of Random Variables Complete the square in v u 2 + v2 − 2ρ uv = u 2 + v2 − 2ρ uv + ρ 2 u2_ ρ 2u 2 = v2 − 2ρ uv + ρ 2u 2 + u 2 + ρ 2u 2 = (v − ρ u )2 + u 2 (1 − ρ 2 ) (4.36) Hence, substituting and using the definition of a Gaussian integral f x (x) = 1 1 exp{− u 2 } 2 12 2 2πσ x (1− ρ ) ∞ ∫−∞ exp{ − 1 (v − ρ u ) 2 dv 2 (1− ρ 2 ) 1 = 1(2π (1− ρ 2 )) 2 1 1 12 1 exp{− u 2 } = ( ) exp{− u 2 } 2 2 2 2πσ x 2πσ x (1 − ρ ) 2 1 2 = (2πσ x2 ) − 2 exp{− 1 Since u = ( x − µx ) / σ x . (4.37) 1 ( x − µ x )2 } 2 σ x2 Figure 4G. 1.6 seconds of two time-series of simultaneous recordings from a front (panel 1) and a back (panel 2) magnetoencephalogram SQUID sensor. Example 4.6. The Joint PDF of Two MEG Sensors. Figure 4H shows the scatterplot of simultaneous recordings from the two MEG SQUID sensors in Figure 4G. Do the data appear to be bivariate Gaussian? Histograms with superimposed estimates of the marginal probability densities are shown in panels 1 and 3 of Figure 4I The corresponding Q-Q (probability) plots for the two marginal probability densities are shown in panels 2 and 4 of Figure 4I. Do the marginal pdf’s appear to be Gaussian? Is the correlation coefficient positive or negative for this probability density? page 12: Lecture 4: Transformations of Random Variables, Joint Distributions of Random Variables Figure 4H. Scatterplot of the simultaneous recordings from a front (x-axis) and a back (yaxis) magnetoencephalogram SQUID sensor. Figure 4I. Histogram (panel 1) and Gaussian probability density estimate (panel 1, dashed curve) of the front SQUID sensor measurements. Histogram (panel 3) and Gaussian probability density estimate (panel 3, dashed curve) of the back SQUID sensor measurements. Q-Q (Probability) plot of the front SQUID sensor measurements (panel 2). Q-Q (Probability) plot of the back SQUID sensor measurements (panel 4, blue dots). The dashed line in panels 2 and 4 show the line of exact agreement between the SQUID measurements and the estimated Gaussian density. D. Independent Random Variables Definition 4.1. The random variables X1, X 2 ,..., X n are independent if their joint cdf can be expressed as a product of their marginals. That is page 13: Lecture 4: Transformations of Random Variables, Joint Distributions of Random Variables n F ( X1 , X 2 ,... X n ) = ∏ Fx ( X i ) i =1 (4.38) i or equivalently, their joint pmf’s for discrete random variables or their joint pdf’s for continuous random variables may be n f ( X1 , X 2 ,..., X n ) = ∏ f x ( X i ). i =1 (4.39) i Remark 4.2. For a bivariate Gaussian probability density, if ρ = 0 the joint probability density is 2 1 1 1 ( x − µ x )2 ( y − µ y ) f ( x, y) = exp{− [ + ]} 1 2π (σ x2σ y2 ) 2 2 σ x2 σ 2y =( 2 1 1 ( x − µ x )2 1 12 1 (y − µy ) ]}( ) exp{− ( )} ) 2 exp{− [ 2 2 2πσ 2y 2πσ x2 σ x2 σ 2y 1 (4.40) = f x ( x) f y ( y ). and hence, we see the very important fact that if two Gaussian random variables are uncorrelated, or more generally, n Gaussian random variables are uncorrelated, then they are also independent. In general, if two random variables are uncorrelated, they are not independent. However, if two random variables are independent, they are uncorrelated. As stated in Lecture 1, independence is a strong condition. We will use an independence assumption frequently in constructing a joint probability density for the data in our statistical models. IV. Summary We have shown how to construct the probability densities for a transformation of a random variable. In addition, we have shown how to define joint probability densities for two or more random variables. Acknowledgments. I am grateful to Julie Scott for technical assistance in preparing this lecture, to Emilio Bizzi and Vincent Cheung for the EMG data in Figures 4A and 4B to Matti Hämäläinen and Antonio Molins for the MEG data in Figures 4G-4I. I am grateful to Vincent Cheung for preparing Figures 4A and 4B , to Antonio Molins for preparing Figures 4G-4I, and to Paymon Hosseini for preparing Figures 4C- 4F. Reference Rice JA. Mathematical Statistics and Data Analysis, 3rd edition. Boston, MA, 2007.