Conditional Distributions and Functions of Jointly Distributed Random Variables

9.07 Introduction to Probability and Statistics for Brain and Cognitive Sciences Emery N. Brown Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables I. Objectives Understand the concept of a conditional distribution in the discrete and continuous cases. Understand how to derive the distribution of the sum of two random variables. Understand how to compute the distribution for the transformation of two or more random variables. II. Conditional Distributions Just as we used conditional probabilities in Lecture 1 to evaluate the likelihood of one event given another, we develop here the concepts of discrete and continuous conditional distributions and discrete and continuous conditional probability mass functions and probability density functions to evaluate the behavior of one random variable given knowledge of another. A. Discrete Conditional Distributions If X and Y are jointly distributed discrete random variables, the conditional probability that X = xi given Y = y j is Pr( X = xi | Y = y j ) = = Pr( X = xi ,Y = y j ) Pr(Y = y j ) pxy ( xi , y j ) (5.1) py ( y j ) provided that p y ( y j ) > 0. This is the conditional probability mass function of X given Y = y j . The numerator is a function of x for a fixed value of y. Similarly, we have Pr(Y = y j | X = xi ) = pxy ( xi , y j ) px ( xi ) (5.2) We also have pxy ( xi , y j ) = px ( xi ) p y|x ( yi | xi ) = p y ( y j ) px| y ( xi | y j ). (5.3) Example 5.1 Melencolia I. The following discrete joint probability mass function is based on a magical square in Albert Dϋrer’s engraving Melencolia I. page 2: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables Y 1 2 3 4 X 1 16 2 5 3 9 4 4 136 136 136 136 13 2 136 136 136 10 8 11 136 136 136 6 7 12 136 136 136 15 14 1 136 136 136 3 (5.4) Find Pr(Y = 1| X = 3) and Pr(X = 4 | Y = 2) Pr(Y = 1| X = 3) = = Pr(Y = 1I X = 3) Pr( X = 3) Pr(Y = 1I X = 3) 4 ∑ Pr(Y = yi I X = 3) i=1 9 1 / (9 + 6 + 7 +12) = 9 34 136 136 Pr( X = 4 I Y = 2) Pr( X = 4 | Y = 2) = = (5.5) 4 ∑ Pr( X = xi I Y = 2) i=1 15 1 = / (3 +10 + 6 +15) 136 136 15 = 34 Example 5.2 Suppose that each acetylcholine molecule released at the neuromuscular junction of a frog has a probability p of being defective. If the distribution of the number of molecules released in a 10 msec time interval is a Poisson distribution with parameter λ , what is the distribution of the number of defective acetylcholine molecules? Let N be the number of acetylcholine molecules released in 10 msec and let X be the number that is defective. If we assume that one molecule being defective is independent of any other being defective, we have that the conditional pmf of X given N = n is the binomial distribution ⎛ n⎞ Pr( X = k | N = n) = ⎜ ⎟ p k (1 − p)n−k ⎝k⎠ And we have that (5.6) page 3: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables Pr( N = n) = λ n e−λ n! (5.7) . By the Law of Total Probability (Lecture 1), we have Pr( X = k ) = ∞ ∑ Pr( N = n) Pr( x = k | N = n) n=0 = = ∞ ∑ λ n e−λ ⎛ n ⎞ ∑ λ n e−λ n=k ∞ n! n! n=k = k (λ p) −λ e k! k n−k ⎜ ⎟ p (1− p) ⎝k ⎠ n! p k (1− p)n − k k !(n − k )! ∞ ∑ λ n−k n=k (1− p) (n − k )! (5.8) n −k k = (λ p) −λ λ (1− p) e e k! = (λ p)k −λ p e k! which is a Poisson model with rate parameter λ p. B. Continuous Conditional Distribution If X and Y are continuous random variables with joint probability density f xy ( x, y), then the conditional probability density of Y given X is defined as f y| x ( y | x ) = f xy ( x, y ) f x ( x) (5.9) for 0 < f x ( x) < ∞ and 0 otherwise. If we define this in terms of differentials we have Pr( y < Y ≤ y + dy | x ≤ X ≤ x + dx) = = = Pr( y < Y ≤ y + dy I x ≤ X ≤ x + dx) Pr(x ≤ X ≤ x + dx) f xy ( x, y )dxdy f x ( x)dx f xy ( x, y )dy f x ( x) . (5.10) Hence, we have page 4: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables ( Pr( y < Y ≤ y + dy | x ≤ X ≤ x + dx) dy dy→∞ f xy ( x, y) f y| x ( y | x) = lim = f x ( x) The numerator is a function of y for a fixed value of x. Similarly, we can define f x| y ( x | y) = f xy ( x, y) f y ( y) (5.11) f xy ( x, y) = f y|x ( y | x) f x (x) and the marginal probability density of Y is fY ( y) = ∫ f y|x ( y | x) f x (x)dx (5.12) which is the Law of Total Probability for continuous random variables. Example 4.6 (continued). Figure 4H shows the joint probability sensors. Consider the distribution of the front sensor (y-axis) given sensor (x-axis) is 0.25. If the bivariate Gaussian probability density these data, then we can show that the conditional distribution of Y probability density defined as density of the two MEG that the value of the back is a reasonable model for given X is the Gaussian σy ⎧ ⎫ [ y − µy − ρ ( x − µ x )]2 ⎪ ⎪ 1 σ 1 ⎪ ⎪ x fY | X ( y | x) = [2πσ 2y (1 − ρ 2 )]− 2 ×exp ⎨− ⎬ 2 2 2 σ y (1 − ρ ) ⎪ ⎪ ⎪⎩ ⎪⎭ (5.13) We have E (Y | X ) = µ y + ρ σy σx (x − µ x ) (5.14) Var(Y | X ) = σ 2y (1− ρ 2 ). Note that if ρ = 0, we have the marginal density of Y is the Gaussian density with E(Y ) = µ y and Var(Y ) = σ 2y , consistent with the idea that X and Y would be independent. Notice also that σ y2 (1 − ρ 2 ) ≤ σ 2y showing that knowledge about X reduces the uncertainty in Y. page 5: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables The conditional mean µ y| x = µ y + ρ σy (x − µ x ) is called the regression line of y on x. It gives the σx “best” prediction of y given x. When we study regression analysis we will make this statement precise. Example 4.5 (continued). In this example we have the joint and marginal densities as f xy ( x, y ) = λ 2 e−λ y 0≤ x≤ y f x ( x) = λ e −λ x 0≤ x f y ( y ) = λ 2 ye−λ y 0≤ y (5.15) The conditional densities are f y| x ( y | x) = f xy ( x, y ) f x ( x) = λ 2 e−λ y λe −λ x = λ e −λ ( y − x) (5.16) for 0 < x < y < ∞, and f x| y ( x | y) = λ 2 e−λ y 2 λ ye −λ y = y −1 (5.17) for 0 < x ≤ y. That is, X is uniformly distributed on (0, y]. Note that f xy ( x, y) = f y|x (y | x) f x (x) = f x| y (x | y) f y (y). (5.18) We can simulate data from this joint probability density by either one of the following two algorithms. Algorithm 5.1 1) 2) Draw X from the exponential density f x ( x) . Draw Y from the exponential density f y|x ( y | x) on the interval [ x, ∞). Algorithm 5.2 1) Draw Y from the gamma density f y ( y) . 2) Draw X from the uniform density on the interval [0,Y ]. page 6: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables We will show later in this lecture that Algorithm 5.2 states that conditional on (or given that) the sum of two exponential random variables which is a gamma random variable, the distribution of the first random variable is uniform. Example 5.3 Dendritic Dynamics (Lee et al., PLoSB 4(2):e29 (2006)). To study the neuronal remodeling that occurs in the brain on a day-to-day basis, Elly Nedivi and colleagues used a multiphoton-based microscopy system for chronic in vivo imaging and reconstruction of entire neurons in the superficial layers of the rodent cerebral cortex. Over a period of months, they observed neurons extending and retracting existing branches, and in rare cases, budding new tips. Thirty-five of 259 non-pyramidal interneuron dendritic tips showed rearrangement and 0 of 124 pyramidal cell dendritic tips showed rearrangement. The objective is to analyze the probability of rearrangement for the two types of neurons. Assume the propensity to change is a probability pi ∈(0,1), where i = 1 is the interneuron propensity and i = 2 is the pyramidal neuron propensity. We will perform Bayesian analysis of this problem. Assume that pi has a beta distribution with parameters α and β . This will represent our knowledge of pi prior to the experiment and is called a prior distribution. Given pi and X i : Binomial (ni , pi ) we have ⎛n ⎞ f xi | pi ( xi | pi ) = ⎜ i ⎟ pi xi (1 − pi )1−xi ⎝ xi ⎠ (5.19) for xi = 0,1,..., ni and f ( pi ) is the beta density f ( pi ) = Γ(α + β ) α −1 pi (1 − pi ) β −1. Γ(α )Γ( β ) (5.20) Let’s compute f ( xi ) and f ( pi | xi ). The joint distribution is the product of the pmf of a discrete random variable xi and a continuous random variable pi and is defined as f ( xi , pi ) = f ( pi ) f ( xi | pi ) = ⎛n ⎞ Γ(α + β ) α −1 pi (1 − pi ) β −1 ⎜ i ⎟ pi xi (1 − pi ) ni − xi Γ(α )Γ( β ) ⎝ xi ⎠ = Γ(ni + 1) Γ(α + β ) α + x −1 p i (1 − pi ) ni + βi −xi −1 Γ(α )Γ( β ) Γ( xi + 1)Γ(ni − xi +1) i (5.21) using the fact that if xi is an integer Γ( xi ) = ( xi −1)! by Factoid 1 from the Addendum to Lecture 3 ⎛ ni ⎞ ni ! Γ(ni +1) = ⎜ ⎟= ⎝ xi ⎠ xi !(ni − xi )! Γ(xi +1)Γ(ni − xi +1) (5.22) page 7: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables To compute f ( xi ) we consider f (xi ) = 1 ∫0 f (xi , pi )dpi . (5.23) From Lecture 3 and the Addendum to Lecture 3, we know that 1 α +x −1 i ∫0 pi (1 − pi )ni + βi −xi −1 dpi = Γ(α + xi )Γ(ni + β − xi ) Γ(α + β + ni ) (5.24) Hence, f ( xi ) = Γ(ni +1) Γ(α + xi )Γ(ni + β − xi ) Γ(α + β ) Γ(α )Γ( β ) Γ(xi +1)Γ(ni − xi +1) Γ(α + β + ni ) (5.25) for xi = 0,1, 2,..., ni . If we take f ( pi ) to be the beta probability density with α = β = 1, then we have the uniform probability density on [0,1] . This means we are indifferent among all the values of p in [0,1] prior to the start of the experiment and f ( xi ) simplifies to f ( xi ) = 1 . ni +1 (5.26) Verify this. For the Bayesian analysis we will be interested in f ( pi | xi ), the most likely values of pi given the data xi for i = 1, 2. It is f ( pi | xi ) = f ( pi , xi ) f ( xi ) Γ(α + β )Γ(ni +1) piα + xi −1 (1− pi )ni + β − xi −1 Γ(α )Γ( β )Γ(xi +1)Γ(ni − xi +1) = Γ(α + β )Γ(ni +1)Γ(α + xi )Γ( ni + β − xi ) Γ(α )Γ( β )Γ(xi +1)Γ(ni − xi +1)Γ(α + β + ni ) = (5.27) Γ(α + β + ni ) α + x −1 p i (1− pi ) ni + β − xi −1 Γ(α + xi )Γ(ni + β − xi ) i which is a beta probability density with parameters α + xi and ni + β − xi . We have now laid the ground work for an analysis we will be performing on these data in a few weeks. page 8: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables III. Functions of Jointly Distributed Random Variables The two functions of jointly distributed random variables X and Y that are most commonly considered are the sum and quotient Z = X +Y (5.28) Z= Y . X We will consider only the sum. The quotient is discussed in Rice, pp. 98-99. A. Sum of Two Random Variables 1. Discrete Random Variables If X and Y are discrete random variables with joint pmf p( X , Y) then Z = X + Y is the set of all events Z = z , X = x, Y = z − x. Hence, ∞ pz ( z ) = ∑ p (x, z − x) (5.29) p x (x) p y (z − x). (5.30) x=−∞ or if X + Y are independent ∞ pz ( z ) = ∑ x=−∞ The sum is called the convolution of px and p y. Example 5.4. Suppose that the number of channel openings at two distant (independent) sites along the frog neuromuscular junction in a 500 msec interval are Poisson random variables with parameters λ1 and λ2 . What is the probability mass function of the total number of channel openings in a 500 msec interval at the two sites? We have X : P(λ1 ), n {Z = X + Y = n} = Y : P(λ2 ) and let Z = X +Y U {X = k I Y = n − k}. Hence, we have k =0 We want to consider the event page 9: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables n Pr( X + Y = n) = ∑ Pr( X = k I Y = n − k) k =0 n = = ∑ Pr( X = k ) Pr(Y = n − k ) k =0 n − λ1 k e λ1 ∑ k! k =0 = e−(λ1 + λ2 ) = = e e−λ2 λ2n−k (n − k)! n ∑ k !(n − k )! k =0 − (λ1 + λ2 ) n n! n! ∑ k !(n − k)!λ1k λ2n−k k =0 n (λ1 + λ2 ) −(λ1 + λ2 ) n! (5.31) λ1k λ2n−k e . We see that the sum of two independent Poisson random variables is again Poisson with parameters λ1 + λ2 . In general, if Z = J ∑ j=1 X j where X j : P(λ j ) then Z : P( J ∑ λ j ). We will be able j=1 to establish this latter result rigorously once we develop the concept of the moment generating function in Lecture 6. Z=X+Y Figure 5.A. Set of X and Y values considered in the computation of a convolution of X+Y. 2. Continuous Random Variables Now we consider in the continuous case Z = X + Y ≤ z which is the region below the line x + y = z. Thus, if the joint pdf of X and Y is f ( x, y) we have page 10: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables Fz ( z ) = Pr( X + Y ≤ z ) = ∫∫R (5.32) f ( X ,Y )dxdy z = ∞ z−x ∫−∞ ∫−∞ f ( x, y)dydx Let y = v − x, then dy = dv and y = z − x implies v = y + x = z − x + x = z = If ∞ z ∫−∞ ∫−∞ f ( x,v − x)dvdx. (5.33) ∞ ∫−∞ f ( x, v − x)dx is continuous at z, then on differentiating, we have f z ( z) = ∞ ∫−∞ f ( x, z − x)dx (5.34) and if X and Y are independent we obtain f z ( z) = ∞ ∫−∞ f x ( x) f y ( z − x)dx (5.35) which is the convolution of f x and f y . X + Y = Z Figure 5B. Convolution of Two Uniform Random Variables Yields a Triangular Distribution. Example 5.5. Suppose X and Y are independent random variables both uniformly distributed on (0,1). Find Z = X + Y (Figure 5B). page 11: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables ⎧1 0 < x < 1 f x ( x) = ⎨ ⎩0 otherwise (5.36) ⎧1 0 < y < 1 f y ( y) = ⎨ ⎩0 otherwise z ∈(0, 2). To derive this density we must consider two cases: Case i) 0 < z ≤ 1 then x + y ≤ z and f z ( z) = 1 ∫0 f (x) f (z − x)dx = z ∫0 dx = z (5.37) Case ii) 1 < z < 2 then x + y = z such that 1 < z < 2 then we must have x ∈ (z −1,1) and f z ( z) = 1 ∫z−1 1 |z−1 = 1− (z −1) = 2 − z dx = x (5.38) f ( z ) is the triangle probability density. X + Y = Z Figure 5C. Convolution of Two Exponential Random Variables Yields a Gamma Distribution. Example 5.4 (continued). Suppose that the length of time that each of the two ion channels is open is an exponential random variable with parameters λ1 for the first and λ2 for the second. Because the channels are distant, we assume that they are independent. Find the probability density for the sum of the channel opening times if λ1 = λ2 = λ (Figure 5C). Let Z = X + Y , then we have f ( x, y) = f x ( x) f y ( y) = λ 2e−λ x e−λ y for x > 0 and y > 0 (5.39) page 12: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables f z ( z) = ∞ z 2 −λ x −λ (z − x ) ∫−∞ f x (x) f y (z − x)dx = ∫0 λ e e dx (5.40) z = λ 2 e−λ z dx 0 ∫ = λ 2 ze−λ z which is the probability density of a gamma random variable with α = 2 and β = λ. Note that the convolution here, as in the case of the uniform random variables, “smooths” out the probability densities and makes values in the middle more likely (Figure. 5C). We saw the start of a similar phenomenon with the sum of two uniform random variables in Example 5.5 (Figure 5B) having a triangular density. This is a key observation for understanding the Central Limit Theorem in Lecture 7. B. General Transformations of Two Random Variables We state the general case as a Proposition. Proposition 5.1. If X and Y are continuous random variables with joint pdf f ( x, y) suppose X and Y are mapped onto U and V by the transformation u = g1 ( x, y) (5.41) v = g 2 ( x, y) and the transformation can be inverted to obtain x = h1 (u , v) (5.42) y = h2 (u , v) Assume that g1 and g 2 have continuous partial derivatives, then the Jacobian is ⎡ ∂h1 ⎢ ∂u J (u , v) = ⎢ ⎢ ∂h2 ⎢⎣ ∂u ∂h1 ⎤ ∂v ⎥ ⎥ ∂h2 ⎥ ∂v ⎦⎥ and the absolute value of its determinant is | J (u, v) |= then ∂h1 ∂h2 ∂h2 ∂h1 − ∂u ∂v ∂u ∂v (5.43) page 13: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables fuv (u, v) = f xy (h1(u, v), h2 (u, v)) | J (u, v) | (5.44) for u = g1 ( x, y) v = g2 ( x, y) for some ( x, y) and 0 elsewhere. Notice that we used a form of Proposition 5.1 to compute the normalizing constant for the beta distribution in the Addendum to Lecture 3. Proof (Heuristic): The function g is a one-to-one mapping of a region in the ( x, y) plane into a region in the (u, v) plane. It must be the case that the probability for any small region in the ( x, y) plane must be equal to the probability of the corresponding small region in the (u, v) plane that it maps into under g . That is, fuv (u, v)d (u, v) = f xy ( x, y)d ( x, y) fuv (u, v) = f xy ( x, y) d ( x, y) d (u, v) fuv (u, v) = f xy ( x, y) | J (u, v) | fuv (u, v) = f xy (h1 (u, v), h2 (u, v)) | J (u, v) | where, by analogy with the univariate change-of-variables in Lecture 4, we define | J (u, v) | as the absolute value of the determinant of J (u, v) to insure that fuv (u,v) ≥ 0. Example 5.6. If X and Y are independent gamma random variables with parameters α and λ and β and λ respectively, find the joint pdf of U = X + Y and V = X (X + Y )−1. The joint density of X and Y is given by f x, y ( x, y) = λα α −1 −λ x λ β β −1 −λ y x e y e Γ(α ) Γ( β ) λα + β = xα −1 y β −1e−λ ( x + y) Γ(α )Γ( β ) Now we have u = g1 ( x, y) = x + y , The Jacobian is (5.45) v = g2 ( x, y) = x(x + y)−1 , x = h1 (u,v) = uv, and y = h2 (u, v) = u(1− v). page 14: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables ∂h1 =v ∂u ∂h2 = 1− v ∂u ∂h1 =u ∂v ∂h2 = −u ∂v (5.46) The absolute value of the determinant of the Jacobian is | J (u,v) |=| −uv −u(1− v) |= u . Thus, fuv (u, v) = f xy (uv, u(1− v))u = λα + β (uv)α −1[u(1 − v)]β −1 e−λu u Γ(α )Γ( β ) = λα + β uα + β −1e−λu vα −1 (1− v) β −1. Γ(α )Γ( β ) (5.47) Multiplying and dividing by Γ(α + β ) yields fuv (u, v) = λα + β α + β −1 −λu Γ(α + β ) α −1 u e v (1− v) β −1 Γ(α + β ) Γ(α )Γ( β ) (5.48) = fu (u ) f v (v) We see that fu (u) is the probability density of a gamma random variable with parameters α + β and λ , whereas fv (v) is the probability density of a beta random variable with parameters α and β . Therefore, the joint probability density fuv (u, v) is the product of a gamma and a beta density. By Lecture 4, because fuv (u,v) = fu (u) fv (v), we conclude that u and v are independent. Notice that the arguments used in Eqs. 5.45 to 5.48 are essentially the arguments used to compute the normalizing constant for the beta probability density in the Addendum to Lecture 3. Example 5.7. Let X and Y be independent standard Gaussian random variables. Find the joint density of U = X and V = X + Y. We have g1 ( x, y) = x g 2 ( x, y) = x + y (5.49) The Jacobian is x = h1 (u,v) = u y = h2 (u,v) = v − u page 15: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables ∂h1 ∂h1 =1 =0 ∂u ∂v ∂h2 ∂h2 = −1 =1 ∂u ∂v and the absolute value of its determinant is | J (u,v) |=1×1−1×0 =1. (5.50) fuv (u , v) = f xy (u, v − u) | J |= f x (u) f y (v − u) | J | 1 = (2π ) −1 exp{− [u 2 + (v − u) 2 ]}×1 2 1 2 2 −1 = (2π ) exp{− [u + v − 2uv + u 2 ]} 2 1 −1 = (2π ) exp{− [2u 2 + v 2 − 2uv]}. 2 (5.51) Therefore, Now matching up the terms on the right hand side of Eq. 5.51 with the terms in the joint density function of the bivariate Gaussian density in Eq. 4.32 yields µu = µv = 0 1 σ uσ v (1 − ρ 2 ) 2 = 1 and σ u2 (1 − ρ 2 ) = 1 2 ⇒ σ v2 = 2 σ v2 (1 − ρ 2 ) = 1 ⇒ ρ 2 = Therefore, U µu = µv = 0, σ u2 = 1, σ v2 and V have a (5.52) 1 1 or ρ = 2− 2 and σ u = 1. 2 bivariate Gaussian probability density with − 12 = 2 and ρ = 2 . Proposition 5.1 generalizes to n random variables (see Rice pp. 102-103). IV. Summary We have shown how to construct conditional pmf’s and pdf’s. In so doing, we have laid important groundwork for our statistical analyses. Similarly, understanding the convolution process that is necessary to compute the pmf or pdf of the sum of two independent random variables gives us key insight into how an important result like the Central Limit Theorem comes about. page 16: Lecture 5: Conditional Distributions and Functions of Jointly Distributed Random Variables Acknowledgments I am grateful to Paymon Hosseini for making the figures, James Mutch for helpful editorial comments and Julie Scott for technical assistance. References Lee W, Huang H, Feng G, Sanes JR, Brown EN, So PT, Nedivi E. Dynamic remodeling of dendritic arbors in non-pyramidal neurons of adult visual cortex. Public Library of Science – Biology. 4(2)e29, 2006. Rice JA. Mathematical Statistics and Data Analysis, 3rd edition. Boston, MA.