Continous Random Variables (I)

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Lecture 10 Fall 2018 CONTINUOUS RANDOM VARIABLES Contents 1. Continuous random variables 2. Examples 3. Expected values 4. Joint distributions 5. Independence 6. Radon-Nikodym derivative Readings: For a less technical version of this material, but with more discussion and examples, see Sections 3.1-3.5 of [BT] and Sections 4.1-4.5 of [GS]. 1 CONTINUOUS RANDOM VARIABLES Recall1 that a random variable X : Ω → R is said to be continuous if its CDF can be written in the form Z P(X ≤ x) = FX (x) = fX (t)dt, (−∞,x) for some nonnegative measurable function f : R → [0, ∞), which is called the Probability Density Function (PDF) of X. We then have, for any Borel set B, Z Z P(X ∈ B) = fX (x) dx = IB (x)fX (x) dx (1) B R Technical remark: All integrals from now Ron are understood as Lebesgue inteb grals, unless stated otherwise. In particular a f (t)dt is a shorthand notation for R R 1(a,b) (t)f (t)dλ(t), where λ is Lebesgue measure on (R, B). 1 The reader should revisit Section 4 of the notes for Lecture 5. 1 We note that fX should be more appropriately called “a” (as opposed to “the”) PDF of X, because it is not unique. For example, if we modify fX at a finite number of points, its integral is unaffected, so multiple densities can correspond to the same CDF. It turns out, however, that any two densities associated with the same CDF are equal except on a set of Lebesgue measure zero. A PDF is in some ways similar to a PMF, except that the value fX (x) cannot be interpreted as a probability. In particular, the value of fX (x) can to be greater than √ one for some x. Recall Example 8 from lecture 5. There the density was 1/(2 t) over t ∈ (0, 1] which is larger than one for small values of t. Instead, the proper intuitive interpretation is the fact that if fX is continuous over a small interval [x, x + δ], then P(x ≤ X ≤ x + δ) ≈ fX (x)δ. Also it is instructive to recall the fundamental theorem of calculus: If FX (x) is continuous and differentiable everywhere except countably many points of R, then Z x FX (x) = FX′ (t)dt −∞ This provides a simple rule to find PDF from CDF in most cases of practical interest. Remark: The fact that a random variable X is continuous has no bearing on the continuity of X as a function from Ω into R. In fact, we have not even defined what it means for a function on Ω to be continuous. But even in the special case where Ω = R, we can have a discontinuous function X : R → R which is a continuous random variable. Here is an example. Let the underlying probability measure on Ω be the Lebesgue measure on the unit interval. Let  ω, 0 ≤ ω ≤ 1/2, X(ω) = 1 + ω, 1/2 < ω ≤ 1. The function X is discontinuous. The random variable X takes values in the set [0, 1/2] ∪ (3/2, 2]. Furthermore, it is not hard to check that X is a continuous random variable with PDF given by  1, x ∈ [0, 1/2] ∪ (3/2, 2] fX (x) = 0 otherwise. 2 EXAMPLES We present here a number of important examples of continuous random variables. 2 2.1 Uniform This is perhaps the simplest continuous random [a, b], and let   0, (x − a)/(b − a), FX (x) =  1, variable. Consider an interval x ≤ a, a < x ≤ b, x > b. It is easy to check that FX satisfies the required properties of CDFs. We denote this distribution by U (a, b). We find that a corresponding PDF is given 1 for x ∈ [a, b], and fX (x) = 0, otherwise. by fX (x) = (dFX /dx)(x) = b−a When [a, b] = [0, 1], the probability law of a uniform random variable is just the Lebesgue measure on [0, 1]. 2.2 Exponential Fix some λ > 0. Let FX (x) = 1 − e−λx , for x ≥ 0, and FX (x) = 0, for x < 0. It is easy to check that FX satisfies the required properties of CDFs. A corresponding PDF is fX (x) = λe−λx , for x ≥ 0, and fX (x) = 0, for x < 0. We denote this distribution by Exp(λ) and write X ∼ Exp(λ). d (Recall notation = and ∼ which stand for ”distributed as ...”) The exponential distribution can be viewed as a “limit” of a geometric distribution. Indeed, if we fix some δ and consider the values of FX (kδ) = 1−e−λδk , for k = 1, 2, . . .. Check that this is P(Y ≤ k), where Y is geometrically distributed with parameter ρ = 1 − e−λδ . Intuitively, the exponential distribution corresponds to a limit of a situation where every δ time units, we toss a coin whose success probability is λδ, and let X be the time elapsed until the first success. We will revisit this intuition later on in the course. The distribution Exp(λ) has the following very important memorylessness property. Theorem 1. Let X be an exponentially distributed random variable. Then, for every x, t ≥ 0, we have P(X > x + t | X > x) = P(X > t). 3 Proof: Let X be exponential with parameter λ. We have P(X > x + t | X > x) = = P(X > x + t, X > x) P(X > x + t) = P(X > x) P(X > x) e−λ(x+t) = e−λt = P(X > t). e−λx Exponential random variables are often used to model memoryless arrival processes, in which the elapsed waiting time does not affect our probabilistic model of the remaining time until an arrival. For example, suppose that the time until the next bus arrival is an exponential random variable with parameter λ = 1/5 (in minutes). Thus, there is probability e−1 that you will have to wait for at least 5 minutes. Suppose that you have already waited for 10 minutes. The probability that you will have to wait for at least another five minutes is still the same, e−1 . Semigroup property of exponential: let X1 ∼ Exp(λ1 ), X2 ∼ Exp(λ2 ), and X1 ⊥⊥ X2 . then min(X1 , X2 ) ∼ Exp(λ1 + λ2 ) . note that min defines a commutative operation on R+ ∪{+∞} with +∞ serving the role of identity. We can see that exponential distribution relates + and min operations on R+ ∪ {+∞}: the addition of λ’s is equivalent to min of X’s. For this statement to hold in full, one naturally understands X ∼ Exp(+∞) as X = 0 a.s., and X ∼ Exp(0) as X = +∞ a.s.. 2.3 Normal distribution Perhaps the most widely used distribution is the normal distribution which is also called Gaussian distribution. It involves parameters µ ∈ R and σ > 0, and the density X ∼ N (µ, σ 2 ) : (x−µ)2 1 fX (x) = √ e− 2σ2 . σ 2π Note also that this PDF is symmetric around x = µ. Namely fX (µ + x) = fX (µ − x) for every x ∈ R. We need to show that this is a legitimate PDF, i.e., that it integrates to one. The special case µ = 0, σ = 1 corresponds to the claim Z ∞ t2 1 √ (2) e− 2 dt = 1 2π −∞ 4 and will be established later, when we deal with transformation of random variables. For now let us assume this and show that the same applies to the case of general µ, σ. We introduce a change of variables z = (t − µ)/σ implying dz = dt/σ. The range t ∈ (−∞, +∞) implies the range z ∈ (−∞, +∞). Then Z ∞ Z ∞ 2 z2 1 1 − (x−µ) 2 2σ √ √ = e− 2 = 1. e 2πσ −∞ 2π −∞ We use the notation N (µ, σ 2 ) to denote the normal distribution with parameters µ, σ. The distribution N (0, 1) is referred to as the standard normal distribution; a corresponding random variable is also said to be standard normal. There is no closed form formula for the corresponding CDF, but numerical tables are available. These tables can also be used to find probabilities associated with general normal variables. This is because of the fact that if X ∼ N (µ, σ 2 ), then (X − µ)/σ ∼ N (0, 1). . Thus, X − µ c − µ P(X ≤ c) = P ≤ = ((c − µ)/σ), σ σ where  is the CDF of the standard normal, available from the normal tables. Semigroup property of normal: Let X1 ∼ N (0, σ12 ), X2 ∼ N (0, σ22 ), and X1 ⊥⊥ X2 . Then X1 + X2 ∼ N (0, σ12 + σ22 ) . 2.4 Cauchy distribution Here, there is only one parameter t and X ∼ Ca(t) : fX (x) = 1 t ,x ∈ R 2 π t + x2 R∞ It is an exercise in calculus to show that −∞ f (t)dt = 1, so that fX is indeed a PDF. The corresponding distribution is called a Cauchy distribution. Semigroup property of Cauchy: Let X1 ∼ Ca(t1 ), X2 ∼ Ca(t2 ) and X1 ⊥⊥ X2 . Then X1 + X2 ∼ Ca(t1 + t2 ) 2.5 Gamma distribution Gamma distribution is parameterized by two positive reals: shape parameter a > 0 and (inverse) scale parameter c > 0. X ∼ �(a, c) : fX (x) = 5 ca xa−1 e−cx ,x > 0 �(a) Semigroup property of Gamma: Let X1 ∼ �(a1 , c), X2 ∼ �(a2 , c) and X1 ⊥⊥ X2 . Then X1 + X2 ∼ �(a1 + a2 , c) 2.6 Power law We have already defined discrete power law distributions. We present here a continuous analog. Our starting point is to introduce tail probabilities that decay according to power law: P(X > x) = β/xα , for x ≥ c > 0, for some parameters α, c > 0. In this case, the CDF is given by FX (x) = 1 − β/xα , x ≥ c, and FX (x) = 0, otherwise. In order for X to be a continuous random variable, FX cannot have a jump at x = c, and we therefore need β = cα and FX (x) = 1 − cα /xα . The corresponding density is of the form fX (t) = αcα dFX (t) = α+1 . dx t 3 EXPECTED VALUES Similar to the discrete case, given a continuous random variable X with PDF fX , we have a simple rule to compute the expectation: Z ∞ E[X] = xfX (x) dx. −∞ R∞ (This was shown in Lecture 8.) This integral is well defined and finite if −∞ |x|fX (x) dx < ∞, in which case we say that the random variable X is integrable. The integral is also well defined, R0 R ∞ but infinite, if one, but not both, of the integrals xf (x) dx and X −∞ 0 xfX (x) dx is infinite. If both of these integrals are infinite, the expected value is not defined. Practically all of the results developed for discrete random variables carry over to the continuous case. Many of them, e.g., E[X + Y ] = E[X] + E[Y ], have the exact same form. We list below two results in which sums need to be replaced by integrals. Proposition 1. Let X be a nonnegative random variable, i.e., P(X < 0) = 0. Then Z ∞ E[X] = (1 − FX (t)) dt. 0 6 Proof: We have Z ∞ Z (1 − FX (t)) dt = 0 ∞ Z Z P(X > t) dt = dt 1{X(ω) > t}dP(ω) Ω 0 R+ Z  Z Z = dP(ω) 1{t < X(ω)}dt = X(ω)dP , E[X] Ω Ω R+ (The interchange of the order of integration is by Fubini’s theorem for nonnegative functions.) Proposition 2. Let X be a continuous random variable with density fX , and suppose that g : R → R is a (Borel) measurable function. Then Z ∞ E[g(X)] = g(t)fX (t) dt −∞ (i.e. the integral and the expectation exist or do not exist simultaneosly, and are equal in the latter case). Proof: This was shown in Lecture 8. Note that for this result to hold, the random variable g(X) need not be continuous. 4 JOINT DISTRIBUTIONS Definition 1. Given a pair of random variables X and Y , defined on the same probability space, their joint distribution PX,Y is a probability measure on (R × R, B × B) defined as PX,Y [B] , P[(X, Y ) ∈ B] for every B ∈ B × B. Exercise: Show that set {ω : (X(ω), Y (ω)) ∈ B} is measurable for any B ∈ B × B. (This provides another justification for the definition of product σ-algebra.) The joint CDF of X, Y is defined as FX,Y (x, y) = P[X ≤ x, Y ≤ y] and we say that X, Y are jointly continuous if there exists a measurable fX,Y : 7 R2 → [0, ∞) such that their joint CDF satisfies Z x Z FX,Y (x, y) = P(X ≤ x, Y ≤ y) = −∞ y fX,Y (u, v) du dv. −∞ The function fX,Y is called a joint PDF of X and Y . At those points where a joint PDF is continuous, we have ∂2F ∂2F (x, y) = (x, y) = fX,Y (x, y). ∂x∂y ∂y∂x Similar to what was mentioned for the case of a single random variable, for any Borel subset B of R2 , we have Z Z 1B (x, y)fX,Y (x, y) dx dy. (3) fX,Y (x, y) dx dy = PX,Y [B] = R2 B Furthermore, if B has Lebesgue measure zero, then PX,Y (B) = 0. We observe that by (3) and Fubini’s theorem Z x Z ∞ P(X ≤ x) = fX,Y (u, v) du dv. −∞ −∞ Thus, X itself is a continuous random variable, with marginal PDF Z ∞ fX (x) = fX,Y (x, y) dy. −∞ We have just argued that if X and Y are jointly continuous, then X (and, similarly, Y ) is a continuous random variables. The converse is not true. For a trivial counterexample, let X be a continuous random variable, and let and Y = X. Then the set {(x, y) ∈ R2 | x = y} has zero area (zero Lebesgue measure), but unit probability, which is impossible for jointly continuous random variables. In particular, the corresponding probability law on R2 is neither discrete nor continuous, hence qualifies as “singular.” Proposition 2 has a natural extension to the case of multiple random variables. Proposition 3. Let X and Y be jointly continuous with PDF fX,Y , and suppose that g : R2 → R is a (Borel) measurable function such that g(X) is integrable. Then, Z ∞Z ∞ E[g(X, Y )] = g(u, v)fX,Y (u, v) du dv. −∞ −∞ 8 4.1 Multivariate measureable functions A non-trivial assumption is for the joint PDF fX,Y to be a measurable function on R2 . How can we ensure that? Of course simple functions are measurable, as are their limits, limsup’s and liminf’s. However, a criterion frequently used in practice is the following. (It is also an excellent exercise for getting some practice with product σ-algebras!) Proposition 4. Let φ(x, y) be a function on R2 such that 1. y 7→ φ(x, y) is measurable for every fixed x ∈ R 2. x 7→ φ(x, y) is right-continuous for every fixed y ∈ R Then φ is jointly measurable in (x, y). Proof. First, it is instructive to understand the proof of Borel measurability of any right-continuous function x 7→ f (x). Let Ra = {x ∈ Q : f (x) < a}. Then for any a ∈ R it follows [ \ [ (4) {f (x) < a} = [r − ǫ2 , r] . ǫ1 >0 ǫ2 >0 r∈Ra−ǫ1 Here we write (abusing notation) ∪ǫ>0 to mean union over arbitrary sequence of ǫn ց 0, so that resulting operations are countable. If (4) holds then {f (x) < a} belongs to B and thus f is Borel. To understand (4)2 note that the set \ [ Lb , [r − ǫ2 , r] ǫ2 >0 r∈Rb corresponds to all points on the real-line R that are decreasing limits of elements of Rb . For a right continuous function f f (x) < b ⇒ x ∈ Lb ⇒ f (x) ≤ b And thus (4) follows. Now to prove Proposition, we only need to notice that sets {y : φ(r, y) < a} are measurable subsets of R by assumption. Hence, by setting \ [ Lb , [r − ǫ2 , r] × {y : φ(r, y) < b} ǫ2 >0 r∈Q 2 It may also be helpful to remember that right-continuity of f : R ! R is equivalent to usual continuity when topology on the domain is refined declaring sets [a, b) open. 9 we infer that φ(x, y) < b ⇒ (x, y) ∈ Lb ⇒ φ(x, y) ≤ b . Thus, we have {(x, y) : φ(x, y) < a} = [ La−ǫ ǫ>0 which is a countable combination of measurable rectangles. 5 INDEPENDENCE Recall that two random variables, X and Y , are said to be independent if for any two Borel subsets, B1 and B2 , of the real line, we have P(X ∈ B1 , Y ∈ B2 ) = P(X ∈ B1 )P(Y ∈ B2 ). This is equivalent to saying PX,Y = PX × PY , which explains why product of measures corresponds to independence. Similar to the discrete case (cf. Proposition 1 and Theorem 1 in Section 3 of Lecture 5), simpler criteria for independence are available. Theorem 2. Let X and Y be jointly continuous random variables defined on the same probability space. The following are equivalent. (a) The random variables X and Y are independent. (b) For any x, y ∈ R, the events {X ≤ x} and {Y ≤ y} are independent. (c) For any x, y ∈ R, we have FX,Y (x, y) = FX (x)FY (y). (d) If fX , fY , and fX,Y are corresponding marginal and joint densities, then fX,Y (x, y) = fX (x)fY (y), for all (x, y) except possibly on a set that has Lebesgue measure zero. The proof parallels the proofs in Lecture 6, except for the last condition, for which the argument is simple when the densities are continuous functions (simply differentiate the CDF), but requires more care otherwise. 6 RADON-NIKODYM DERIVATIVE In this section, we address a natural question: Given a random variable X (or X, Y ) how do we know if it is (jointly) continuous? Notice that Lebesgue measure plays a distinguished role in the definition of continuity. Thus a more general approach requires the following definition: 10 Definition 2. Let (Ω, F, λ) be a measure space. Let µ be another measure on (Ω, F). Then function f : Ω → R+ is called a Radon-Nikodym derivative dµ dλ if Z µ[E] = f dλ E for any E ∈ F. According to this definition: X is a continuous random variable if and only dPX if there exists a Radon-Nikodym derivative dLeb on R. Similarly, X and Y are dPX,Y 2 jointly continuous if dLeb exists on R , etc. One simple consequence of (1) is that X cannot be a continuous random variable if PX has atoms, namely if P[X = a] 6= 0 for some a ∈ R. However, as “singular” example in Section 4 shows the absence of atoms is not sufficient for continuity. The following definition and Theorem describe the necessary and sufficient condition: Definition 3. Measure µ is absolutely continuous with respect to λ (notation: µ ≪ λ), if for every E λ(E) = 0 ⇒ µ(E) = 0 . Note that from (1) we see: if X is continuous then PX ≪ Leb and similarly for joint continuity. Remarkably, the converse holds as well: Theorem 3 (Radon-Nikodym). Let µ and λ be σ-finite measures on (Ω, F). There exists a Radon-Nikodym derivative dµ dλ if and only if µ ≪ λ. Proof of this theorem is outside of the scope of this class (mainly it relies on some basic properties of Hilbert spaces). 11