Continous Random Variables (II)

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Lecture 11 Fall 2018 CONTINUOUS RANDOM VARIABLES - II Contents 1. Review of joint distributions 2. From conditional distribution to joint (Markov kernels) 3. From joint to conditional (disintegration) 4. Example: The bivariate normal distribution 5. Conditional expectation 6. Mixed versions of Bayes’ rule 1 REVIEW OF JOINT DISTRIBUTIONS Recall that two random variables X and Y are said to be jointly continuous if there exists a nonnegative measurable function fX,Y such that Z x Z y P(X ≤ x, Y ≤ y) = fX,Y (u, v) dv du. −∞ −∞ Once we have in our hands a general definition of integrals, this can be used to establish that for every Borel subset of R2 , we have P((X, Y ) ∈ B) = ∫f X,Y (u, v) du dv. B Furthermore, X is itself a continuous random variable, with density fX given by fX (x) = ∫ ∞ fX,Y (x, y) dy. −∞ 1 R Finally, recall that E[g(X)] = g(x)fX (x) dx. Similar to the discrete case, the expectation of g(X) = X m and g(X) = (X − E[X])m is called the mth moment and the mth central moment, respectively, of X. In particular, var(X) , E[(X − E[X])2 ] is the variance of X. We note that all of the definitions and formulas have obvious extensions to the case of more than two random variables. 2 MARKOV KERNELS Random variables X and Y endowed with a product measures PX × PY are necessarily independent X ⊥⊥ Y . How do we construct PX,Y for dependent variables? One method is to define X and Y on the same probability space and compute PX,Y using the Definition given in Lecture 10. Another method involves the following concept: Definition 1. K : Ω1 × F2 → [0, 1] is called a transition probability kernel (or a Markov kernel) acting from (Ω1 , F1 ) to (Ω2 , F2 ) if: 1. K(ω1 , ·) is a probability measure on (Ω2 , F2 ) for each ω1 ∈ Ω1 2. ω1 → K(ω1 , B) is an F1 -measurable function for each B ∈ F2 . In some disciplines, it is common to abuse notation and say “Let K : (Ω1 , F1 ) → (Ω2 , F2 ) be a Markov kernel” or even “Let K : Ω1 → Ω2 be a Markov kernel”, even though K is not a map between spaces. Example: When Ω1 and Ω2 are finite, any Markov kernel K acting from (Ω1 , 2Ω1 ) to (Ω2 , 2Ω2 ) is simply an |Ω1 | × |Ω2 | matrix of non-negative values with row-sums all equal to 1. Such matrices are called stochastic (or rightstochastic, or row-stochastic). Example: The following transition probability kernel acts between (R, B) and (R, B). It is called the additive Gaussian noise channel: K(x, dy) = √ 1 2πσ 2 e− (y−x)2 2σ 2 dy x, y ∈ R. This kernel “blurs” every point into a Gaussian cloud of width σ. 2 Theorem 1. For any probability measure P1 and transition probability kernel K there exists a unique probability measure π (denoted P1 × K ) on (Ω1 × Ω2 , F1 × F2 ) such that π(A × B) = ∫ K(ω , B)P (dω ). 1 1 1 A Furthermore, whenever f ≥ 0 or f is π -integrable we have ∫ f (ω1 , ω2 )dπ = Ω1 ×Ω2 ∫ P1 (dω1 ) Ω1 ∫ f (ω1 , ω2 )K(ω1 , dω2 ) . (1) Ω2 Proof. Repeat the steps in the proofs of Theorems 2 and 3 in Lecture 9 with trivial modifications. The measure π on Ω1 × Ω2 corresponds to the following stochastic experiment: • Draw ω1 in accordance with distribution P1 (·). • Then draw ω2 in accordance with distribution K(ω1 , ·). • Output pair (ω1 , ω2 ). Caution: Many different kernels can lead to the same product measure, i.e. P1 × K = P1 × K ′ ̸⇒ K = K ′ . Indeed if P1 (A) = 0, then K(x, ·) can be defined arbitrarily for all x ∈ A without affecting the product measure. 2.1 Measure-kernel-function Markov kernels can act on functions and on measures and these actions are associative. Proposition 1. Let K be a Markov kernel from (Ω1 , F1 ) to (Ω2 , F2 ). Then 1. The kernel K pulls back non-negative functions f on Ω2 to non-negative functions on Ω1 : Z (Kf )(ω1 ) , f (ω2 )K(ω1 , dω2 ), Ω2 and this map ω1 → (Kf )(ω1 ) is F1 measurable. 3 2. The kernel K pushes forward probability measures from Ω1 to Ω2 . Namely for each µ on (Ω1 , F1 ) there exists a unique probability measure ν = µK on (Ω2 , F2 ) satisfying ν(B) = ∫ K(ω1 , B)dµ. (2) Ω1 3. These actions are compatible: for any µ on Ω1 and f ≥ 0 on Ω2 ∫ (Kf )(ω1 ) dµ = Ω1 ∫ f (ω2 ) dν . (3) Ω2 Proof. We only sketch the details. For 1 notice that measurability of Kf for simple functions follows from the definition of Markov kernel. This extends to general functions by taking limits. For 2 notice that by the MCT assignment (2) indeed defines a σ-additive probability measure. Finally, 3 is obvious for simple functions and otherwise take limits. R It is common to denote the integral f dµ as µf or µ(f ), i.e. the action of µ on f . In such notation, result (3) can be stated as (µK)f = µ(Kf ), (4) and this justifies the so-called measure-kernel-function notation: µKf (without parentheses). When Ω1 and Ω2 are finite it is customary to represent a measures µ as a row-vector, a kernel K as a stochastic matrix and a function f as as column vector. In that case, (4) is equivalent to associativity of matrix multiplication. 2.2 Conditional CDFs and PDFs Here we give a general method for constructing Markov kernels (and via Theorem 1 – joint distributions PX,Y ). Proposition 2. The following define Markov kernels acting from (R, B) to itself: (a) Let fX|Y (x|y) be a non-negative function jointly measurable in (x, y) and satisfying1 ∫ fX|Y (x|y) dx = 1 R y ∈ R, (5) then K(y, dx) = fX|Y (x|y)dx defines a Markov kernel. 1 Such functions are known as conditional PDFs. 4 (6) (b) Let FX|Y (x|y) be a function jointly measurable in (x, y), such that FX|Y (·|y) is a CDF 2 for every y ∈ R. Then there exists a unique Markov kernel s.t. K(y, (a, b]) = FX|Y (b|y) − FX|Y (a|y). (7) Proof. Part (a) is easy: (6) is a measure for every fixed y by (5). The function y→ ∫ fX|Y (x|y)dx B is measurable for every B ∈ B by Fubini’s theorem. For Part (b) again (7) extends to a unique probability measure. We need to verify that the map y → K(y, B) is measurable for every B. For B = ∪ ni=1 (ai , bi ] – a finite disjoint union of intervals – this follows from (7) and measurability of finite sums. Then define the collection: L = {B ∈ B : y → K(y, B) –measurable function}. We have shown that L contains algebra of finite unions of intervals (a, b]. It is easy to show that L is a monotone class. Thus, L = B and we are done. Example. Take PDF fY and conditional PDF fX|Y . Let PY (dy) = fY (y)dy K(y, dx) = fX|Y (x|y)dx (8) (9) Then the product measure π = PY × K constructed in Theorem 1 satisfies π(dx dy) = fX|Y (x|y)fY (y) dx dy In particular, π is a jointly continuous distribution with density fX,Y = fX|Y fY . 3 DISINTEGRATION OF JOINT DISTRIBUTIONS Main question we will address here: given PX,Y does there exist PY and K such that PX,Y = PY × K? 2 Such functions FX|Y are known as conditional CDFs. 5 Definition 2. Let PX,Y be a probability measure on (R2 , B 2 ) with marginal PY . A Markov kernel K(y, ·) is called a regular branch of conditional probability for X given Y (denoted PX|Y (·|y)) if PX,Y = PY × K (10) in the sense of Theorem 1. Equivalently, if PX,Y [A × B] = ∫ K(y, A) dPY (11) B for all A, B ∈ B . Furthermore, if K is defined via (6) then fX|Y is called a conditional PDF, and if K is defined via (7) then FX|Y is called a conditional CDF. Note: One should not confuse “a regular branch of conditional probability” (which is a Markov kernel) with conditional probability P[X ∈ A|Y ] (which is a random variable; see below). It should also be clear that neither PX|Y (a kernel), nor fX|Y (a conditional PDF, when it exists), nor FX|Y (a conditional CDF) are unique. Finally, equivalence of (10) and (11) follows from the fact that {A × B} is a generating p-system for B × B. 3.1 Simple case: jointly-continuous PX,Y For the case of discrete random variables, the conditional CDF is defined by FX|Y (x | y) = P(X ≤ x | Y = y), for any y such that P(Y = y) > 0. However, this definition cannot be extended to the continuous case because P(Y = y) = 0, for every y. Instead, we should think of FX|Y (x | y) as a limit of P(X ≤ x | y ≤ Y ≤ y + δ), as δ decreases to zero. Note that FX|Y (x | y) ≈ P(X ≤ x | y ≤ Y ≤ y + δ) P(X ≤ x, y ≤ Y ≤ y + δ) P(y ≤ Y ≤ y + δ) R x R y+δ f (u, v) dv du = ≈ ≈ = X,Y −∞ y δ R δfY (y) x −∞ fX,Y (u, y) du δfY (y) −∞ fX,Y (u, y) du Rx fY (y) 6 . This heuristic motivates the next result. Proposition 3. Let fX,Y be a joint PDF. Then (a) Let fY be (any) marginal PDF of Y . Then the following is a conditional CDF of X given Y FX|Y (x | y) = ∫ x −∞ fX,Y (u, y) du, fY (y) R for every y Rsatisfying the following: a) fY (y) > 0; b) fX,Y (u, y) du < ∞; and c) fX,Y (u, y) du = fY (y). For other y we set FX|Y (x|y) = 1{x ≥ 0}. (b) The following is a conditional PDF of X given Y fX|Y (x | y) = fX,Y (x, y) , fY (y) for every y such that fY (y) > 0. For any other y we set fX|Y (x | y) = 1{0 ≤ x ≤ 1}. Proof. Joint measurability in (x, y) follows from Fubini (in both cases). Next, it is clear that PY -measure of all y’s satisfying conditions a)-c) is 1. Thus definition of FX|Y for other y’s is immaterial for (11). For “good” y’s from the DCT we have lim x↘x0 ∫ x fX,Y (u, y) du = −∞ ∫ x0 fX,Y (u, y) du , −∞ which shows that FX|Y (·|y) is right-continuous. FX|Y is clearly monotone. The property limx→−∞ FX | Y (x | y) = 0 follows from the DCT again. Also lim FX|Y (x | y) = x→∞ ∫ ∞ −∞ fX,Y (u, y) du = 1, fY (y) since the integral of the numerator is exactly fY (y), by condition c). The proof concludes by a verification of (11) which is left as an exercise. 3.2 General case: arbitrary PX,Y Theorem 2 (Disintegration). Let PX,Y be a probability measure on (R2 , B 2 ). Then there exists a regular branch of conditional probability PX|Y (·|y) of X given Y , i.e. PX,Y = PY × PX|Y . 7 We will prove this result in Section 5.1. We note that similar disintegration works for product spaces other than R × R. E.g. X can take values in any complete metric space (not just R), while Y can be arbitrary. For the proof see [Cinlar, Section II.4.2]. 4 EXAMPLE: THE BIVARIATE NORMAL DISTRIBUTION Let us fix some ρ ∈ (−1, 1) and consider the function, called the standard bivariate normal PDF, f (x, y) =  x2 − 2ρxy + y 2  exp − . 2(1 − ρ2 ) 1 − ρ2 1 2π p Let X and Y be two jointly continuous random variables, defined on the same probability space, whose joint PDF is f . Therefore, their law satisfies PX,Y (dx, dy) = f (x, y) dx dy. Proposition 4. (a) The function f is a indeed a PDF (integrates to 1). (b) The marginal density of X and Y is N (0, 1), the standard normal PDF. (c) We have ρ(X, Y ) = ρ. Also, X and Y are independent iff ρ = 0. (d) The conditional density of X , given Y = y , is N (ρy, 1 − ρ2 ). In other words, 2 1 − (x−ρy)2 2(1−ρ ) dx e K(y, dx) = p 2π(1 − ρ2 ) is a regular branch of conditional probability for X given Y (i.e. PX,Y = N (0, 1) × K ). Interpretation of (d): X = ρY + normals. p 1 − ρ2 Z, where Y ⊥⊥ Z are standard √ 2 ) is a PDF Proof: We will use repeatedly the fact that 1/( 2πσ) exp(− (x−µ) 2σ2 (namely, the PDF of the N (µ, σ 2 ) distribution), and thus integrates to one. (a)-(b) We note that x2 − 2ρxy + y 2 = x2 − 2ρxy + ρ2 y 2 + (1 − ρ2 )y 2 , and 8 obtain   2 )y 2 Z ∞  (x − ρy )2  exp − (1−ρ 2 2(1−ρ ) p fY (y) = f (x, y)dx = exp − dx 2(1 − ρ2 ) 2π 1 − ρ2 −∞ −∞ Z  (x − ρy )2  exp(−y 2 /2) ∞ p exp − dx = 2(1 − ρ2 ) 2π 1 − ρ2 −∞ Z ∞ But we recognize p ∞  (x − ρy )2  exp − dx 2(1 − ρ2 ) 2π(1 − ρ2 ) −∞ 1 Z as the PDF of the N (ρy, 1 − ρ2 ) distribution. Thus, the integral of this density equals one, and we obtain exp(−y 2 /2) √ , 2π R∞ which is the standard normal PDF. Since −∞ fY (y) dy = 1, we conclude that f (x, y) integrates to one, and is a legitimate joint PDF. Furthermore, we have verified that the marginal PDF of Y (and by symmetry, also the marginal PDF of X) is the standard normal PDF, N (0, 1). fY (y) = (c) We have Cov(X, Y ) = E[XY ] − E[X]E[Y ] = E[XY ], since X and Y are standard normal, and therefore have zero mean. We now have E[XY ] = ∫∫ xyf (x, y) dy dx. Applying the same trick as above, we obtain for every y, Z Z  (x − ρy )2  exp(−y 2 /2) ∞ p dx. x exp − xf (x, y)dx = 2(1 − ρ2 ) 2π 1 − ρ2 −∞ But p ∞  (x − ρy )2  x exp − dx = ρy, 2(1 − ρ2 ) 2π(1 − ρ2 ) −∞ 1 Z since this is the expected value for the N (ρy, 1 − ρ2 ) distribution. Thus, E[XY ] = ∫ ∫ xyf (x, y) dx dy = ∫ yρyf (y) dy = ρ ∫ y f (y) dy = ρ, Y 9 2 Y since the integral is the second moment of the standard normal, which is equal to one. We have established that Cov(X, Y ) = ρ. Since the variances of X and Y are equal to unity, we obtain ρ(X, Y ) = ρ. If X and Y are independent, then ρ(X, Y ) = 0, implying that ρ = 0. Conversely, if ρ = 0, then  x2 + y 2  1 f (x, y) = exp − = fX (x)fY (y), 2π 2 and therefore X and Y are independent. Note that the condition ρ(X, Y ) = 0 implies independence, for the special case of the bivariate normal, whereas this implication is not always true, for general random variables. (d) Let us now compute the conditional PDF. Using the expression for fY (y) f (x, y) fY (y)  x2 − 2ρxy + y 2 √ 1 2π exp(y 2 /2) = p exp − 2) 2 2(1 − ρ 2π 1 − ρ  x2 − 2ρxy + ρ2 y 2  1 =p exp − 2(1 − ρ2 ) 2π(1 − ρ2 )  (x − ρy )2  1 =p exp − , 2(1 − ρ2 ) 2π(1 − ρ2 ) fX|Y (x | y) = which we recognize as the N (ρy, 1 − ρ2 ) PDF. We have discussed above the special case of a bivariate normal PDF, in which the means are zero and the variances are equal to one. More generally, the bivariate normal PDF is specified by five parameters, µ1 , µ2 , σ1 , σ2 , ρ, and is given by  1  1 p f (x, y) = exp − Q(x, y) , 2 2πσ1 σ2 1 − ρ2 where Q(x, y) = 1 h (x − µ1 )2 (x − µ1 ) (y − µ2 ) (y − µ2 )2 i − 2ρ + . 1 − ρ2 σ1 σ2 σ12 σ22 For this case, it can be verified that E[X] = µ1 , var(X) = σ12 , E[Y ] = µ2 , 10 var(Y ) = σ22 , ρ(X, Y ) = ρ. These properties can be derived by extending the tedious calculations in the preceding proof. There is a further generalization to more than two random variables, resulting in the multivariate normal distribution. It will be carried out in a more elegant manner in a later lecture. 5 CONDITIONAL EXPECTATION P Recall that in the discrete case we defined E[X|Y = y] = x∈R xpX|Y (x|y). We have also defined E[X | Y ] to be a a random variable that takes the value E[X | Y = y], whenever Y = y and P[Y = y] > 0. The general case is more delicate: Definition 3. Let X be integrable. A function g(y) is a conditional expectation, denoted E[X|Y = y], of X given Y if E[f (Y )X] = E[f (Y )g(Y )] (12) for every bounded measurable f . The random variable g(Y ), denoted E[X|Y ], is also called a conditional expectation of Y given X . In the special case of X = 1B we write P[B|Y = y] or P[B|Y ] to denote a conditional probability of B given Y . 11 Theorem 3. Let X, Y be random variables defined on a common probability space and X integrable. (a) A conditional expectation E[X|Y ] exists. (b) If g1 and g2 are two conditional expectations of X given Y then P[g1 (Y ) ̸= g2 (Y )] = 0 . (13) (c) If K is a regular branch of conditional probability of X given Y then g(y) = ∫ xK(y, dx) (14) R is a conditional expectation E[X|Y = y]. In particular, if a conditional PDF fX|Y exists then g(y) = ∫ xfX|Y (x|y)dx . R Note: Conditional expectation is not unique. However as (13) shows – this non-uniqueness is immaterial in most cases. Nevertheless, it is a mistake (and a very common one!) to ask for the value of P[B|Y = 0], which can be set to anything unless P[Y = 0] > 0. The correct question is to find a function y → P[B|Y = y] (defined upto almost-sure equivalence). Proof. (a) Let X = X + − X − and define for any Borel set B ν + (B) , E[1B (Y )X + ] which evidently defines a finite (E[X + ] < ∞) measure on (R, B). Furthermore, if PY (B) = 0 then {Y ∈ B} has probability 0 and thus ν ≪ PY . By RadonNikodym theorem there exists a measurable function g + such that E[1B (Y )X + ] = E[g + (Y )1B (Y )]. Similarly, we may define ν − (B) via X − and apply Radon-Nikodym theorem to get g − . Setting g = g + − g − we have for every Borel set B: E[1B (Y )X] = E[1B (Y )g(Y )]. Thus, g(Y ) verifies (12) for all f = 1B . By linearity of expectation (12) is also verified for all simple functions. The general case of bounded f follows by the DCT. 12 (b) If g1 (Y ) and g2 (Y ) are two conditional expectations then setting f (Y ) = 1{g1 (Y ) > g2 (Y )} − 1{g1 (Y ) < g2 (Y )} from (12) we get E[|g1 (Y ) − g2 (Y )|] = 0 implying g1 = g2 with PY -probability 1. (c) By definition if K is a regular branch of conditional expectation then PX,Y = PY × K, which by Theorem 1 implies ∫ R2 φ(x, y)PX,Y (dx dy) = ∫ PY (dy) R ∫ φ(x, y)K(y, dx) . R Taking φ(x, y) = xf (y) and using integrability of X property (12) follows by Fubini. Example. One might expect that when X and Y are jointly continuous, then E[X | Y ] is a continuous random variable, but this is not the case. To see this, suppose that X and Y are independent, in which case E[X | Y = y] = E[X], which also implies that E[X | Y ] = E[X]. Thus, E[X | Y ] takes a constant value, and is therefore a trivial case of a discrete random variable. Example. We have a stick of unit length [0, 1], and break it at X, where X is uniformly distributed on [0, 1]. Given the value x of X, we let Y be uniformly distributed on [0, x], and let Z be uniformly distributed on [0, 1 − x]. We assume that conditioned on X = x, the random variables Y and Z are independent. We are interested in the distribution of Y and Z, their expected values, and the expected value of their product. It is clear from symmetry that Y and Z have the same marginal distribution, so we focus on Y . Let us first find the joint distribution of Y and X. We have fX (x) = 1, for x ∈ [0, 1], and fY |X (y | x) = 1/x, for y ∈ [0, x]. Thus, the joint PDF is 1 1 ·1= , x x fX,Y (x, y) = fY |X (y | x)fX (x) = We can now find the PDF of Y : fY (y) = ∫ 1 fX,Y (x, y) dx = ∫ 1 fX,Y (x, y) dx = y 0 ∫ y 1 0 ≤ y ≤ x ≤ 1. 1 dx = log x x 1 y = log(1/y). (check that this indeed integrates to unity). Integrating by parts, we then obtain E[Y ] = ∫ 1 yfY (y)dy = 0 ∫ 0 1 y log(1/y) dy = 1 . 4 The above calculation is more involved than necessary. For a simpler argument, simply observe that E[Y | X = x] = x/2, since Y conditioned on X = x is uniform on 13 [0, x]. In particular, E[Y | X] = X/2. It follows that E[Y ] = E[E[Y | X]] = E[X/2] = 1/4. For an alternative version of this argument, consider the random variable Y /X. Conditioned on the event X = x, this random variable takes values in the range [0, 1], is uniformly distributed on that range, and has mean 1/2. Thus, the conditional PDF of Y /X is not affected by the value x of X. This implies that Y /X is independent of X, and we have E[Y ] = E[(Y /X)X] = E[Y /X] · E[X] = 1 1 1 · = . 2 2 4 To find E[Y Z] we use the fact that, conditional on X = x, Y and Z are independent, and obtain E[Y Z] = E E[Y Z | X] = E E[Y | X] · E[Z | X] 1 X 1−X( x(1 − x) 1 =E · = dx = . 4 24 2 2 0 ∫ Exercise 1. Find the joint PDF of Y and Z. Find the probability P(Y + Z ≤ 1/3). Find E[X|Y ], E[X|Z], and ρ(Y, Z). 5.1 Other properties of E[·|Y ] We note that many properties of Lebesgue integration carry over without change to conditional expectation: 1. Monotonicity: X ≤ X ′ ⇒ E[X|Y ] ≤ E[X ′ |Y ] 2. Linearity: E[aX + bX ′ |Y ] = aE[X|Y ] + bE[X ′ |Y ] 3. MCT: 0 ≤ Xn ↗ X ⇒ E[Xn |Y ] ↗ E[X|Y ] 4. DCT: |Xn | ≤ Z, Z-integrable, Xn → X, then ⇒ E[Xn |Y ] → E[X|Y ] 5. Fatou’s lemma: Xn ≥ 0, E[lim inf n Xn |Y ] ≤ lim inf n E[Xn |Y ] 6. Jensen’s inequality: f convex ⇒ E[f (X)|Y ] ≥ f (E[X|Y ]) Caution: Right-hand sides of each of these implications only hold almost surely! Proofs of all of these are simple: assume right-hand side is violated on a set E with P[Y ∈ E] > 0, then using 1E and (12) construct a counter-example to the uncoditional version of the same property. As an application we prove disintegration theorem: 14 Proof of Theorem 2. Let {rn }∞=1 be enumeration of rational numbers Q. Denote gn (y) , P[X ∈ (−∞; rn ] | Y = y] . By monotonicity property for any k and n such that rk ≤ rn we have P[gk (Y ) ≤ gn (Y )] = 1 Therefore the set E0 = {y : gk (y) ≤ gn (y) ∀(k, n) : rk ≤ rn } has PY -measure 1. Similarly, sets E1 = {y : inf gn (y) = 0} (15) E2 = {y : sup gn (y) = 1} (16) n n both also have PY -measure 1. All together, for every y in the set E , E0 ∩ E1 ∩ E2 closure of the sequence of points (rn , gn (y)) on R × [0, 1] is a graph of a CDF. Thus, we may define: ( 1{x ≥ 0}, y ̸∈ E FX|Y (x|y) = sup{gn (y) : rn ≤ x}, y ∈ E Notice that y → FX|Y (x|y) is measurable (as a countable supremum of measurable gn ’s). And the function x → FX|Y (x|y) is right-continuous, monotonically non-decreasing and growing from 0 to 1 on R. Thus by Proposition ?? function FX|Y is jointly measurable. Consequently, it satisfies all requirements of a conditional CDF and by Proposition 2.(b) there exists a Markov kernel K(y, dx) satisfying (7). But then for every set (−∞, rn ] × B we have ∫ P (dy)K(y, (−∞, r ]) = ∫ g (y)P (dy) (PY × K)((−∞, rn ] × B) = Y n (17) B n Y (18) B = E[1(−∞,rn ] (X)1B (Y )] (19) = PX,Y ((−∞, rn ] × B) , (20) 15 where first step is by (7), second by definition of FX|Y (rn |y) and since PY (E) = 1, third is by definition of gn and (12), and fourth is just by definition of PX,Y . Since sets (−∞, rn ] × B form a generating p-system for B × B we conclude PX,Y = PY × K which proves the Theorem. 5.2 Optimality properties of conditional expectations The conditional expectation E[X | Y ] can be viewed as an estimate of X, based on the value of Y . In fact, it is an optimal estimate, in the sense that the mean square of the resulting estimation error, X − E[X | Y ], is as small as possible. Theorem 4. Suppose that E[X 2 ] < ∞. Then, for any measurable function g : R → R, we have     E (X − E[X | Y ])2 ≤ E (X − g(Y ))2 . Proof: We have      E (X − g(Y ))2 = E[(X − E[X | Y ])2 + E (E[X | Y ] − g(Y ))2  +2E[(X − E[X | Y ])(E[X | Y ] − g(Y ))  ≥ E[(X − E[X | Y ])2 .   The inequality above is obtained by noticing that the term E (X − g(Y ))2 is always nonnegative, and that the term E[(X − E[X | Y ])(E[X | Y ] − g(Y )) is of the form E[(X − E[X | Y ])ψ(Y ) for ψ(Y ) = E[X | Y ] − g(Y ), and is therefore equal to zero, by Eq. (12). Notice that the preceding proof only relies on the property (12). As we have discussed, we can view this as the defining property of conditional expectations, for general random variables. It follows that the preceding theorem is true for all kinds of random variables. 6 MIXED VERSIONS OF BAYES’ RULE Let X be an unobserved random variable, with known CDF, FX . We observe the value of a related random variable, Y , whose distribution depends on the value of X. This dependence can be captured by a conditional CDF, FY |X . On the basis of the observed value y of Y , would like to make an inference on 16 the unknown value of X. While sometimes, this inference aims at a numerical estimate for X, the most complete answer, which includes everything that can be said about X, is the conditional distribution of X, given Y . This conditional distribution can be obtained by using an appropriate form of Bayes’ rule. When X and Y are both discrete, Bayes’ rule takes the simple form pX|Y (x | y) = pX (x)pY |X (y | x) = pY (y) pX (x)pY |X (y | x) . ′ ′ x′ pX (x )pY |X (y | x ) When X and Y are both continuous, Bayes’ rule takes a similar form, fX|Y (x | y) = fX (x)fY |X (y | x) = fY (y) fX (x)fY |X (y | x) , fX (x′ )fY |X (y | x′ ) dx which follows readily from the definition of the conditional PDF. It remains to consider the case where one random variable is discrete and the other continuous. Suppose that K is a discrete random variable and Z is a continuous random variable. We describe their joint distribution in terms of a function fK,Z (k, z) that satisfies ∫ P(K = k, Z ≤ z) = pK (k) = P(K = k) = FZ (z) = P(Z ≤ z) = X k ∫ fK,Z (k, t) dt. −∞ We then have and3 z ∫ ∞ fK,Z (k, t) dt, −∞ z fK,Z (k, t) dz = −∞ ∫ z X fK,Z (k, t) dz, −∞ k which implies that fZ (z) = X fK,Z (k, z), k is the PDF of Z. Note that if P(K = k) > 0, then P(Z ≤ z | K = k) = ∫ z −∞ fK,Z (k, t) dt, pK (k) 3 The interchange of the summation and the integration can be rigorously justified, because the terms inside are nonnegative. 17 and therefore, it is reasonable to define fZ|K (z | k) = fK,Z (k, z)/pK (k). Finally, for z such that fZ (z) > 0, we define pK|Z (k | z) = fK,Z (k, z)/fZ (z), and interpret it as the conditional probability of the event K = k, given that Z = z. (Note that we are conditioning on a zero probability event; a more accurate interpretation is obtained by conditioning on the event z ≤ Z ≤ z + δ, and let δ → 0.) With these definitions, we have fK,Z (k, z) = pK (k)fZ|K (z | k) = fZ (z)pK|Z (k | z), for every (k, z) for which fK,Z (k, z) > 0. By rearranging, we obtain two more versions of the Bayes’ rule: fZ|K (z | k) = and pK|Z (k | z) = fZ (z)pK|Z (k | z) fZ (z)pK|Z (k | z) =R , pK (k) fZ (z ′ )pK|Z (k | z ′ ) dz ′ pK (k)fZ|K (z | k) pK (k)fZ|k (z | k) =P . ′ ′ fZ (z) k ′ pK (k )fZ|K (z | k ) Note that all four versions of Bayes’ rule take the exact same form; the only difference is that we use PMFs and summations for discrete random variables, as opposed to PDFs and integrals for continuous random variables. 18