Uniform Integrability, Convergence of Series

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Lecture 19 Fall 2018 Uniform integrability, convergence of series Contents 1. L1 convergence (aka convergence in mean), L1 LLN. 2. Uniform integrability 3. Convergence of series of independent summands 1 CONVERGENCE IN L1 Definition 1 (Convergence in mean). A sequence of integrable random varibles Xj is said to converge in L1 to X (also known as “convergence in mean”), L denoted Xj →1 X if E[|Xj − X|] → 0 j → ∞. Lp For p > 1 we define Xj → X if E[|Xj |p ] < ∞ and E[|Xj − X|p ] → 0. Some simple properties are given below: Proposition 1. L L (i) Xn →1 X implies E[|X|] < ∞. i.p. (ii) Xj →1 X implies Xj → X a.s. L (iii) Xj →1 X does not imply and is not implied by Xj → X. (iv) The space of integrable random variables on (Ω, F, P) modulo almostsure equivalence is a Banach space, denoted as L1 (Ω, F, P) with norm kXk1 , E[|X|]. Similarly for Lp ( Ω, F, P). 1 Proof: (i) Follows from taking expectation in the triangle inequality: |X| ≤ |Xn − X| + |Xn | (1) (ii) and (iii) is an exercise. (iv) is outside the scope of this class. Our goal is to show the following the following (third!) variation of the LLN: Proposition 2 (L1 -LLN). Let Xj be iid random variables with finite expectation, then n 1X L Xj →1 E[X] n j=1 The proof of this proposition follows by Theorem 1 and Corollary 1 below. 2 UNIFORM INTEGRABILITY Definition 2. A collection of random variables Xα , α ∈ S is uniformly integrable if k(b) , sup E[|Xα |1{|Xα | > b}] → 0 b → ∞ . (2) α Some useful criteria for checking u.i.: Proposition 3. The following hold: (i) If E[|X|] < ∞ then {X} is u.i. d (ii) When Xα = Yα then {X α}–u.i. iff {Yα }–u.i. (iii) {X α}-u.i. iff {Xα} is L1 -bounded and uniformly continuous: sup E[|Xα |] < ∞ (3) α sup E[|Xα |1E ] → 0 α (iv) If G : R+ → R+ is such that G(t) t (4) → ∞ as t grows without bound1 then sup E[G(|Xα |)] < ∞ 1 as P[E] → 0 ⇒ Some typical choices are G(t) = t2 , |t|1+ǫ and |t log t|. 2 {Xα }–u.i. a.s. (v) If Xn → X and E[|Xn |] → E[|X|] then {Xn } is u.i. Proof: (i) follows from the MCT, (ii) is obvious from the definition, (v) is part of the homework. For (iii), first notice that E[|X α|] ≤ k(b) + b for every b > 0 and thus (3) holds. Similarly, notice that for every b: E[|Xα |1E ] ≤ E[|Xα |1{|Xα | > b}] + bP[E] ≤ k(b) + bP[E] and thus by taking P[E] → 0 and b → ∞ we prove (4). Conversely, if (3) and (4) hold, but {Xα } is not uniformly integrable then for some sequence αk and ǫ0 > 0 we have E[|Xα k |1{|Xα k | > k}] ≥ ǫ0 > 0 (5) On the other hand, by (3) and Markov inequality P[|X αk | > k] → 0. Consequently, (5) contradicts (4). Finally, to see (iv) just notice that for every a > 0 there exists b > 0 such that G(t) t ≥ a for all t > b. Then, G(|Xα |) ≥ a|Xα |1{|Xα | > b} and taking expectation here we obtain: k(b) ≤ 1 sup E[G(|X α|)] a α from which (2) follows by taking a → ∞. As a simple consequence of the Proposition we get: necessarily independent!) Corollary 1. Let Xj be identically distributed (not n o Pn and integrable. Then collection of normalized sums n1 j=1 Xj , n = 1, . . . is uniformly integrable. Proof: Indeed, by Proposition 3(i) and (ii) Pnwe get that {Xj , j = 1, . . .} is uniformly integrable. Now defining Yn = n1 j=1 Xj we have sup E[|Yn |] ≤ E[|X|] < ∞ n and on the other hand sup E[|Yn |1E ] ≤ sup E[|Xn |1E ] → 0 as P[E] → 0 , n n 3 where the last step follows by (4) applied to {Xj }. Uniform integrability of {Yj } then follows from Proposition 3 (iii). The main value of studying uniform integrability is the following: Theorem 1. We have L Xn →1 X ⇐⇒ i.p. Xn → X and {Xn }– u.i. Proof: The ⇒ direction follows from Markov’s inequality and Proposition 3(iii). Indeed, by (1) we have the inequality E[|Xn |1E ] ≤ E[|X|1E ] + E[|Xn − X|1E ] (6) For very large n ≥ n0 the second term is smaller than ǫ and hence     lim sup sup E[|Xn |1E ] ≤ ǫ+lim sup E[|X|1E ] + max E[|Xn − X|1E ] P[E]→0 n 1≤n≤n0 P[E]→0 where the second term is zero by (4) because {|X|, |X1 − X|, . . . |Xn0 − X|} is a uniformly integrable collection. Consequently, taking ǫ → 0 we have shown sup E[Xn 1E ] → 0 P[E] → 0 n Setting E = in (6) we verify (4). Thus Proposition 3(iii) implies that the infinite collection {Xn } is also u.i. For the converse direction, we first notice that by characterization of convera.s. gence in probability there must exist a subsequence Xnk → X. Then by Fatou’s lemma and (3) we have E[|X|] = E[lim inf |Xnk |] ≤ lim inf E[|Xnk ] < ∞ k k Thus, the limit random variable is integrable and consequently (Exercise!) coli.p. lection of nonnegative random variables {Yn , n = 1, . . .} is u.i. and Yn → 0 , where Yn , |Xn − X| . Then, we have for every ǫ > 0 E[Yn ] = E[Yn 1{Yn > ǫ}] + E[Yn 1{Yn ≤ ǫ}] ≤ ǫ + E[Yn 1{Yn > ǫ}] . 4 (7) (8) i.p. Since Yn → 0 we have P[Yn > ǫ] → 0. Then by (4) the second term converges to zero as n → ∞. Hence, for all ǫ > 0 lim sup E[Yn ] ≤ ǫ , n→∞ L which shows Yn →1 0. As a corollary we obtain a result we assumed before (in proving that convergence of characteristic functions implies convergence in distribution). Corollary 2. Let fn (x) → f (x) ∀x ∈ R be a pointwise convergent sequence d of pdfs. If Xn ∼ fn and X ∼ f then Xn → X. P −n−1 f (x). It is clear that φ(x) is another Proof. Let φ(x) = 21 f (x) + ∞ n n=1 2 pdf. Let (  , F, P) be defined as  = R, F = B, P(dx) = φ(x)dx and define random variables Yn (x) = fn (x)/φ(x). Note that as a consequence of our definition of φ this ratio is well-defined almost everywhere: φ(x) = 0 implies fn (x) = 0 and P({x : φ(x) = 0}) = 0. Similarly, define Y (x) = f (x)/φ(x). a.s. We have Yn → Y . Furthermore, by construction E[|Yn |] = 1 = E[|Y |]. Thus, by Prop. 3(v) the collection {Yn } is u.i.. Consequently, from the previous Theorem we have E[|Yn − Y |] → 0. Rewriting this last statement explicitly, we have shown Z |fn (x) − f (x)|dx → 0 n → ∞. (9) R In particular, for any E ∈ B we have Z Z f (x) = PX [E] , fn (x) → PXn [E] = E E and taking E = (−∞, a] shows convergence of CDFs of Xn to the CDF of X at every point a. (In fact, (9) is usually stated as “distribution of Xn converges to the distribution of X in total-variation”. This is a stronger mode of convergence than convergence in distribution.) 3 SUMS OF INDEPENDENT RANDOM VARIABLES A classical topic tightly related to the SLLN is convergence of sums Sn = n X j=1 5 Xj when Xj are independent (and not identically distributed, of course). There is a great deal results about properties of Sn , and here we will only mention a core principle: Convergence behavior of Sn , its centralPmoments, concentration properties, etc are largely encoded in the behavior of nj=1 var[Xj ]. We start with an example. Consider two independent sequences: 1 , 2n 1 1 P[Yn = ± ] = . n 2 P[Xn = ±1] = P[Xn = 0] = 1 − 1 , n (10) (11) 1 First, we notice that E[|Xn |] = E[|Yn |] = 2n , so that P sum of first P moments diverges at the same speed. Furthermore, both series Xn and Yn do not absolutely converge: " # " # X X |Yn | = +∞ = 1 . P |Xn | = +∞ = P n n Indeed, for Yn this is obvious as |Yn | = n1 , while for Xn it follows from BorelCantelli that: P[|Xn | =P 1–i.o] = 1.P So far, we see that Xn and Yn behave quite similarly. However, as we will see next it turns out that " # X X P Xn converges = 0 var[Xn ] = +∞ (12) n P " X n n # Yn converges = 1 X var[Yn ] < +∞ (13) n The explanation of this phenomena is the following: While both series diverge absolutely, the rapidly decreasing variances ofPterms in Yn allows for “sign cancellation” effect to kick in making the series Yn converge (similar to converP (−1)n gence of n n ). 6 Theorem 2 (Kolmogorov, Khintchine). Let Xj be independent and µ, ∞ X E[Xj ] , j=1 σ2 , ∞ X j=1 |µ| < ∞ , var[Xj ] < ∞ then Sn = n X (14) (15) Xj j=1 converges almost surely and in L2 to a limit S with E[S] = µ, var[S] = σ 2 . a.s. Conversely, if |Xj | ≤ c for some constant c and Sn → S with realvalued S, then conditions (14)-(15) hold. Proof: We prove the direct part. Without loss of generality we assume E[Xj ] = 0. As we have shown in the homework (Cauchy criterion of almost sure convergence) it is sufficient to show that P[sup |Sn+k − Sn | > ǫ] → 0 k≥1 n→∞ (16) Kolmogorov’s inequality (see Theorem following the proof) shows that E[sup |Sn+k − Sn |2 ] ≤ 2 k≥1 ∞ X var[Xj ] . (17) n By Chebyshev’s inequality we obtain then P[sup |Sn+k − Sn | > ǫ] ≤ k≥1 ∞ 2 X var[Xj ] ǫ2 n (18) Since sum of variances converges, the left-hand side of (18) decreases to 0 as a.s. n → ∞ and thus (16) is shown. The proof of Sn → S is complete. Notice that (17) with n = 0 shows that “life-time maximum” M∞ , sup |Sn | n≥1 has finite second moment. Since |Sn − S| ≤ 2M∞ , 7 by the DCT it follows that E[|Sn − S|2 ] → 0 and similarly for E[Sn ] → E[S], E[Sn2 ] → E[S]. We proceed to proving the converse. First, assume E[Xj ] = 0 and suppose a.s. Sn → S but n X var[Xj ] ր ∞ Dn = j=1 Then, notice that E[|Xj |3 ] ≤ E[|Xj |2 ]c and thus n X j=1 E[|Xj |3 ] ≤ Dn c Consequently, by the CLT for non-identically distributed random variables we have 1 d √ Sn → Z ∼ N (0, 1) . Dn On the other hand, we have for every t and s > 0 and for all n large enough p (19) P[Sn > t] ≥ P[Sn > s Dn ] . d Since Sn → S we also have for all t such that P[S = t] = 0 that P[Sn > t] → P[S > t] . However, upon taking the limit in (19) as n → ∞ we get for all t and s > 0: P[S > t] ≥ P[Z > s] Taking s → 0 we get 1 ∀t ∈ R 2 which is a contradiction, as no distribution of S can satisfy such inequality. Next, if E[Xj ] = µj , then let Yj = Xj − Xj′ , where Xj′ is an independent copy of Xj . In this way E[Yj ] = 0, var[Yj ] = 2var[Xj ] and P[S > t] ≥ n X j=1 a.s. Yj = Sn − Sn′ → S − S ′ . 8 Hence, by the previous argument we have n X j=1 var[Xj ] < ∞ and by the direct part of the theorem n X j=1 (Xj − µj ) = Sn − n X µj j=1 converges almost surely. Since Sn converges by assumption, so must do a.s. P µj . Remark: Conditions (14) are necessary for convergence Sn → S only under assumption of the boundedness of Xj (see homework). We also mention that instead of relying on the CLT in the proof of the converse direction, we may have followed a more conventional route, based on the inequality P[ max |Sk | > a] ≥ 1 − 1≤k≤n (a + c)2 . var[Sn ] The proof of this inequality, however, would appear rather unnatural without mentioning stopping times. Either method, however, really just shows that condition p |Xj | ≤ c guarantees the width of the distribution of Sn has the same order as var[Sn ]. (For unbounded Xj , rare large jumps may significantly increase the variance, while having very little effect on the bulk of the distribution of Sn ). Theorem 3 (Kolmogorov). Let Xj be independent, zero-mean with finite second moments and let Mn = sup k X 1≤k≤n j=1 Xj , 1 ≤ n ≤ ∞ . Then we have for any 1 ≤ n ≤ ∞ n X   2 E[Xj2 ] . E |Mn | ≤ 2 j=1 Let Proof: The case of n = ∞ follows from the case of finite n by the MCT. Sk = X1 + · · · + Xk , An = max Sk , 1≤k≤n 9 (20) (21) Note that for n = 1 we clearly have E[A2n ] ≤ E[Sn2 ] . (22) Assume (by induction) that (22) is shown for all sums of upto n − 1 random variables. Then, notice that An = X1 + max(0, X2 , X2 + X3 , . . . , n X Xj ) . 2 Since first and second terms are independent and E[X1 ] = 0 we get E[A2n ] = E[X12 ] + E[max(0, X2 , X2 + X3 , . . . , n X Xj )2 ] (23) 2 ≤ E[X12 ] + E[max(X2 , X2 + X3 , . . . , ≤ n X n X Xj )2 ] (24) 2 E[Xj2 ] , (25) j=1 where the first inequality follows from (x+ )2 ≤ x2 and the second one is by the inductive assumption. Thus (22) holds for all n. By symmetry, we also must have E[Bn2 ] ≤ E[Sn2 ] , Bn = max −Sk . 1≤k≤n Finally, since Mn2 = max(A2n , Bn2 ) and using max(a, b) ≤ a + b we complete the proof. 10