Modes of Convergence and Convergence Theorems

Advanced Stochastic Processes. LECTURE 8 Modes of convergence and convergence theorems Lecture outline • Modes of convergence. • Convergence theorems. Some remarks The materials of this lecture is a technical toolkit for future lectures. It is a reference type summary of relevant results on different modes of convergence in probability and stochastic processes. You do not need to memorize this, as the content might seem rather chaotic. The goal is to develop enough intuition and be aware of the fact how different modes of convergence are not equivalent to each other. 8.1. Modes of convergence There are many ways in which a sequence of random variables Xn can be said to converge to a random variable X. They are not all equivalent and in this section we introduce some of them and review their properties. Some of these properties are immediate, some we will prove and some require more complicated proof which we skip. We already discussed almost sure convergence: given a probability space (Ω, F , P) and a sequence of random variables X, Xn : Ω → R, we say that Xn → X a.s. if P(ω ∈ Ω : Xn (ω) → X(ω)) = 1. Let us consider some other types of convergences. • Convergence in probability. Given X, Xn : Ω → R we say that Xn → X in proba­ bility if for every � > 0 we have (8.1) lim P(|Xn − X | > �) = 0. n 1 2 Proposition 1. Almost sure convergence implies convergence in probability. The con­ verse is not true in general. Remark. This is why there is a difference in what Weak Law of Large Numbers and Strong Law of Large Numbers are stating. Proof. Suppose Xn → X a.s. Let An = {ω : |Xn (ω) − X(ω)| > �} and Bn = ∪m≥n Am . Bn is a monotone non-increasing sequence of events. Therefore by continuity theorem P(∩n Bn ) = limn P(Bn ). Notice that ∩Bn is exactly the set of events ω such that |Xn (ω)− X(ω)| > � for infinitely many n. Since Xn → X a.s. then P(∩n Bn ) = 0. Therefore limn P(Bn ) = 0. Since An ⊂ Bn , then also limn P(An ) = 0. Counterexample A. Now we build an example of convergence in probability not im­ plying almost sure convergence. Let Ω = [0, 1]. Consider Borel σ-field B on it and uniform probability measure P. Define a double-index sequence Xn,m for all 1 ≤ m ≤ 2n of random variables as follows Xn,m (ω) = 1 for ω ∈ [ m−1 , 2mn ] and = 0 otherwise. We 2n create a single subscript sequence from Xn,m by incrementing m for fixed n till m = 2n and then increment n and start with m = 1. Note that Xn,m → 0 in probability, since the measure of the set {ω : Xn,m (ω) > �} = {ω : Xn,m (ω) = 1} is 1/2n . However, for no ω the convergence Xn,m (ω) → 0 holds since every ω ∈ [0, 1] belongs to infinitely many intervals [ m−1 , 2mn ]. � 2n Convergence in r-th moment. Fix r ≥ 1. We say that Xn → X in r-the moment if limn E[|Xn − X |r ] = 0. When r = 2, we also say that Xn converges to X in L2 norm. Proposition 2. Convergence in r-th moment implies convergence in probability. Proof. We have by Markov’s inequality P(|Xn − X | > �) ≤ E[|Xn − X|r ] . �r Convergence in probability then follows. � The three modes of convergence we discussed corresponded to the case when Xn and X where defined on the same probability space. But sometimes the convergence is considered purely with respect to the distribution of Xn and X and/or their expectations. • Convergence in distribution. Consider a sequence of random variables Xn defined on probability spaces (Ωn , Fn , Pn ) (which might be all different) and a random variable X, defined on (Ω, F , P). Let Fn (t) and F (t) be the corresponding distribution functions. Xn is said to converge to X in distribution (written Xn ⇒ X or Fn ⇒ F ) if for every point t at which F is continuous lim Fn (t) = F (t). n Remark. Why do we require t be the continuity point of F ? Consider the following example. Xn = 1/n with probability one, Xn = 0 with probability 1 − 1/n, and let X = 0 with probability one. Then Fn (t) = 0, t < n1 and Fn (t) = 1, t ≥ n1 ; and F (t) = 0, t < 0, F (t) = 1, t ≥ 0. It makes sense to say that Xn ⇒ X as they ”almost” have the same distribution. Yet F (0) = 1 �= limn Fn (0) = 0. This is why we exclude the points where continuity does not hold. Later on when we study weak convergence of probability measures we will elaborate on this. LECTURE 8. MODES OF CONVERGENCE 3 Proposition 3. Convergence in probability implies convergence in distribution. We skip the proof. Convergence of expectations. Under the same setting, we define Xn to converge to X in expectation if limn E[|Xn |] = E[|X |]. We also say that convergence up to r moments holds if limn E[|Xnr |] = E[|X r |] Clearly convergence in expectation does not imply convergence in distribution. Think about any sequence of variables which have the same expected value but different distributions. Does the converse hold? In fact we could ask for a weaker question. Does almost sure convergence imply convergence in expectation? Convergence a.s. implies convergence in probability which implies convergence in distribution. So having a no answer would imply no for the previous question as well. Here is an example when a.s. convergence does not imply convergence in expectation. Let Ω = [0, 1] equipped with Borel σ-field. Consider uniform probability measure P on [0, 1]. Consider a uniform probability measure on [0, 1]. Let X(ω) = 0 for all ω ∈ [0, 1], and let Xn (ω) = n for ω ∈ [0, 1/n] and Xn (ω) = 0 otherwise. For all non=zero ω we have Xn (ω) → 0. So Xn → 0 a.s. But E[Xn ] = 1 and E[X] = 0. As we mentioned it does not make sense to expect that convergence in distribution implies almost sure convergence or convergence in probability, since in the first case the underlying random variables may be defined on different probability spaces. But here is an interesting result. Theorem 8.2 (Skorokhod’s Representation Theorem). Suppose Xn → X in distribution. There exists a probability space (Ω� , F � , P� ) and a sequence of random variables Y, Yn : Ω� → R d d such that Xn = Yn , X = Y , and Yn → Y a.s. In other words, there is a representation of Xn , X on a single probability space, where the convergence occurs almost surely. Why does not this contradict our counterexample showing the convergence in probability does not imply convergence in distribution? Problem 1. Consider the Counterexample A showing that conv in probability does not imply conv almost surely. Construct a representation Yn,m , Y of Xn,m , X on some common probability space such that Yn,m → Y almost surely. HINT: start by computing probability distribution functions Fn,m of Xn,m . 8.2. Tightness Tightness is an important concept, which is often used for establishing some limiting properties of a series of probability distributions, or, in general, probability measures. To introduce the intuition behind the concept, consider a sequence of zero mean normal random variables N (0, σn ). If σn are uniformly bounded supn σn < ∞, then, by a theorem from real analysis, there exists a subsequence n(k) along which there is a convergence to some limit: σn(k) → σ ∗ . In some sense N (0, σ ∗ ) is a limit of distributions N (0, σn(k) ). Similarly, consider a sequence of uniform distributions on intervals [an , bn ] such that −∞ < inf an ≤ supn bn < ∞. Then again we can find a subsequence n(k) such that an(k) → a∗ and bn(k) → b∗ . Then in some sense the uniform 4 distributions on [an(k) , bn(k) ] converge to the uniform distribution [a∗ , b∗ ]. However, in the first example if sup σn = ∞ we see that there is no meaningful limit of normal distributions N (0, σn ) along any subsequence. Similarly if either an → −∞ or bn → ∞, then there is no meaningful limit of uniform distributions on [an , bn ] along any subsequence. It turns out that the important conditions for having a limit point is not so much the shaper of the distribution, but simply the fact that most of ”mass” is concentrated in some bounded interval. Definition 8.3. A sequence of distribution functions Fn is defined to be tight if for every � > 0 there exists a sufficiently large K > 0 such that for all n, Fn (K) − Fn (−K) > 1 − �. In other words, if Xn is sequence of random variables with distribution function Fn , then P(−K ≤ Xn ≤ K) > 1 − �. Problem 2. Prove that a sequence N (µn , σn ) is tight iff supn |µn | < ∞ and sup σn < ∞. The following result is the main reason for introducing the concept of tightness. This theorem is one of the possible paths for proving the existence of Wiener measure. Theorem 8.4. Suppose Fn is a tight sequence of distribution functions. Then there exists a distribution function F and a subsequence n(k) such that along this subsequence the convergence in distribution holds: Fn(k) ⇒ F . That, there exists a distribution function F such that for every continuity point t of F , we have limk Fn(k) (t) = F (t). A converse of this theorem also holds, but we will not prove it. Proof. Consider an enumeration q1 , q2 , . . . , qr , . . . of all the rational points in R. Since the sequence of values of Fn (q1 ) is in [0, 1], then there exists a subsequence n1 = n1 (k) along which there is a convergence of Fn1 (k) (q1 ) to some value G(q1 ): limk Fn1 (k) (q1 ) = G(q1 ). Similarly, since Fn1 (k) (q2 ) ∈ [0, 1], then there exists a subsequence n2 ⊂ n1 along which F n2 (k)(q2 ) converges to some value which we denote by G(q2 ). We continue this subsequently for all the rational values qr obtaining a nested sequence of subsequences n1 ⊃ n2 ⊃ n3 ⊃ · · · . No consider the diagonal subsequence n∗ (k) = nk (k). Along this sequence we have convergence for all rational values simultaneously: limk Fn∗ (k) (qr ) = G(qr ). Now, consider the following function F : R → R: F (t) = inf G(qr ). r:qr >x We claim that F is indeed a distribution function and Fn∗ (k) ⇒ F . First, we have that F (t) ∈ [0, 1] since each G(qr ) is a limit of values of Fn∗ (k) . By construction F is non-decreasing: for every pair t1 < t2 we have {r : qr > t2 } ⊂ {r : qr > t2 }. Now we show that F is right-continuous in every point t. Fix � > 0. By definition, we can find a rational value qr > t such that G(qr ) < F (t) + �. Select any δ < qr − t. Then if t < t� < t + δ, then t� < qr , implying F (t� ) = inf{G(q) : q > t� } ≤ G(qr ) < F (t) + �. We showed that any point t� ∈ [t, t + δ) satisfies F (t� ) ≤ F (t) + � and right-continuity is established. We showed that F is a right-continuous non-decreasing function taking values in [0, 1]. It remains to show that limt→−∞ F (t) = 0, limt→∞ F (t) = 1. But first we show that in every continuity point t of F we have limk Fn∗ (k) (t) = F (t), and then we return to proving the limits. For the ease of presentation, let us assume that we have limn F (qr ) = G(qr ) for every rational value qr . (That is we substitute subsequence n∗ for the original sequence n). Thus let t be the continuity point of F . Fix � > 0. Pick rationals r1 , r2 , s with r1 < r2 < t < s, so that F (t) − � < F (r1 ) ≤ F (r2 ) ≤ F (t) ≤ F (s) ≤ F (t) + �. LECTURE 8. MODES OF CONVERGENCE 5 This is possible by continuity and monotonicity of F . Now Fn (r2 ) → G(r2 ) ≥ F (r1 ), and Fn (s) → G(s) ≤ F (s). Therefore, for all sufficiently large n we have F (t) − � < Fn (r2 ) ≤ Fn (t) ≤ Fn (s) ≤ F (t) + �. We conclude that limn Fn (t) = F (t). It remains to show that limt→−∞ F (t) = 0, limt→∞ F (t) = 1, and this is exactly where we use tightness. Fix � > 0. There exists K such that Fn (K) ≥ Fn (K) − Fn (−K) ≥ 1 − �. Without loss of generality take K to be the continuity point of F . (The set of points of discontinuity of F has Lebeasgue measure zero, per real analysis, so we can simply find K � > K which is a continuity point). By monotonicity of Fn , we will have Fn (K � ) − Fn (−K � ) ≥ 1 − �). Then Fn (K) → F (K) and Fn (−K) → F (−K). Therefore 1 ≥ F (K) ≥ F (K) − F (−K) ≥ 1 − �. This implies that limt→∞ F (t) = 1. From the same bound we have 1 − F (−K) ≥ F (K) − F (−K) ≥ 1 − � or F (−K) ≤ �, implying limt→−∞ F (t) = 0. This proves that F is indeed a distribution function. � 8.3. Convergence theorems We now state without proof several useful convergence theorems. Theorem 8.5 (Dominated Convergence Theorem). Suppose Xn → X a.s. and suppose there exists a random variable Y ≥ |Xn | a.s. such that E[Y ] < ∞. Then E[Xn ] → E[X] as n → ∞. In particular, suppose Xn are a.s. bounded. That is |Xn | ≤ B a.s., for some B > 0. Then E[Xn ] → E[X]. Note that it is not enough to assume that just supn E[|Xn |] < ∞: consider the example where almost sure convergence does not imply convergence in expectation. There the expected value of each variable was E[Xn ] = 1, but E[X] = 0. Theorem 8.6 (Monotone Convergence Theorem). Suppose Xn → X a.s. and suppose for each sample ω, Xn (ω), n ∈ N is non-decreasing sequence. Then E[Xn ] → E[X] as n → ∞. Theorem 8.7 (Continuous Mapping Theorem). Suppose Xn → X in distribution. Let g : R → R be continuous function. Then g(Xn ) → g(X) in distribution. If, in addition g is bounded, then E[g(Xn )] → E[g(X)]. Proof. We use Skorokhod Representation Theorem. Find a representation Yn , Y of Xn , X on some probability space where Yn → Y a.s. For every sample ω, such that Yn (ω) → Y (ω), we have by continuity, g(Yn (ω)) → g(Y (ω)). Since the set of such samples ω has measure 1, then g(Yn ) → g(Y ) a.s. This implies that convergence also holds in probability and therefore also in distribution. Therefore g(Yn ) ⇒ g(Y ). Suppose in addition g is bounded. Then by DCT, we have E[g(Yn )] → E[g(Y )]. This implies E[g(Xn )] → E[g(X)]. � 8.4. Additional reading materials • Grimmett and Stirzaker [2] Section 7.2. • Durrett [1] Sections 1.3, 2.2.