Law of Large Numbers & Central Limit Theorem

Advanced Stochastic Processes. LECTURE 20 Functional Strong Law of Large Numbers and Functional Central Limit Theorem Lecture outline • Additional technical results on weak convergence • Functional Strong Law of Large Numbers • Existence of Wiener measure (Brownian motion) 20.1. Additional technical results on weak convergence Given two metric spaces S1 , S2 and a measurable function f : S1 → S2 , suppose S1 is equipped with some probability measure P. This induces a probability measure on S2 which is denoted by Pf −1 and is defined by Pf −1 (A) = P(f −1 (A) for every measurable set A ⊂ S2 . Then for any random variable X : S2 → R, its expectation EPf −1 [X] is equal to EP [X(f )]. (Convince yourself that this is the case by looking at the special case when f is a simple function ). Theorem 20.1 (Mapping Theorem). Suppose Pn ⇒ P for a sequence of probability measures P, Pn on S1 and suppose f : S1 → S2 is continuous. Then Pn f −1 ⇒ Pf −1 on S2 . Proof. We use Portmentau theorem, in particular weak convergence characterization using bounded continuous functions. Thus let g : S2 → R be any bounded continuous function. Since it is continuous, it is also measurable, thus it is also a random variable defined on (S2 , B2 ), where B2 is the Borel σ­field on S2 . We have, EPn f −1 [g] = EPn [g(f )]. Since g is a bounded continuous, then the composition is also bounded continuous. Therefore, by Portmentau theorem EPn [g(f )] → EP [g(f )] = EPf −1 [g] � 1 2 We have defined in Lecture 8 tightness for a sequence of distributions. We now extend this notion to a sequence of measures. Definition 20.2. A sequence of probability measures Pn on metric space S is defined to be tight if for every � > 0 there exists n0 and a compact set K ⊂ S, such that Pn (K) > 1 − � for all n > n0 . Theorem 20.3 (Prohorov’s Theorem). Suppose sequence Pn is tight. Then it contains a weakly convergent subsequence Pn(k) ⇒ P. Some version of the opposite is also true, but we will not need it. Recall that Arzela­Ascoli Theorem provides a characterization of compact sets in C[0, T ]. We can use it now for characterization of tightness. Proposition 1. Suppose a sequence of measures Pn on C[0, T ] satisfies the following conditions: (i) There exists a ≥ 0 such that limn Pn (x(0) ≥ a) = 0. (ii) For each � > 0, limδ→0 lim supn Pn ({x : wx (δ) > �}) = 0. Then the sequence Pn is tight. Proof. Fix � > 0. From (i) we can find a ˆ and n0 large enough so that Pn (x(0) > a) ˆ < � for all n > n0 . For every m ≤ n0 we can also find am large enough so that Pm (x(0) > am ) < �. Take a = max(â, am ). Then Pn (x(0) > a) < � for all n. Let B = {x : x(0) ≤ a}. We just showed Pn (B) ≥ 1 − �. Similarly, for every k > 0 we can find δk such that Pn (wx (δk ) > �) < �/2k . Let Bk = {x : wx (δk ) ≤ �}. Since Pn (Bkc ) < �/2k then Pn (∪k Bkc ) < � and Pn (∩k Bk ) ≥ 1 − �, for all n. Therefore Pn (B ∩ ∩k Bk ) ≥ 1 − 2� for all n. Then set K = B ∩ ∩k Bk is closed (check) and satisfies the � conditions of Arzela­Ascoli Theorem. Therefore it is compact. 20.2. Functional Strong Law of Large Numbers (FSLLN) We are about to establish two very important limit results in the theory of stochastic processes. In probability theory two cornerstone theorems are (Weak or Strong) Law of Large Numbers and Central Limit Theorem. These theorems have direct analogue in the theory of stochastic processes as Functional Strong Law of Large Numbers (FSLLN) and Functional Central Limit Theorem (FCLT) also known as Donsker Theorem. The second theorem contains in it the fact that Wiener Measure exists. We first describe the setup. Consider a sequence of i.i.d. random variables X1 , X2 , . . . , Xn , . . .. We assume that E[X1 ] = 0, E[X12 ] = σ 2 . We can view each realization of an infinite sequence (Xn (ω)) as a sample in a product space R∞ equipped with product type σ­field F and probability by the probability distribution of X1 . measure Pi.i.d. , induced � Define Sn = 1≤k≤n Xk . Fix an interval [0, T ] and for each n consider the following function (20.4) Nn (t) = X�nt�+1 (ω) S�nt� (ω) . + (nt − �nt�) n n This is a piece­wise linear continuous function in C[0, T ]. Theorem 20.5 (Functional Strong Law of Large Numbers (FSLLN)). Given an i.i.d. sequence (Xn ), n ≥ 1 with E[X1 ] = 0, E[|X1 |] < ∞, for every T > 0, the sequence of functions LECTURE 20. WEAK CONVERGENCE 3 Nn : [0, T ] → R converges to zero almost surely. Namely P(�Nn (ω)�T → 0) = P( sup |Nn (t, ω)| → 0) = 1 0≤t≤T As we see, just as SLLN, the FSLLN holds without any assumptions on the variance of X1 , that is even if σ = ∞. Here is another way to state FSLLN. We may consider functions Nn defined on entire [0, ∞) using the same defining identity (20.6). Recall that sets [0, T ] are compact in R. An equivalent way of saying FSLLN is Nn converges to zero almost surely uniformly on compact sets. Proof. Fix � > 0 and T > 0. By SLLN we have that for each realization ω of an sequence X1 (ω), X2 (ω), . . ., there exists n0 (ω) such that for all n > n0 (ω), � S (ω) � � � n � � �< n T We let M (ω) = max1≤m≤n0 (ω) Sm (ω). We claim that for n > M (ω)/�, there holds sup |Nn (t)| < �. 0≤t≤T We consider two cases. Suppose t ∈ [0, T ] is such that nt > n0 (ω). Then Nn (t) = S�nt� (ω) �nt� S�nt� (ω) � = ≤ t ≤ �. n �nt� n T Suppose now t is such that nt ≤ n0 (ω). Then M (ω) < �, n since, by our choice n > M (ω)/�. We conclude sup0≤t≤T |Nn (t)| < � for all n > M (ω)/�. This concludes the proof. � |Nn (t)| ≤ 20.3. Weiner measure FSSLN was a simpler functional limit theorem. Here we consider instead a ”Gaussian” scaling of a random walk Sn and establish existence of the Weiner measure (Brownian motion) as well as FCLT. Thus suppose we have a sequence of i.i.d. random variables X1 , . . . , Xn with mean zero but finite variance σ 2 < ∞. Instead of function (20.6) consider the following function (20.6) Nn (t) = S�nt� (ω) X�nt�+1 (ω) √ √ + (nt − �nt�) . σ n σ n This is again a piece­wise linear continuous function. Then for each n we obtain a mapping ψn : R∞ → C[0, T ]. Of course, for each n, the mapping ψn depends only on the first nT + 1 coordinates of samples in R∞ . Lemma 20.7. Each mapping ψn is measurable. 4 Proof. Here is where it helps to know that Kolmogorov field is identical to Borel field on C[0, T ], that is Theorem 19.4 from Lecture 19. Indeed, now it suffices to show that that ψn−1 (A) is measurable for each set A of the form A = πt−1 (−∞, y], as these sets generate Kolmogorov/Borel σ­field. Each set ψn−1 (πt−1 (−∞, y]) is the set of all realizations Nn (ω) such that � 1≤k≤m Xk (ω) + (nt − m)Xm+1 (ω) ≤ y. Nn (t, ω) = σn where m = �nt�. This defines a measurable subset of Rm+1 ⊂ R∞ .P One way to see this is to xk + (nt − m)xm+1 is observe that the function f : Rm+1 → R defined by f (x1 , . . . , xm ) = 1≤k≤m σn � continuous and therefore is measurable. We conclude that ψn is measurable for each n. Thus each ψn induces a probability measure on C[0, T ], which we denote by Pn . This proba­ bility measure is defined by Pn (A) = Pi.i.d. (ψn−1 (A)) = Pi.i.d. (Nn (ω) ∈ A). We now establish the principal result of this lecture – existence of Weiner measure, namely, the existence of a Brownian motion. Theorem 20.8 (Existence of Wiener measure). A sequence of measures Pn has a weak limit P∗ which satisfies the property of Wiener measure on C[0, T ]. The proof of this fact is quite involved and we give only its scheme, skipping some technical results. First let us outline the main steps in the proof. In the previous lecture we considered projection mappings Pt : C[0, T ] → R. Similarly, for any collection 0 ≤ t1 < · · · < tk we can consider πt1 ,...,tk (x) = (x(t1 ), . . . , x(tk )) ∈ Rk . (a) We first show that the sequence of measures Pn on C[0, T ] is tight. We use this to argue that there exists a subsequence Pn(k) which converges to some measure π ∗ . (b) We show that π ∗ satisfies the properties of Wiener measures. For this purposes we look at the projected measures πt1 ,...,tk (π ∗ ) on Rk and show that these give a joint Gaussian distribution, the kind arising in a Brownian motion (that is the joint distribution of (B(t1 ), B(t2 ), . . . , B(tk ))). At this point the existence of Wiener measure is established. (c) We then show that in fact the weak convergence πn∗ ⇒ π holds. Proof sketch. We begin with the following technical and quite delicate result about random walks. Lemma 20.9. The following identity holds for random walks Sn = X1 + · · · + Xn : √ (20.10) lim lim sup λ2 P(max |Sk | ≥ λσ n) = 0. λ→∞ n→∞ k≤n Note, that this is indeed a very subtle result. We could try to use sub­martingale inequality, since Sk is sub­martingale. It will give √ E[S 2 ] 1 P(max |Sk | ≥ λσ n) ≤ 2 2n = 2 . k≤n λ λσ n So by taking a product with λ2 we do not obtain convergence to zero. The proof of this fact is technical and you can find it in Billingsley [1]. The next result which we also do not prove says that the property (20.10) implies tightness of the sequence of measures Pn on C[0, T ]. LECTURE 20. WEAK CONVERGENCE 5 Lemma 20.11. The following convergence holds for every � > 0. (20.12) lim lim sup P(wNn (δ) ≥ �) = 0. δ →0 n→∞ As a result the sequence of measures Pn is tight. We will not prove the first part of the result, just mention that it can be established using Lemma 20.9 via some technical arguments. Let us now see why this implies tightness. We use characterization given by Proposition 1. First Pn (x(0) ≥ a) = P(Nn (0) ≥ 0) = 0 for every positive a since Nn (0) = 0. Now Pn (wx (δ) > �) = P(wNn (δ) > �). From the first part of the lemma we know that the double convergence (20.12) holds. This means that condition (ii) of Proposition 1 holds as well. We now return to the construction of Wiener measure. Lemma 20.11 implies that the sequence of probability measures Pn on C[0, T ] is tight. Therefore, by Prohorov’s Theorem, it contains a weakly convergent subsequence Pn(k) ⇒ P∗ . Proposition 2. P∗ satisfies the property of the Wiener measure. Proof. Since P∗ is defined on the space of continuous functions, then continuity of every sample is immediate. We need to establish independence of increments and the fact that increments are stationary Gaussian. Thus we fix 0 ≤ t1 < · · · < tk and y1 , . . . , yk ∈ Rk . To preserve the continuity, we still denote elements of C[0, T ] by x, x(t) or x(ω, t) whereas before we used notations ω, B(ω), B(t, ω). Consider the random vector πt1 (Nn ) = Nn (t1 ). This is simply the random variable X�nt1 �+1 (ω) S�nt1 � (ω) √ √ + (nt1 − �nt1 �) σ n σ n The second term in the sum converges to zero in probability. The first term we rewrite as S�nt1 � (ω) �nt1 � √ . σ�nt1 � n and by CLT it converges to a normal N (0, t1 ) distribution. Similarly, consider � X�nt1 �+1 (ω) X�nt2 �+1 (ω) nt1