Recitation 7: Empirical Process Theory, Random Walk Asymptotics

Empirical Process Theory 1 14.384 Time Series Analysis, Fall 2007 Recitation by Paul Schrimpf Supplementary to lectures given by Anna Mikusheva October 17, 2008 Recitation 7 Empirical Process Theory Let xt be a real-valued random k × 1 vector. Consider some �n valued function gt (xt , τ ) for τ ∈ Θ, where Θ is a subset of some metric space. Remark 1. In time series applications, generally, Θ = [0, 1] Let T 1 � ξT (τ ) = √ (gt (xt , τ ) − Egt (xt , τ )) T t=1 ξT (τ ) is a random function; it maps each τ ∈ Θ to an �n valued random variable. ξT (τ ) is called an empirical process. Under very general conditions, standard arguments show that ξT (τ ) converges pointwise, i.e. ∀τ0 ∈ Θ, ξT (τ0 ) ⇒ N (0, σ 2 (τ0 )). Also, standard arguments imply that on a finite collection of points, (τ1 , ..., τp ), ⎡ ⎤ ξT (τ1 ) ⎢ ⎥ .. (1) ⎣ ⎦ ⇒ N (0, Σ(τ1 , ..., τp )) . ξT (τp ) We would like to generalize this sort of result so that we talk about the convergence of ξT (). Example 2. Suppose you want to test whether xt has cdf F (x). The cdf of xt can be estimated by its empirical cdf, 1� F̂T (x) = 1(xt ≤ x) T Two possible statistics for testing whether F̂n (x) equals F (x) are the Kolmogorov-Smrinov statistic, √ sup n(F̂n (x) − F (x)) x and the Cramer-von Mises statistic � n (F̂n (x) − F (x))2 dF (x) This fits into the setup above with � 1 �� 1(xt ≤ τ ) − F (τ ) ξT (τ ) = √ T . For independent xt , finite dimensional covergence is easy to verify and for any τ1 , τ2 we have � � � � �� F (τ1 )(1 − F (τ1 )) F (τ1 ) ∧ F (τ2 ) − F (τ1 )F (τ2 ) ξT (τ1 ) ⇒ N 0, F (τ1 ) ∧ F (τ2 ) − F (τ1 )F (τ2 ) F (τ2 )(1 − F (τ2 )) ξT (τ2 ) Definition 3. We define a metric for functions on Θ as d(b1 , b2 ) = supτ ∈Θ |b1 (τ ) − b2 (τ )| Sufficient Conditions for Stochastic Equicontinuity 2 Definition 4. B = bounded functions on Θ Definition 5. U(B) = class of uniformly continuous (wrt d()) bounded functionals from B to � Example 6. Examples of elements of U(B) include: • Evaluation at a point: fτ0 (ξ) = ξ(τ0 ) � • Integration: f (ξ) = Θ ξ(τ )dτ Definition 7. convergence in B: ξT ⇒ ξ iff ∀f ∈ U(B) we have Ef (ξT ) → Ef (ξ) Remark 8. This definition of convergence implies pointwise convergence. If ξT ⇒ ξ, then by definition for each τ0 and k, EξT (τ0 )k → Eξ(τ0 )k . Then, if the distribution of ξ(τ0 ) is completely determined by its moments (as it is if, for example, ξ(τ0 ) is normal or has bounded support), it follows that ξT (τ0 ) ⇒ ξ(τ0 ). Definition 9. ξ is stochastically equicontinuous if ∀� > 0, ∀η > 0, there exists δ > 0 s.t. lim P ( T →∞ |ξT (τ1 ) − ξT (τ2 )| > η) < � sup |τ1 −τ2 |< δ Theorem 10. Functional Central Limit Theorem: If 1. Θ is bounded 2. there exists a finite-dimensional distribution convergence of ξT to ξ (as in (1)) 3. {ξT } are stochastically equicontinuous then ξT ⇒ ξ Remark 11. Condition 1 can be removed. Without it, condition 3 must be strengthened to: ∀ �, η > 0 there exists a partition of Θ into finitely many sets, Θ1 , ...Θk such that lim sup P (max T →∞ i sup |ξT (τ1 ) − ξT (τ2 )| > η) < � τ1 ,τ2 ∈Θi Proving the theorem involves constructing a metric on Θ such that Θ is bounded with respect to that metric, so condition 1 is really a consequence of this stronger version of condition 3. Remark 12. Condition 2 can be checked. Condition 3 is difficult to check, but lots of work has been done to derive simpler sufficient conditions. See Andrews (1994 HoE) for some sufficient conditions. Necessary and sufficient conditions for stochastic equicontinuity are not known. However, very general sufficient conditions are known. Classes of functions for which the functional CLT holds are called P-Donsker. Sufficient Conditions for Stochastic Equicontinuity This is largely tangential to what we will do in class. Definition 13. A class of functions, G, is P-Donsker if for every g ∈ G, � 1 �� √ g(xt , ·) − E[g(xt , ·)] ⇒ ξ T where ξ ∈ �∞ (G) In order for a class of functions to be P-Donsker, stochastic equicontinuity requires that the function class not be too complex. One way of measuring the complexity of a function class is by bracketing numbers. An � bracket in L2 , [l, u] is the set of all functions, f , such that l ≤ f ≤ u pointwise with E[|l − u|2 ]1/2 < �. The � bracketing number written as N[] (�, G) is the minimal number of � brackets needed to cover G. An important sufficient condition for a class to be P-Donsker is the following: Continuous Mapping Theorem 3 Theorem 14. Every class G of measurable functions with � 1� log N[] (�, G )d� < ∞ 0 is P-Donsker. 1 Although this condition looks strange and difficult, it can be verified in a number of interesting situations. Example 15. Classes that are P-Donsker include • Distribution functions: using brackets of the form [1(x < xi ), 1(x < xi+1 )] with F (xi+1 ) − F (xi ) < � we can cover G with C/�2 brackets, so 1 � 1 � � log N[] (�, G)d� ≤ 0 � log(c/�2 )d� = log(c) + 1 0 is finite. • Parametric Classes: if G = {gθ : θ ∈ Θ ⊂ Rk } with Θ bounded, and a Lipschitz condition holds: |gθ1 (x) − gθ2 (x)| ≤ m(x)||θ1 − θ2 || with E[m(x)2 ] < ∞. • Smooth functions from Rd → R with uniformly bounded derivates of order up to α > d/2 Another way of characterizing complexity is through uniform covering numbers and uniform entropy integrals, but I am not going to say anything about it here. Continuous Mapping Theorem The following theorem is important for making the functional central limit theorem useful. Theorem 16. Continuous Mapping Theorem: if ξT ⇒ ξ, then ∀ continuous functionals, f , f (ξT ) ⇒ f (ξ) Example 17. We can use the continuous mapping theorem � to get the distribution of the Kolmogorov-Smirnov and Cramer-von Mises statistics. Both: supτ ξ(τ ) and ξ(τ )2 dF (τ ) are continuous functionals, so √ d sup n(F̂n (x) − F (x)) → sup ξ(x) x and � n x d (F̂n (x) − F (x))2 dF (x) → � ξ(x)2 dF (x) where � ξ(x) is a Brownian bridge, i.e. Gaussian with covariance function as above. We can simulate supx ξ(x) and ξ(x)2 dF (x) to find critical values for hypothesis tests. 1 This theorem definitely holds for iid data. It might need to be modified for dependent data (e.g. the form of the integral depends on mixing coefficients), but I’m not certain. Random Walk Asymptotics 4 Random Walk Asymptotics In lecture 12, we saw that if yt is a random walk and we estimate an AR(1), then T (ρ̂ − 1) ⇒ �1 1 2 2 (W (1) − 1) �1 W (s)2 ds 0 W (s)dW (s) = 0� 1 W 2 (s)ds 0 It is important to understand how we derived these expressions because, unlike in the stationary case, small changes to the estimated model can greatly alter the asymptotic distribution. For example, suppose we estimate an AR(1) with a constant, so we estimate, yt = α + ρyt−1 + ut Let β = [α ρ]� = [0 1]� and β̂ be the OLS estimate. We know that: � α̂ − α ρ̂ − ρ � � = �T yt−1 �−1 � � � � � ut � yt2−1 yt−1 yt−1 ut (2) To find the asymptotic distribution, we need to examine each of the sums, determine appropriate scaling factors, and write down what they converge to. We’ve already seen each of these sums in lecture 12, so I won’t rewrite the steps here, but recall that 1 T 3/2 T −2 T � yt ⇒σ t=1 W (t)dt 0 t=1 T � 1 � yt2 ⇒σ 2 1 � W 2 (s)ds 0 1 � √ ut ⇒σW (1) T 1� σ yt−1 ut ⇒ (W 2 (1) − σ 2 ) T 2 � 1/2 � 0 T These results suggest scalling β̂ by to arrive at a nondegenerate asymptotic distribution, i.e. 0 T � � � 1/2 ��−1 � 1/2 � T yt−1 0 T T � � = yt2−1 yt−1 0 T 0 � � � � � � −1 1� T −3/2� yt−1 ut T −1/2 � = 2 −3/2 −2 T −1 yt−1 ut T yt−1 T yt−1 � � � �� �−1 � � −1/2 � W (1) σ 0 W (s)ds 1 (α̂ − α) T � � ⇒ 1 2 0 1 W (s)ds W (s)2 ds T −1 (ρ̂ − ρ) 2 (W (1) − 1) T 1/2 0 0 T �� α̂ − α ρ̂ − ρ � �� T 1/2 0 0 T �� 0 T �� � � � ut yt−1 ut √ From which, we see that neither α̂ nor ρ̂ are asymptotically normal. Also, α̂ converges at the usual 1/ T rate, but ρ̂ converges at rate 1/T . with Drift Now, let’s consider another modification of the model. Suppose that yt is a random walk with drift, yt = yt−1 + α + et . As above, let’s assume we estimate by OLS an AR(1) with a constant. As above we need to analyze each of the sums in the matrices in (2). We cannot just use the results from lecture 12 because now the process for yt is different. with Drift • � 5 yt−1 : � yt−1 = � � (α(t − 1) + y0 + es ) t =( s< t � α(t − 1)) + T y0 + � t ξT ((t − 1)/T ) t � For t αt to have a finite limit, we must normalize it by T −2 . We know that T −2 T y0 → 0, and � � � −1/2 ξT ((t−1)/T ) � √ T ⇒ W (s)ds). ThereT −2 ξT ((t − 1)/T ) → 0 (since T −3/2 ξT ((t − 1)/T ) = T1 T � � fore, T −2 yt−1 → lim T −2 α(t − 1) = α/2. � 2 � 2 yt−1 → α2 /3 • yt−1 : identical reasoning shows that we must normalize by T −3 and � • et : is unchanged, → σW (1) � • yt−1 et : � � � ((t − 1)α + y0 + es )et yt−1 et = s< t = � et (t − 1)α + � � 1 et y0 + (yT2 − e2t ) 2 � � The first term, et (t − 1)α is Op (T 3/2 ), so we must normalize by at least T −3/2 . et y0 is Op (T 1/2 ) � 2 2 and yT − et is Op (T ), so they vanish. This leaves, � � T −3/2 yt−1 et ⇒T −3/2 et (t − 1)α ⇒ N (0, α2 /3) Furthermore, jointly we have: � � � � T −1/2 e 1 t 2 � ⇒ N (0, σ α/2 T −3/2 yt−1 et α/2 α2 /3 � ) Combining these results, we see that � T 1/2 (α̂ − α) T 3/2 (ρ̂ − ρ) � ⇒ N (0, σ 2 � 1 α/2 α/2 α2 /3 �−1 ) Thus, we obtain asymptotic normality when we estimate a random walk with drift. Also, the asymptotic variance matrix is the same as standard OLS. However, ρ̂ converges at a faster rate than usual.