14.382 Spring 2017 Recitation Notes 4: Asymptotics, Take 1

R4: Asymptotics, Take 1 Mar 1, 2015 Abstract. We set out to derive the asymptotic distribution of the indirect least squares estimator of the IV model with one instrument, one endogenous variable and some covariates. The objective here is to be very explicit and detailed. To that end, we first recall the essential tools of probability theory necessary to complete the job. 1. Stochastic convergence In order to derive the asymptotic distribution of the indirect least squares estimator, we will need the following tools: LLN, CLT, CMT, Slutsky’s lemma, Delta method. Rather than giving short but unmotivated statements of these results, I would like to put things in some perspective. 1.1 Probability space. (Ω, F, P ) is a “black box” that controls/models random outcomes. Ω -set of points that represent all possible configurations of random outcomes, or equivalently, all possible states of nature. F -collection of subsets of Ω, called events. Events are those combinations of states of nature, to which a probability statement can be assigned. For technical reasons, not all subsets of Ω are events. P -probability function that assigns probabilities to events in F. That is it! 1.2 Random variables. Are measurable maps from the black box (Ω, F, P ) into the real world: Y : (Ω, F, P ) → R or X : (Ω, F, P ) → Rd . Measurability simply means that events in R (e.g. intervals) traced back to to the probability space via Y or X correspond to events there. This allows us to make probability statements about Y and X. 1.3 Pointwise convergence of random variables. Classical probability is about large n behavior of sequences of random variables. At least three different notions of convergence are relevant in econometrics. We start with the simpler two convergence concepts. In what follows we work with a fixed sequence of random variables or vectors Xn : (Ω, F, P ) → R or Rd . 1.3.1 A.s. convergence If we fix a particular realization ω, we are then looking at a sequence real numbers or vectors {Xn (ω)}n . A.s. convergence requires that these sequences of numbers converge for sufficiently many realizations ω: a.s. Xn → X iff ∀ω ∈ Ω \ N, Xn (ω) −−−→ X(ω), n→∞ where N must be an event with probability zero. This is simply pointwise convergence of functions. 1.3.2 Convergence in P Instead of fixing a realization ω and looking at the sequence of outcomes, we can fix a particular element of the sequence Xn and ask for the proportion of outcomes for which Xn is far from the 1 limit to be small: P Xn − →X P {ω ; |Xn (ω) − X(ω)| > } −−−→ 0. iff n→∞ =:An This is a weaker notion of pointwise convergence: any given configuration ω may enter and leave the sequence of bad evens An infinitely often as we move along the sequence, but the total mass of bad configurations must tend to zero as we move along the sequence. These convergence notions are simply convergence of functions that we learned in calculus/mathematical analysis. The next concept is not about functions at all. 1.4 Weak convergence. From the above sequence of random variables Xn : (Ω, F, P ) → R, we can construct a sequence of probability measures on the real world R that correspond to the following CDFs: Fn (x) := P {Xn ≤ x}. It may seem strange at first, but we can defined a notion of distance/closeness for pairs of probability measures. In words, two probability measures are close if they assign approximately the same mass to approximately the same locations. This intuitive description can be made formal to define a topology that corresponds to the notion of weak convergence. Here are some examples: 1. Point masses at location n1 converge weakly to a point mass at 0, denoted δ1/n h i n 1 δ1/n + n+1 δ5 δ0 . 2. n+1 3. N (µn , σn2 ) N (µ, σ 2 ) iff hµ i n 2 σn →  δ0 . µ  σ2 . 4. For i.i.d. centered1 sequence Yi , let Xn := N (0, EY12 ). √ n En Yi and Fn (x) := P {Xn ≤ x}. Then Fn The point worth emphasizing here is that weak convergence and the CLT we apply to our estimators have nothing to do with the random variables as maps from Ω. Contrary to what shorthand notation may suggest, the objects under investigation here are the probability distributions that the estimators induce on R. These summarize their ‘global’ behavior rather pointwise behavior as in Section 1.3. Important: Xn X does not imply that Xn (ω) converges at all for any ω. Example: take Y ∼ N (0, 1), and construct Xn := (−1)n Y. But at the same time Xn X does imply that √ P the sequence Xn remains bounded, so in particular Xn / n − → 0. How exactly do we define weak convergence? There are many equivalent ways, which is always a good thing!, as it gives as multiple tools for proofs: • via distribution functions (convergence at continuity points) • via probabilities attached to open/closed/nice sets • via integrating nice “test functions” This characterization is called Portmanteau lemma. 1 EY1 = 0 2 1.5 Limit theorems. Classical probability is all about product measures, which concentrate in the “middle” of the space as dimension grows. In the limit, the infinite product measure concentrates at a single point, but under appropriate rescaling looks like a Gaussian distribution. The framework of limits of statistical experiments makes a very nice use of this phenomena. Modern statistics is about studying the rate of this convergence to singularity in and interplay with the dimensionality of each coordinate, rather than looking only at the limit for fixed d. It is amazing that the most demanding results mentioned in this note take only two lines to state: LLN If Xn : (Ω, F, P ) → R or Rd are i.i.d. then a.s. En Xi −−→ EX1 iff E|X1 |1 < ∞. CLT If in addition E|X1 |22 < ∞ then √ nEn (Xi − EX1 )   N 0, E(X1 − EX1 )(X1 − EX1 )T . 1.6 Continuous mapping theorem. (CMT) All three notions of convergence mentioned above are preserved under smooth transformations. That is, if g : R → R or Rd → Rk is continuous, then a.s. a.s. Xn −−→ X implies g(Xn ) −−→ g(X); P P Xn − → X implies g(Xn ) − → g(X); Xn X implies g(Xn ) g(X); 1.7 Slutsky’s lemma. If Xn (Xn , Yn ) (X, Y ). P X and Yn − → c a constant, then we have joint weak convergence Remark. The result may seem trivial. It is not. Unlike pointwise (in ω) convergence where coordinatewise convergence is equivalent to joint (as whole vector) convergence, this is very different for joint probability measures. The source of the difference is in the fact that while coordinates determine the vector, marginal distributions do not determine the joint law. Examples: 1. Take X, Y ∼ N (0, 1) and compare (X, Y ) and (X, −Y ). i.i.d. 2. Fix Zi ∼ N (0, 1). Two sequences Xn := Zn and Yn := (−1)n Zn both converge weakly (to what?), but the vector (Xn , Yn ) does not converge in any sense (why?). Slutsky’s lemma indicated that we must be careful when making statements about joint convergence of two estimators. The reason the lemma is true, is due to the hypothesis that Yn convergence to a degenerate distribution concentrated at a single point. With on of the marginals being degenerate, there is only one way to construct the joint law. Corollary Under conditions of Slutsky’s lemma, we have that from CMT Xn + Yn X +c Yn · Xn c·X we use both of these every time we derive an asy distribution of an estimator. The last statement is used particularly often: under the conditions for LLN and CLT, √ En Yi · nEn [Xi − EXi ] EY1 · N (0, Var(X1 )). 3 1.8 Delta method. Suppose √ n(Xn − x0 ) N (0, Σ). 1. If g(x) = g(x0 ) + rgx0 (x − x0 ) is a linear map, then we get from CMT that h√ i √ rg · N (0, Σ). n(g(Xn ) − g(x0 )) = rgx0 n(Xn − x0 ) 2. For g that is differentiable at x0 but possibly nonlinear the same is also true. √ √ Intuition: If n(Xn − x0 ) N (0, Σ), then n(Xn − x0 ) is a bounded sequence, and therefore √ Xn converges to x0 with the rate of 1/ n tending to zero. Now, the smoothness of g at x0 implies that it is locally linear at x0 , so we are practically back to the linear case. 1.9 Stochastic o and O notation. It is very convenient to have the following short-hand notation P that we can use in proofs. We say that a sequence Xn of random variables is oP (1) if Xn − → 0. We think of oP (1) as a sequence that gets arbitrarily small eventually, but without any control on the rate of this convergence. We say that Xn is OP (1) if it is bounded in the following sense: for any probability tolerance level  > 0 there is a sufficiently large uniform deterministic bound M on the sequence so that P {ω ; |Xn (ω)| > M } ≤  for all n. This property is called uniform tightness. We think of OP (1) as a sequence that is bounded. Equivalently, uniform tightness is a compactness property of the sequence of distributions of Xn . Some examples: 1. If Yi is an i.i.d. sequence obeying LLN, then En (Yi − EY1 ) is oP . √ 2. If Yi is an i.i.d. sequence obeying CLT, then n En (Yi − EY1 ) is OP . 3. Combining previous two examples with Slutksy’s lemma, we have in this specific case √ En (Yi − EY1 ) · n En (Yi − EY1 ) = oP (1) · OP (1) = oP (1). Rules like the one above are extremely helpful in econometrics, we list some here: oP (1) + oP (1) = oP (1) oP (1) + OP (1) = OP (1) OP (1) + OP (1) = OP (1) oP (1) · OP (1) = oP (1) oP (1) · oP (1) = oP (1) (positive + oP (1))−1 = OP (1). It is a good exercise to prove these properties by combining definitions of op , OP and convergence in probability, uniform tightness. With this excellent notation, we can restate results of Slutsky’s lemma: if Xn X, then Xn + oP (1) X is P oP (1) · Xn − →0 is OP (1) oP (1). We are now fully equipped to study the asymptotic distribution of many estimators in econometrics. 4 2. Example: Asymptotic distribution of Indirect LS Recall our first structural model Y = α1 D + α2 W + U, U ⊥ Z, W D = β1 Z + β2 W + V, V ⊥ Z, W (IVM) In order to focus on the structural coefficient of interest α1 we partial out the controls W : Y D Z W U V = γY W = γDW = γZW = 1 = 0 = 0 W W W W W W + + + + + + Ye e D Ze 0 U V (ortho) is the orthogonal decomposition of all variables w.r.t. span(W ). Upon applying the linear operation of partialling out to eq. (IVM) we obtain e e + U, U ⊥ Z Ye = α1 D e = β1 Ze + V, V ⊥ Z. e D (toyIVM) Indirect least squares approach is based on the observation that orthogonality condition in eq. (toyIVM) e on Z, e the so called first-stage. We push this idea furimply that β1 is a projection coefficient of D ther: upon substituting the first stage into the main structural equation, we obtain the following reduced form: Ye = α1 β1 Ze + U + α1 V =: =:γ and observe again that the orthogonality conditions of eq. (toyIVM) imply that γ is the projection e Thus, assuming that there is a non-trivial first stage, β1 6= 0, we can identify coefficient of Ye onto Z. the structural coefficient α1 as γ α1 = . (1) β1 Our goal here is to characterize the asymptotic distribution of an estimator based on eq. (1). The first step is to wright down an explicit formula for the projection coefficients in eq. (1). It will be necessary to analyze these coefficients jointly, so I set up a vector: " # " # e) β1 (EZe2 )−1 (EZeD θ := = , γ (EZe2 )−1 (EZeYe ) where I used the formula for orthogonal projection coefficient. We obtain an estimator θe by replacing expectations (which we don’t know how to compute because we don’t know the underlying P ) with empirical expectation (which works asymptotically as good as P by LLN): " # e i) e2 )−1 (En Z ei D (E Z n i (2) θe := ei Ye i ) . (En Ze2i )−1 (En Z e i , Ye i } is required. But we only have a sample of In order to compute θe, a sample of {Zei , D {Zi , Di , Yi , Wi }, since we do no know the projection coefficients in eq. (ortho) we cannot construct a sample of tilded variables. But we can estimate all the projection coefficients in eq. (ortho) and 5 construct estimates of the tilded variables, b which we will denote with checks: Yb := Y − γ bY W W b (ortho) D bDW W b := D − γ Z := Z − γ bZW W Here γ b’s are OLS estimates; we will use the fact that OLS estimators are consistent in the what follows. We define an estimator based on checks: b b b # " 2 −1 (En Z b i ) (En Z b iD b i) . θb := 2 −1 (En Z i ) (En Z i Y i ) The plan is to establish asymptotic properties of θe and then show that θb is equivalent to θe in the sense that √ √ n(θb − θ) = n(θe − θ) + oP (1), (3) which would imply that θe and θb have the same asymptotic distribution, see Section 1.9. First we make the simple observation that θe is consistent, so long as the LLN works, as can be see directly from eq. (2) via CMT. A sufficient condition for the LLN here is EkZk22 , EkDk22 , EkY k22 < ∞. e and the reduced form equation for Ye into Next we substitute the first stage equation for D formula (2) " # " # ei D e i) ei β1 Zei + Zei Vi ) (E Z (E Z n n −1 −1 2 2 = (En Zei ) θe := (En Zei ) (En Zei Ye i ) (En Zei γZei + Zei i ) after regrouping and rescaling we obtain √ n(θe − θ) = (En Ze2i ) From the last display we read off h that e )2 E(ZV and Slutsky’s lemma, where Vθ = −1 √ " # (Zei Vi ) nEn e (Z i i ) (4) √ e n(θ −i θ) (EZe2 )−2 N (0, Vθ ) by LLN, CLT, CMT 2 e ) E(V Z e 2 . A sufficient condition for the CLT here is E(Z) EkZk44 , EkDk44 , EkY k44 < ∞. Finally we can use the Delta method to get the asymptotic distribution of α e = φ(θe), for x2 x2 1 φ(x1 , x2 ) = x1 , with rφx = [− x2 x1 ]: 1 √ n(α e − α) = √ n(φ(θe) − φ(θ)) [− γ 1 ](EZe2 )−2 N (0, Vθ ).2 β12 β1 We now argue eq. (3). Per discussion in Section 1.7, it is enough to show this for each component √ of θ. Since the analysis for each component is identical, we will only show that n(βb − βe) = oP (1). Here we again begin by looking at the exact formula for each estimator: b b b i 2 √ √ h −1 b e ei n(β − β ) = n (En Z i ) En Z i Di − (En Ze2i )−1 En Zei D b b b b b i √ h i 2 2 2 √ h −1 −1 e e e i − (En Zei2 )−1 En Zei D ei = n (En Z i ) En Z i Di − (En Z i ) En Z i Di + n (En Z i )−1 En Zei D b b hb b i 2 −1 √ e ei + nEn Z i (Di − Di ) + (Z i − Zei )D ... = (En Z i ) chunk1 chunk2 chunk3 Term2 It is good to break the problem into small chunks: We work with the two large terms in the last 2 compare this to Theorem 2 of L2 6 screen separately, and break each of the terms into three small chunks. To compare checks and b tildas, we combine eq. (ortho) and eq. b (ortho) to obtain: e (5) D bDW )Wi bi − Di = (γDW − γ Z i − Zei = (γZW − γ (6) bZW )Wi Chunk1 From eq. (6) b 2 En Z i = En Ze2i + 2(γZW − γ bZW )En Zei Wi + (γZW − γ bZW )2 En Wi2 = En Ze2i + oP (1)OP (1) + oP (1)OP (1) = EZe2i + oP (1). Chunk2 From eq. (5) and b eq. b (6) b b √ √ √ e nEn Z i (Di − Di ) = (γDW − γ bDW ) nEn Z i Wi = oP (1) nEn Z i Wi h√ i √ = oP (1) nEn Zei Wi + oP (1) nEn Wi2 = oP (1)OP (1) = oP (1). Chunk3 From eq. (5) b √ √ e i = oP (1) nEn D b i Wi nEn (Z i − Zei )D = oP (1)OP (1) = oP (1). Putting these observations together, we conclude that the first term is h i Term1 = (positive + oP (1))−1 oP (1) + oP (1) = OP (1)oP (1) = oP (1). Analysis for Term2 is similar. Remark. It is difficult to keep track of so many different empirical sums of various random variables as on the screen at bottom of previous page. Notation oP (1) and OP (1) is extremely helpful here, allowing us to focus on the size of each bit and parse the whole formula by applying simple rules stated in Section 1.9. References [1] S. Resnick. “A Probability Path” (1998). [2] A. W. Van der Vaart. Asymptotic statistics. Vol. 3. Cambridge university press, 2000. A deeper overview of results mentioned in Section 1 can be found in Vaart [2]. A rigorous and very explicit development of probability theory from ground zero with lots of excellent exercises and links to statistics can be found in Resnick [1]. 7