Lecture Notes 9: Bootstrap

Introduction 1 14.384 Time Series Analysis Professor Anna Mikusheva Lecture 9 Bootstrap The goal oof this lecture is to cover the basics for bootstrap procedures. This lecture is NOT specific to Time series. For some unexplainable reasons bootstrap is missing in our econometric sequence. I have to teach you this topic since we’ll be using some bootstrap-based procedures. A good reference is Chapter 52 in Handbook of Econometrics by Horowitz. Introduction We have a sample z = {zi , i = 1, ..., n} from a distribution F0 . We have a statistic of interest, T (z) whose distribution we want to know because we want to test a hypothesis or to construct a confidence set based on this statistic. Let Gn (t, F0 ) = P (T (z) ≤ t) be the cdf of T (z). In general, Gn is some complicated function of F0 . If we knew F0 we would be able to calculate Gn . However, in general F0 is unknown, and this poses the problem. There are pretty much two ways to get around: asymptotics and simulation techniques. In both cases we want to approximate Gn . Asymptotics One way is to create an asymptotic approximation (in which the dependence on F0 disappears) to the distribution Gn by letting the sample size n increase to infinity and using CLT and delta-method. If we can get Gn (t, F0 ) → G∞ (t) as n → ∞, then we can pretend that Gn (t, F0 ) ≈ G∞ (t) and use quantiles of the latter to make inferences. Example 1. An example of asymptotic approach is as follows: Let zi be iid with an unknown distribution F0 , which has an unknown mean Ezi = µ, and variance Ezi2 = σ 2 . Suppose we want√to test a hypothesis H0 : µ = µ0 and to construct a confidence set for the mean P by using t-statistic T (z) = σbn ( n1 zi − µ0 ). In general T (z) has some complicated distribution depending on F0 . Say, if F0 is Gaussian, then T (z) is Student-t distributed, but in general, it’s not known. If we knew F0 , we can simulate Gn . However, we know that T (z) ⇒ N (0, 1) so we can believe that Gn is very close to Φ for large sample size n. This gives us a justification for using the normal distribution to compute p-values for hypothesis testing and to compute confidence intervals. For example, let zα be α-quantile of the standard normal distribution H0 : if P (Φ(z√α ) = α). Then P we accept √ Zα/2 ≤ T (z) ≤ Z1−α/2 . And a 95% confidence interval would be [ n1 zi −σ b nZ1−α/2 , n1 zi −σ b nZ1−α/2 ]. Bootstrap 2 Bootstrap The bootstrap is another approach to approximating Gn (t, F0 ). Instead ofPusing the asymptotic distribution n to approximate Gn (·, F0 ), we use Gn (·, F0 ) ≈ Gn (·, F̂n ) where F̂n (t) = n1 i=1 1(x ≤ t) is the empirical cdf. The reasoning is based on a presumption that Fn is probably very close to F0 , and Gn depends on F0 in a continuous way. It is usually numerically cumbersome to calculate how Gn depends on F0 . In practice, it is much easier to use simulations to compute Gn (·, F̂n ). A general algorithm for the bootstrap with iid data is ∗ 1. Generate bootstrap sample zb∗ = {z1∗b , ..., znb } independently drawn from F̂n for b = 1..B. By this, we ∗ mean that zib are drawn independently with replacement from {zi }ni=1 . 2. Calculate Tb∗ = T (zb∗ ) PB 1 ∗ 3. GB n (t, F̂n ) = B b=1 1(Tb ≤ t). ˆ If B is large, then GB n (t, F̂n ) is very close to Gn (t, Fn ) (the simulation accuracy is in our hands). We will let Zq be the qth quantile of Gn (t, F̂n ). In our procedure it is Zq ≈ T([∗Bα]) , here [·] is a whole part, and T(i) is i−th order statistics. Consistency Theorem 2. Conditions sufficient for bootstrap consistency are: 1. limn→∞ Pn [ρ(F0 , F̂n ) > ] = 0, ∀ > 0, where ρ() is some metric (the exact metric depends on the application) 2. G∞ (τ, F ) is continuous in τ 3. ∀τ and Hn s.t. ρ(Hn , F ) → 0, we have Gn (τ, Hn ) ⇒ G∞ (τ, F ) p Under these conditions the bootstrap is weakly consistent, i.e. supτ |Gn (τ, Fn ) − Gn (τ, F0 )| → 0. Remark 3. The bootstrap can also be consistent under weaker conditions. Remark 4. We never said that G∞ (τ, F ) should be normal, but in the majority of applications it is. Asymptotic Refinement Example 5. Consider the same setup as the previous example. Consider calculating two different statistics from the bootstrap: √ T1 (z) = n(z̄ − µ) ! √ z̄ − µ T2 (z) = n p s2 (z) The bootstrap analogs of these are √ T1∗,b (z) = n(z¯∗ − z¯) √ T2∗,b (z) = n z¯∗ − z̄ p s2 (z ∗ ) ! We can use these two statistics to compute two different confidence intervals. 1 √1 Hall’s interval is: [z̄ − Z11−α/2 √1n , z̄ + Zα/2 ] n Asymptotic Refinement 3 q q s2 (z) s2 (z) 2 2 t-percentile: [z̄ − Z1−α/2 , z̄ + Z α/2 n ] n ∗ where Zαi are quantiles of Ti,b (z). Which confidence set may be better? The set based on the first statistic uses that: T1 (z) ⇒ N (0, σ 2 ) and T1∗ (z) ⇒ N (0, σ 2 ) as n → ∞ T2 (z) ⇒ N (0, 1) and T2∗ (z) ⇒ N (0, 1) as n → ∞ Definition 6. A statistic is (asymptotically) pivotal if its (asymptotic) distribution does not depend on any nuisance parameters. The t-statistic (statistic T2 (z) above) is pivotal, T1 (z) is not. The bootstrap usually provides an asymptotic refinement if used for a pivotal statistics. It means that the approximation by bootstrap is of higher order than the approximation achieved by asymptotic approach. In our case, under some technical assumptions (moment and Cramer conditions) we have what’s called Edgeworth expansion: P( z̄ − µ √ 1 1 1 n ≤ t) = Φ(t) + √ h1 (t, F0 ) + h2 (t, F0 ) + O( 3/2 ) s(z ) n n n √ Edgeworth expansion characterize the closeness of P ( z̄−µ s(z) n ≤ t) and Φ(t). We see that the difference between them is of order √1n . It is also known that h1 (t, F0 ) is continuous function in t and F0 and depend only on first 3 moments of F0 . There is also Edgeworth expansion for the bootstrapped distribution: P( z̄ ∗ − z̄ √ 1 1 1 n ≤ t) = Φ(t) + √ h1 (t, F̂n ) + h2 (t, F̂n ) + O( 3/2 ) ∗ s(z ) n n n Taking the difference between these two equations we have:  z̄ − µ √ z̄ ∗ − z̄ √ 1  ˆn ) + O( 1 ) = O( 1 ) √ n ≤ t) − P ( n ≤ t) = h (t, F ) − h (t, F 1 0 1 n n s(z ) s(z ∗ ) n   The fact that h1 () is uniformly continuous and F̂n − F0 = O( √1n ) tells us that √1n h1 (t, F0 ) − h1 (t, Fˆn ) = P( O( n1 ). That is, when we bootstrap a pivotal statistic in our simple example, the accuracy of the approximation is O( n1 ), whereas the accuracy of the asymptotic approximation is O( √1n ). This gain is accuracy is called asymptotic refinement. Note that for this argument to work, we needed our statistic to be pivotal because otherwise, the first term in the Edgeworth expansion would not be the same for the true distribution and the bootstrap distribution. Consider: √ 1 P ((z̄ − µ) n ≤ t) =Φ(t/σ) + O( √ ) n √ 1 P ((z̄ ∗ − z̄) n ≤ t) =Φ(t/s(z)) + O( √ ) n The difference is √ √ 1 P ((z̄ − µ) n ≤ t) − P ((z̄ ∗ − z̄) n ≤ t) =Φ(t/σ) − Φ(t/s(z)) + O( √ ) n 1 1 1 ≈φ(x/σ) 2 (σ − s(z)) + O( √ ) = O( √ ) σ n n Bias Correction 4 Bias Correction Another task for which bootstrap is used is bias-correction. Suppose, Ez = µ and we’re interested in a non-linear function of µ, say θ = g(µ). One approach would be to take an unbiased estimate of µ, say z̄ and plug it into g(), θ̂ = g(z̄). θ̂ is consistent, but it will not be unbiased unless g() is linear. The bias is Bias = Eθ̂ − g(µ). We can estimate the bias using the bootstrap: ∗ } 1. Generate bootstrap sample, zb∗ = {zib 2. Estimate θb∗ = g(z̄b∗ ) PB 3. Bias∗ = B1 b=1 θb∗ − θˆ ≈ Bias 4. Use θ̃ = θ̂ − Bias∗ as your estimate Why it works? Let’s denote G1 (µ) = dg(µ) dµ and G2 (µ) = d2 g(µ) dµ2 . Then 1 1 θb − θ = G1 (µ)(z̄ − µ) + G2 (µ)(z̄ − µ)2 + op ( ) 2 n 1 1 σ2 1 Bias = E(θb − θ) = G2 (µ)E(z̄ − µ)2 = G2 (µ) + o( ) 2 2 n n similarly Bias∗ = 1 s2 1 G2 (z̄) + op ( ) 2 n n As a result, 1 Bias∗ − Bias = op ( ) n Remark 7. This procedure required a consistent estimator to begin with. Bootstrap do’s and don’ts • If you have a pivotal statistic, bootstrap can give a refinement. So, if you have choice of statistics, bootstrap a pivotal one. • Bootstrap may fix a finite-sample bias, but cannot help if you have inconsistent estimator. • In general, if something does not work with traditional asymptotics, the bootstrap cannot fix your problem. For example, if we have an inconsistent estimate, the bootstrap bias correction does not fix anything. • Bootstrap could not fix the following problems: weak instruments, parameter on a boundary, unit root, persistent regressors. • Bootstrap requires re-centering (the null hypothesis to be true). The next section is about it. Re-centering The constraints of our model should also be satisfied in our bootstrap replications of the model. For example, assume you are doing estimation using GMM for a population moment condition (that is, you are working under assumption that it is true), Eh(zi , θ) = 0 Bootstrap Variants and OLS 5 When you are simulating bootstrap replications the analogous condition must hold: E ∗ h(zi∗ , θ̂) = 0 which is equivalent to 1X h(zi , θ̂) = 0 n If the model is overidentified, this condition won’t hold. To make it hold, we redefine h̃(zi∗ , θ) = h(zi∗ , θ) − 1X h(zi , θ̂) n and use h̃() to compute the bootstrap estimates, θb∗ . More generally, we need to make sure that our null hypothesis holds in our bootstrap population. Bootstrap Variants and OLS Model: yt = xt β + e t we estimate it by OLS and get β̂ and êt . Now we generate bootstrapped samples such that yt∗ = x∗t βˆ + e∗t There are at least three ways to do sampling : • Take zi = {(xi , êi } (xi is always drawn with êi ). This approach preserves any dependence there might be between x and e. For example, if we have heteroskedasticity. • If xt is independent of et , we can sample them independently. Draw x∗t from {xt } and independently draw e∗t from {êt }. This is likely to be more accurate than the first approach when we really have independence. • Parametric bootstrap: Draw x∗t from {xt }, and independently draw e∗t from N (0, σ̂ 2 ) Remark 8. In time series, all of these approaches might be inappropriate. If {xt , et } is auto-correlated, then these approaches would not preserve the time dependence among errors. One way to proceed is to use the block bootstrap, i.e. sample contiguous blocks of {xt , et } together.