Locally Linear Regression

Locally Linear Regression: There is another local method, locally linear regression, that is thought to be superior to kernel regression. It is based on locally fitting a line rather than a constant. Unlike kernel regression, locally linear estimation would have no bias if the true model were linear. In general, locally linear estimation removes a bias term from the kernel estimator, that makes it have better behavior near the boundary of the x’s and smaller MSE everywhere. To describe this estimator, let Kh (u) = h−r K(u/h) as before. Consider the estimator ĝ(x) given by the solution to min g,β n � (Yi − g − (x − xi )� β)2 Kh (x − xi ). i=1 That is ĝ(x) is the constant term in a weighted least squares regression of Yi on (1, x − xi ), with weights Kh (x − xi ). For ⎛ ⎞ ⎛ ⎞ Y1 1 (x − x1 )� ⎜ . ⎟ ⎜ . ⎟ .. ⎟ ⎜ ⎟ Y =⎜ . ⎝ .. ⎠ , X = ⎝ .. ⎠ � Yn 1 (x − xn ) W = diag(Kh (x − x1 ), . . . , Kh (x − xn )) and e1 a (r + 1) × 1 vector with 1 in first position and zeros elsewhere, we have ĝ(x) = e�1 (X � W X)−1 X � W Y. This estimator depends on x both through the weights Kh (x − xi ) and through the regressors x − xi . This estimator is a locally linear fit of the data. It runs a regression with weights that are smaller for observations that are farther from x. In constrast, the kernel regression estimator solves this same minimization problem but with β constrained to be zero, i.e., kernel regression minimizes n � (Yi − g)2 Kh (x − xi ) i=1 Removing the constriant β = 0 leads to lower bias without increasing variance when g0 (x) is twice differentiable. It is also of interest to note that β̂ from the above minimization problem estimates the gradient ∂g0 (x)/∂x. Like kernel regression, this estimator can be interpreted as a weighted average of the Yi observations, though the weights are a bit more complicated. Let S0 = n � i=1 Kh (x − xi ), S1 = n � Kh (x − xi )(x − xi ), S2 = i=1 n � i=1 1 Kh (x − xi )(x − xi )(x − xi )� m̂0 = n � Kh (x − xi )Yi , m̂1 = i=1 n � Kh (x − xi )(x − xi )Yi . i=1 Then, by the usual partitioned inverse formula �−1 � � � S0 S1� m̂0 = (S0 − S1� S2−1 S1 )−1 (m̂0 − S1� S2−1 m̂1 ) ĝ(x) = S1 S2 m̂1 �n ai Yi = �i=1 , ai = Kh (x − xi )[1 − S1� S2−1 (x − xi )] n i=1 ai e�1 It is straightforward though a little involved to find asymptotic approximations to the MSE. For simplicity we do this for scalar x case. Note that for g0 = (g0 (x1 ), . . . , g0 (xn ))� , ĝ(x) − g0 (x) = e�1 (X � W X)−1 X � W (Y − g0 ) + e�1 (X � W X)−1X � W g0 − g0 (x). Then for Σ = diag(σ 2 (x1 ), . . . , σ 2 (xn )), � � E (ĝ(x) − g0 (x))2 |x1 , . . . , xn = e�1 (X � W X)−1 X � W ΣW X(X � W X)−1e1 � + e�1 (X � W X)−1 X � W g0 − g0 (x) �2 An asymptotic approximation to MSE is obtained by taking the limit as n grows. Note that we have n−1 h−j Sj = n 1� Kh (x − xi )[(x − xi )/h]j n i=1 Then, by the change of variables u = (x − xi )/h, � −1 −j � � j E n h Sj = E Kh (x − xi ) ((x − xi )/h) for µj = � � = � K(u)uj f0 (x − hu)du = µj f0 (x) + o(1). K(u)uj du and h −→ 0. Also, −1 −j −1 � 2 var(n h Sj ) ≤ n E Kh (x − xi ) ((x − xi )/h) ≤ Cn−1 h−1 −→ 0 2j � −1 −1 ≤n h � K(u)2 u2j f0 (x − hu)du for nh −→ ∞. Therefore, for h → 0 and nh → ∞ n−1 h−j Sj = µj f0 (x) + op (1). Now let H = diag(1, h). Then by µ0 = 1 and µ1 = 0 we have n−1 H −1 X � W XH −1 = n−1 � S0 h−1 S1 −1 h S1 h−2 S2 2 � = f0 (x) � 1 0 0 µ2 � + op (1). Next let νj = � K(u)2 uj du. Then by a similar argument we have h· n 1� Kh (x − xi )2 [(x − xi )/h]j σ 2 (xi ) = νj f0 (x)σ 2 (x) + op (1). n i=1 It follows by ν1 = 0 that −1 n hH −1 � X W ΣW XH −1 2 = f0 (x)σ (x) � ν0 0 0 ν2 � + op (1). Then we have, for the variance term, by H −1 e1 = e1 , e�1 (X � W X)−1 X � W ΣW X(X � W X)−1e1 = n−1 h−1 e�1 H −1 ⎡⎛ = n−1 h−1 ⎣⎝e�1 � � H −1 X � W XH −1 n 1 0 0 µ2 �−1 � �−1 ν0 ν1 ν1 ν2 hH −1 X � W ΣW XH −1 n �� 1 0 0 µ2 �−1 � ⎞ H −1 X � W XH −1 n �−1 ⎤ σ 2 (x) e1 ⎠ + op (1)⎦ . f (x) Assuming that µ1 = 0 as usual for a symmetric kernel we obtain e�1 (X � W X)−1X � W ΣW X(X � W X)−1 e1 −1 −1 =n h � � σ 2 (x) + op (1) . ν0 f (x) For the bias consider an expansion 1 1 g(xi ) = g0 (x) + g0� (x)(xi − x) + g0�� (x)(xi − x)2 + g0��� (x̄i )(xi − x)3 . 2 6 Let ri = g0 (xi ) − g0 (x) − [dg0 (x)/dx](xi − x). Then by the form of X we have g = (g0 (x1 ), . . . , g0 (xn ))� = g0 (x)W e1 − g0� (x)W e2 + r It follows by e�1 e2 = 0 that the bias term is e�1 (X � W X)−1 X � W g − g0 (x) = e�1 (X � W X)−1 X � W Xe1 g0 (x) − g0 (x) +e�1 (X � W X)−1 X � W Xe2 g0� (x) + e�1 (X � W X)−1 X � W r = e�1 (X � W X)−1 X � W r. Recall that n−1 h−j Sj = µj f0 (x) + op (1). Therefore 3 H −1 e1 1 � � � 2 1 S2 µ2 1 �� g0��(x) = f0 (x) g (x) + op (1). S3 2 µ3 2 0 n−1 h−2 H −1 X � W ((x − X1 )2 , . . . , (x − Xn )2 )� = � n−1 h−2 n−1 h−3 Also, by g0��� (x̄i ) bounded � � � �� � � −1 −2 −1 � �n h H X W (x − x1 )3 g ��� (x̄1 ), . . . , (x − xn )3 g ��� (x̄n ) � � 0 0 � � � � Kh (x − xi )|x − xi |3 , n−1 h−2 S4 −→ 0. ≤ C max n−1 h−2 i Therefore, we have (H −1 X � W XH −1 )−1 h−2 H −1 X � W r · n n � �−1 � � 2 2 h �� h 1 0 µ2 = g0 (x)e�1 = g0�� (x)µ2 . 0 µ2 µ3 2 2 e�1 (X � W X)−1 X � W r = h2 e�1 H −1 Exercise: Apply analogous calculation to show kernel regression bias is 2 µ2 h � f � (x) 1 �� g0 (x) + g0� (x) 0 2 f0 (x) � Notice bias is zero if function is linear. Combining the bias and variance expression, we have the following form for asymptotic MSE: 1 σ 2 (x) h4 �� 2 2 ν0 + g0 (x) µ2 . nh f0 (x) 4 In constrast, the kernel MSE is � f � (x) 1 σ 2 (x) h4 �� ν0 + g0 (x) + 2g0� (x) 0 nh f0 (x) 4 f0 (x) �2 µ22 . Bias will be much bigger near boundary of the support where f0� (x)/f0 (x) is large. For example, if f0 (x) is approximately xα for x > 0 near zero, then f0� (x)/f0 (x) grows like 1/x as x gets close to zero. Thus, locally linear has smaller boundary bias. Also, locally linear has no bias if g0 (x) is linear but kernel obviously does. Simple method is to take expected value of MSE. 4