Simple Linear Regression Least Squares Estimates of β0 and β1

Simple Linear Regression Least Squares Estimates of β0 and β1 Simple linear regression involves the model Ŷ = µY |X = β0 + β1 X. This document derives the least squares estimates of β0 and β1 . It is simply for your own information. You will not be held responsible for this derivation. The least squares estimates of β0 and β1 are: ∑n i=1 (Xi − X̄)(Yi − Ȳ ) ∑n βˆ1 = 2 i=1 (Xi − X̄) βˆ0 = Ȳ − βˆ1 X̄ The classic derivation of the least squares estimates uses calculus to∑find the β0 and β1 parameter estimates that minimize the error sum of squares: SSE = ni=1 (Yi − Ŷi )2 . This derivation uses no calculus, only some lengthy algebra. It uses a very clever method that may be found in: Im, Eric Iksoon, A Note On Derivation of the Least Squares Estimator, Working Paper Series No. 96-11, University of Hawai’i at Manoa Department of Economics, 1996. The Derivation The least squares estimates are estimates βˆ0 and βˆ1 that minimize the error sum of squares SSE = n ∑ i=1 1 (Yi − Ŷi )2 . We can algebraically manipulate things to get SSE = n ∑ (Yi − Ŷi )2 i=1 = = n ∑ i=1 n ∑ (Yi − β0 − β1 Xi )2 [ (Yi + Ȳ − Ȳ ) − β0 − β1 (Xi + X̄ − X̄) ]2 i=1 = n ∑ [ (Ȳ − β0 − β1 X̄) + Yi − Ȳ − β1 Xi + β1 X̄ ]2 i=1 = = n ∑ [ i=1 n ∑ [ ]2 (Ȳ − β0 − β1 X̄) − (β1 Xi − β1 X̄ − Yi + Ȳ ) n ] ∑ [ ]2 (Ȳ − β0 − β1 X̄)2 + (β1 Xi − β1 X̄ − Yi + Ȳ ) i=1 i=1 n ∑ [ ]2 2 = n(Ȳ − β0 − β1 X̄) + (β1 Xi − β1 X̄ − Yi + Ȳ ) = n(Ȳ − β0 − β1 X̄)2 + i=1 n ∑ [ ]2 β1 (Xi − X̄) − (Yi − Ȳ ) i=1 n ∑ [ 2 ] = n(Ȳ − β0 − β1 X̄)2 + β1 (Xi − X̄)2 − 2β1 (Xi − X̄)(Yi − Ȳ ) + (Yi − Ȳ )2 i=1 = n(Ȳ − β0 − β1 X̄) + 2 β12 n n n ∑ ∑ ∑ 2 (Xi − X̄) − 2β1 (Xi − X̄)(Yi − Ȳ ) + (Yi − Ȳ )2 i=1 i=1 i=1 = n(Ȳ − β0 − β1 X̄) ∑n ∑ )∑ ( n 2 2β1 ni=1 (Xi − X̄)(Yi − Ȳ ) 2 i=1 (Yi − Ȳ ) ∑n + ∑n (Xi − X̄)2 + β1 − 2 2 (X − X̄) (X − X̄) i i i=1 i=1 i=1 2 = n(Ȳ − β0 − β1 X̄)2  [ ∑n ]2  ∑n 2β (Xi −X̄)(Yi −Ȳ ) (X −X̄)(Yi −Ȳ ) n i=1 ∑n i ∑ β12 − 1 ∑i=1 + n 2 2 i=1 (Xi −X̄) i=1 (Xi −X̄)   ] [ + (Xi − X̄)2 ∑n ∑n 2 2 (Y − Ȳ ) (X − X̄)(Y − Ȳ ) i i i i=1 ∑n + ∑ni=1(Xi −X̄)2 − i=1 2 i=1 i=1 (Xi −X̄) ∑n ( )2 (X − X̄)(Yi − Ȳ ) ∑n i = n(Ȳ − β0 − β1 X̄)2 + β1 − i=1 2 i=1 (Xi − X̄)  [ ]2  ∑n n ∑ (Xi − X̄)(Yi − Ȳ )  + (Yi − Ȳ )2 1 − √∑n i=1 ∑n 2 2 (Y − Ȳ ) (X − X̄) i i=1 i i=1 i=1 2 We’re still trying to minimize the SSE, and we’ve split the SSE into the sum of three terms. Note that the first two terms involve the parameters β0 and β1 . The first two terms are also squared terms, so they can never be less than zero. The third term is only a function of the data and not the parameter. So, we know that ∑n ( )2 2 i=1 (Xi − X̄)(Yi − Ȳ ) ∑n SSE = n(Ȳ − β0 − β1 X̄) + β1 − 2 i=1 (Xi − X̄)  [ ]2  ∑n n ∑ (Xi − X̄)(Yi − Ȳ )  + (Yi − Ȳ )2 1 − √∑n i=1 ∑n 2 2 (X − X̄) (Y − Ȳ ) i i=1 i=1 i i=1  [ ]2  ∑n n ∑ (Xi − X̄)(Yi − Ȳ )  ≥ (Yi − Ȳ )2 1 − √∑n i=1 ∑n 2 2 (X − X̄) (Y − Ȳ ) i i i=1 i=1 i=1 This is the minimum possible value for the SSE. We actually achieve this minimum value when the first two terms of the equation above are zero. Setting each of these two terms equal to zero gives us two equations in two unknowns, so we can solve for β0 and β1 . 0 = n(Ȳ − β0 − β1 X̄)2 ∑n ( )2 i=1 (Xi − X̄)(Yi − Ȳ ) ∑n 0 = β1 − 2 i=1 (Xi − X̄) From the first equation we get 0 = n(Ȳ − β0 − β1 X̄)2 ⇒ 0 = Ȳ − β0 − β1 X̄ ⇒ β0 = Ȳ − β1 X̄ From the second equation we get ∑n ( )2 i=1 (Xi − X̄)(Yi − Ȳ ) ∑n 0 = β1 − 2 i=1 (Xi − X̄) ∑n (X − X̄)(Yi − Ȳ ) ∑n i ⇒ 0 = β1 − i=1 2 i=1 (Xi − X̄) ∑n (X − X̄)(Yi − Ȳ ) ∑n i ⇒ β1 = i=1 2 i=1 (Xi − X̄) As these are estimates, we put hats on them. We are done! We’ve now shown that ∑n i=1 (Xi − X̄)(Yi − Ȳ ) ∑n βˆ1 = 2 i=1 (Xi − X̄) βˆ0 = Ȳ − βˆ1 X̄. 3