Orthogonal Projections

6.3 Orthogonal Projections Math 2331 – Linear Algebra 6.3 Orthogonal Projections Math 2331, Linear Algebra 1 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation 6.3 Orthogonal Projections Orthogonal Projection: Review Orthogonal Projection: Examples The Orthogonal Decomposition Theorem The Orthogonal Decomposition: Example Geometric Interpretation of Orthogonal Projections The Best Approximation Theorem The Best Approximation Theorem: Example New View of Matrix Multiplication Orthogonal Projection: Theorem Math 2331, Linear Algebra 2 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation Orthogonal Projection: Review b y= y·u u·u u is the orthogonal projection of onto . Suppose {u1 , . . . , up } is an orthogonal basis for W in Rn . For each y in W ,     y · up y · u1 y= u1 + · · · + up u1 · u1 up · up Math 2331, Linear Algebra 3 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation Orthogonal Projection: Example Example Suppose {u1 , u2 , u3 } is an orthogonal basis for R3 and let W =Span{u1 , u2 }. Write y in R3 as the sum of a vector b y in W and a vector z in W ⊥ . Math 2331, Linear Algebra 4 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation Orthogonal Projection: Example (cont.) Solution: Write       y · u1 y · u2 y · u3 y= u + u2 + u3 u1 · u1 u2 · u2 u3 · u3 1 where  b y= y · u1 u1 · u1   u + 1 y · u2 u2 · u2   u2 , z= y · u3 u3 · u3  u . 3 To show that z is orthogonal to every vector in W , show that z is orthogonal to the vectors in {u1 , u2 } . Since z · u1 = = =0 z · u2 = = =0 Math 2331, Linear Algebra 5 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation The Orthogonal Decomposition Theorem Theorem (8) Let W be a subspace of Rn . Then each y in Rn can be uniquely represented in the form y =b y+z where b y is in W and z is in W ⊥ . In fact, if {u1 , . . . , up } is any orthogonal basis of W , then     y · up y · u1 b u1 + · · · + up y= u1 · u1 up · up and z = y−b y. The vector b y is called the orthogonal projection of y onto W . Math 2331, Linear Algebra 6 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation The Orthogonal Decomposition Theorem Math 2331, Linear Algebra 7 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation The Orthogonal Decomposition: Example Example       3 0 0 Let u1 =  0 , u2 =  1  , and y =  3 . Observe that 1 0 10 {u1 , u2 } is an orthogonal basis for W =Span{u1 , u2 }. Write y as the sum of a vector in W and a vector orthogonal to W . Solution: projW y = b y=  y·u1 u1 ·u1  u1 +  y·u2 u2 ·u2  u2       3 0 3      =( ) 0 +( ) 1 = 3  1 0 1       0 3 −3 z = y−b y = 3 − 3 = 0  10 1 9 Math 2331, Linear Algebra 8 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation Geometric Interpretation of Orthogonal Projections Math 2331, Linear Algebra 9 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation The Best Approximation Theorem Theorem (9 The Best Approximation Theorem) Let W be a subspace of Rn , y any vector in Rn , and b y the orthogonal projection of y onto W . Then b y is the point in W closest to y, in the sense that ky−b yk < ky − vk for all v in W distinct from b y. Math 2331, Linear Algebra 10 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation The Best Approximation Theorem (cont.) Outline of Proof: Let v in W distinct from b y. Then v−b y is also in W (why?) z = y−b y is orthogonal to W y − v = (y−b y) + (b y−v) =⇒ ⇒ y−b y is orthogonal to v−b y ky − vk2 = ky−b yk2 + kb y−vk2 . ky − vk2 > ky−b y k2 Hence, ky−b yk < ky − vk. Math 2331, Linear Algebra 11 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation The Best Approximation Theorem: Example Example Find the closest point to y in Span{u1 , u2 } where       2 1 0  4   1   0       y =  0 , u1 =  0 , and u2 =  1 . −2 0 1 Solution:  b y=  =( y · u1 u1 · u1  1  1   )  0 +( 0   u1 + y · u2 u2 · u2  u2   0  0   )  1 = 1 Math 2331, Linear Algebra 12 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation New View of Matrix Multiplication Part of Theorem 10 below is based upon another way to view matrix multiplication where A is m × p and B is p × n   row1 B     row2 B  AB = col1 A col2 A · · · colp A   ..   . rowp B = (col1 A) (row1 B) + · · · + (colp A) (rowp B) Math 2331, Linear Algebra 13 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation New View of Matrix Multiplication: Example Example  5 6 3 1    = 5 3   2 1 3 4 0 −2 5 6 3 1  2 1 3    34 5 3 10 3 7 = 2 1 3 4 0 −2  + 6 1     4 0 −2  = Math 2331, Linear Algebra 14 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation Orthogonal Matrix  So if U =  u1 u2 · · ·    up . Then U T =   uT 1 uT 2 .. . uT p    . So  T T UU T = u1 uT 1 + u2 u2 + · · · + up up

T T UU T y = u1 uT 1 + u2 u2 + · · · + up up y


T T = u1 uT 1 y + u2 u2 y + · · · + up up y


T T = u1 uT 1 y + u2 u2 y + · · · + up up y = (y · u1 ) u1 + (y · u2 ) u2 + · · · + (y · up ) up
⇒ UU T y = (y · u1 ) u1 + (y · u2 ) u2 + · · · + (y · up ) up Math 2331, Linear Algebra 15 / 16 6.3 Orthogonal Projections Orthogonal Projection Decomposition Best Approximation Orthogonal Projection: Theorem Theorem (10) If {u1 , . . . , up } is an orthonormal basis for a subspace W of Rn , then
projW y = (y · u1 ) u1 + · · · + y · up up If U =  u1 u2 · · ·  up , then projW y =UU T y for all y in Rn . Outline of Proof: projW y =  y·u1 u1 ·u1  u1 + · · · +  y·up up ·up  up
= (y · u1 ) u1 + · · · + y · up up = UU T y. Math 2331, Linear Algebra 16 / 16