Lecture 26: The Projection Formula

Lecture 26: The Projection Formula 26 26.1 The Projection Formula Review: Symmetric and Hermitian Forms Last time, we were talking about diferent kinds of pairings or bilinear forms on vector spaces. In particular, we will be studying two cases in parallel: vector spaces V over R with symmetric forms on them, and vector spaces over C, with Hermitian forms on them. A Hermitian form is almost symmetric, with a complex conjugate thrown in. Then, we discussed the idea of vectors being orthogonal to each other with respect to the form if the pairing is zero. A form is non-degenerate if and only if the space of vectors orthogonal to the entire vector space V is {0}, so there are no nonzero vectors orthogonal to all other vectors. Such a vector lies in the kernel of the matrix of the form, so a matrix with nonzero determinant will correspond to a non-degenerate form. 26.2 Orthogonality Recall this theorem about the restriction of a bilinear form to a subspace. We’ll prove it now. Theorem 26.1 Let W ⊆ V. If ⟨·, ·⟩|W is non-degenerate on W, then V = W ⊕ W ⊥ , which means that every vector v ∈ V #» + #» is equal to w u uniquely, where w ∈ W, u ∈ W ⊥ . It is possible for the restriction of a non-degenerate form to be degenerate; for example the form A =   0 1 1 0 is non-degenerate but is just given by A′ = 0 when W = Span(⃗e1 ), which is clearly degenerate. Proof. If ⟨·, ·⟩|W is non-degenerate, then W ∩ W ⊥ = {0}. We have W ⊕ W ⊥ ⊂ V, so it sufces to show that V ⊂ W ⊕ W ⊥ . Pick a basis of W, {w1 , . . . , wk }, and defne a linear transformation φ : V −→ Ck ⃗v 7−→ (⟨w1 , v, ⟩, . . . , ⟨wk , v⟩). This is a linear transformation just by the properties of a Hermitian form. The kernel is ker(φ) = W ⊥ , since W = Span{w ⃗i }. Also, dim im φ ≤ k = dim W, so by the dimension formula, dim V = dim ker φ + dim im φ ≤ dim W ⊥ + dim W. Consider the mapping W ⊕ W ⊥ −→ V (w, u) 7−→ w + u. It has kernel {0}, since W ∩ W ⊥ = {0}, so dim W + dim W ⊥ ≤ dim V, and thus dim W + dim W ⊥ = dim V and therefore V = W ⊕ W ⊥ . To emphasize, the geometric version of this with respect to the dot product feels obvious and works in most cases. For general forms, we have to have this condition that our form is non-degenerate on the subspace. The splitting V = W ⊕ W ⊥ is helpful, in particular, for inductive arguments, because it is possible to reduce some property of V to being true on W and W ⊥ . Lecture 26: The Projection Formula 26.3 Orthogonal Bases By applying a change of basis, it is always possible to put an arbitrary matrix into Jordan normal form, and if there are distinct eigenvalues, it is in fact possible to diagonalize it. What about the matrix of a bilinear form? Guiding Question Given a vector space V and a bilinear form ⟨·, ·⟩, how simple can we get the form to be? First, it is always possible to fnd a basis orthogonal with respect to the bilinear form. Theorem 26.2 For a symmetric or Hermitian form ⟨·, ·⟩, the vector space V has an orthogonal basis {v1 , · · · , vn }, which is when ⟨vi , vj ⟩ = 0 for i ̸= j. The matrix for the pairing in the basis will then be diagonal, since it is given by the inner product from the form. Proof. To prove this, induct on dim V = n. • Case 1. There is some u such that ⟨u, u⟩ ̸= 0. Then, the one-dimensional subspace W = Span(u), ⟨·, ·⟩|W is non-degenerate. By induction, W ⊥ has an orthogonal basis {v2 , · · · , vn }, so {u, v2 , · · · , vn } is an orthogonal basis for V. • Case 2. Otherwise, for every v ∈ V, ⟨v, v⟩ = 0. This is a very strong constraint on the form, and in fact it forces ⟨v, w⟩ = 0 for all v, w, which forces any basis to be an orthogonal basis. To see this, consider the inner product on a sum of two vectors with itself: 0 = ⟨v + w, v + w⟩ = ⟨v, v⟩ + ⟨w, w⟩ + ⟨v, w⟩ + ⟨w, v⟩ = 2⟨v, w⟩. When F = R, we have ⟨v, w⟩ = 0, by the symmetry of the form. Otherwise, for F = C, Re(⟨v, w⟩) = 0, and the same process can also be done for v and iw to show that ⟨v, w⟩ = 0. Then the inner product is 0 on any two vectors so every basis is orthogonal. We can simplify the basis even further. Corollary 26.3 In fact, V has an orthogonal basis {v1 , · · · , vk } where ⟨vi , vi ⟩ = 1, −1, or 0. Proof. Take an orthogonal basis {x1 , · · · , xk }. Consider ⟨xi , xi ⟩, which is a real number. • If the pairing is 0, then let vi = xi . • Otherwise, we can normalize and take vi = √ depending on the sign of ⟨xi , xi ⟩. 1 xi ; |⟨xi ,xi ⟩| then ⟨vi , vi ⟩ = ⟨xi ,xi ⟩ |⟨xi ,xi ⟩| , so it will be 1 or -1 Lecture 26: The Projection Formula In particular, if ⟨·, ·⟩ is non-degenerate, only ±1 occur. Also, if ⟨·, ·⟩ is positive defnite, by defnition, ⟨v, v⟩ > 0 if v ̸= 0, so only +1s occur, so in that basis, the form looks just like the dot product or the standard Hermitian product. The following claim will be shown in the upcoming problem set. Claim 26.4 (Sylvester’s Law). In fact, given V and ⟨·, ·⟩, the number of 1s, the number of -1s, and the number of 0s that occur in the diagonal form are determined by V and ⟨·, ·⟩, and not by the choice of orthogonal basis.92 This is called Sylvester’s Law, and the number of 1s, -1s, and 0s is called the signature of the form.   −1   1 , the signature is (3, 1, 0). For example, in the form used in special relativity,   1  1 In matrix form, the corollary states that for a symmetric matrix A ∈ Matn×n (R), for which AT = A, there exists some matrix P ∈ GLn (R) such that P T AP is a diagonal matrix with all 1s, −1s, or 0s on the diagonal:   1   ..   .     −1   P T AP =  . . ..       0   .. . If A is positive defnite, which is when xT Ax > 0, there exists P such that P T AP = In implies that A = QT Q, where Q = P −1 . The statement is similar for complex matrices, where we replace the transposes with adjoints. 26.4 Projection Formula Consider a vector space V and a form ⟨·, ·⟩, as well as a subspace W for which ⟨·, ·⟩|W is non-degenerate. By Theorem 26.1, V = W ⊕ W ⊥ such that v = w + u. Guiding Question How can we compute w and u? To do so, we use the orthogonal projection. We want a map π : V −→ W v 7−→ w, so that v = π(v) ⊥ W.93 92 They are similar to eigenvalues in that while there are many choices of orthogonal basis, the number of 1s, -1s, and 0s are not dependent on the particular basis. 93 This is an extremely useful application of linear algebra! In geometric situations, the vector w is the vector closest to v of the vector in the plane, and perhaps these vectors are in a vector space of data points. Finding a formula for w explicitly is called least-squares regression. Lecture 26: The Projection Formula Assuming there exists an orthogonal basis {w1 , · · · , wk } for W , the formula for π is simple.94 The vector can be written as v = cw1 + · · · + cwk + u, where u ⊥ W. Then for all i, ⟨wi , v⟩ = 0 + · · · + 0 + ci ⟨wi , wi ⟩ + 0 + · · · + 0, so ci = ⟨wi , v⟩ . ⟨wi , wi ⟩ It is not possible for ⟨wi , wi ⟩ = 0, because the form would be degenerate. In fact, this formula is useful when W = V , because it provides a formula for the coordinates of some vector with respect to the orthogonal basis. Example 26.5 Let V = R3 and ⟨·, ·⟩ be the dot product. Then let W be the span of w ⃗ 1 = (1, 1, 1)T and w ⃗ 1 = (1, 1, −2)T . The pairings are ⟨w1 , w1 ⟩ = 3, ⟨w2 , w2 ⟩ = 6, ⟨w1 , v⟩ = 6, and ⟨w2 , v⟩ = −3. The projection of (1, 2, 3) is then   3/2 6 1 π(v) = w1 − w2 = 3/2 . 3 2 3 To verify, v − π(v) = (−1/2, 1/2, 0), which is orthogonal both to w1 and w2 . 94 Once we’ve developed the machinery for bilinear forms, these ideas become a lot simpler!