Lecture 6 Notes: Photometric Stereo, Noise Gain, Error Amplification, Eigenvalues and Eigenvectors Review

6.801/6.866: Machine Vision, Lecture 6 1 Lecture 6: Photometric Stereo, Noise Gain, Error Amplification, Eigenvalues and Eigenvectors review This lecture covers an overview of some relevant concepts from linear algebra. If you would like a refresher on some of these concepts, please check out Professor Horn’s posted linear algebra review here. 1.1 Applications of Linear Algebra to Motion Estimation/Noise Gain Recall our problem of noise gain, in that if we observe a noisy estimate of y and we’re asked to derive x, the noise from our observed value can greatly affect our estimate of x. We’ve analyzed this problem in 1D, and now we’re ready to generalize this to higher dimensions using eigenvalues and eigenvectors. For the sake of brevity, the notes from the linear algebra review will not be given here, but they are in the handwritten notes for this lecture, as well as in the linear algebra handout posted above. These notes cover: • Definitions of eigenvalues/eigenvectors • Characteristic Polynomials • Rewriting vectors using an eigenbasis 1.1.1 Application Revisited: Photometric Stereo Let us now return to an application where we determine shape from shading: photometric stereo. Let us first consider the simple case where the surface we seek to reconstruct is Lambertian: E ∝ cos θi = ρn̂ · ŝi Recall the following, to help frame the setup of this problem: • For shape from shading problems, our goal is to estimate and reconstruct a 3D scene from a 2D image given information about the shading and surface orientation of the scene. • In 2D, this shape from shading problem has 2 Degrees of Freedom (abbreviated in these course notes as DOF). • Recall that we can use albedo (ρ) to actually make this problem easier - can create a problem with 3 unknowns and 3 equations. Without the use of albedo, recall that we had a quadratic constraint we had to follow, which, by Bezout’s theorem, suggests that there are at two solutions to such a system of equations. In this case, we have 3 linear equations and 3 unknowns, so by Bezout’s theorem we have only one solution, as desired. Recall that for such a system, suppose we have three brightness measurements of the form: 1. E1 = ρn̂ · ŝ1 - Intuitively, look at pixel with light source located at s1 and take measurement E1 . 2. E2 = ρn̂ · ŝ2 - Intuitively, look at pixel with light source located at s2 and take measurement E2 . 1 3. E3 = ρn̂ · ŝ3 - Intuitively, look at pixel with light source located at s3 and take measurement E3 . Let’s review the linear algebraic system of equations by combining the equations above into a matrix-vector product. ⎡ ⎤ ⎡ ⎤ −s1T E1 ⎢ T⎥ ⎣−s2 ⎦ n = ⎣E2 ⎦ E3 −sT3 Note the following: • Since the matrix on the left-hand side corresponds to the three-dimensional position of the light source in the given frame of reference, we have: ⎡ ⎤ ⎡ ⎤ −s1T E1 ⎢ T⎥ 3×3 3 ⎣−s2 ⎦ ∈ R , ⎣E2 ⎦ ∈ R T E3 −s3 • n = ρn̂, i.e. we do not need to deal with the second order constraint n̂T n̂ = 1. This eliminates the second-order constraint from our set of equations and ensures we are able to derive a unique solutions by solving a system of only linear equations. ⎡ ⎤ −sT1 Δ ⎢ ⎥ • Define S = ⎣−sT2 ⎦ −sT3 Like many other linear system of equations we encounter of the form Ax = b, we typically want to solve for x. In this case, we want to solve for n, which provides information about the surface orientation of our object of interest: Sn = E −→ n = S−1 E Like the other “inverse” problems we have solved so far in this class, we need to determine when this problem is ill-posed, i.e. when S is not invertible. Recall from linear algebra that this occurs when the columns of S are not linearly independent/the rank of S is not full. An example of when this problem is ill-posed is when the light source vectors are coplanar with one another, i.e. s1 + s2 + s3 = 0 (Note that this can be verified by simply computing the vector sum of any three coplanar vectors.) Therefore, for optimal performance and to guarantee that S is invertible: • We make the vectors from the light sources to the objects (si ) as non-coplanar as possible. • The best configuration for this problem is to have the vectors from the light sources to the surface be orthogonal to one another. Intuitively, consider that this configuration captures the most variation in the angle between a given surface normal and three light sources. Example: Determining Depth of Craters on the Moon: As it turns out, we cannot simply image the moon directly, since the plane between the earth/moon and the sun makes it such that all our vectors si will lie in the same plane/are coplanar. Thus, we cannot observe the moon’s tomography from the surface of the earth. 1.2 Lambertian Objects and Brightness Fun Fact: “Lambert’s Law” was derived from a monk observing light passing through an oil spot in a piece of paper. “Lambert’s Law”: Ei ∝ cos θi = n̂ · ŝi To better understand this law, let us talk more about surface orientation. 2 1.2.1 Surface Orientation • For an extended surface, we can take small patches of the surface that are “approximately planar”, which results in us constraining the surface to be a 2-manifold [1] (technically, the language here is “locally planar”). • Using these “approximately planar” patches of the surface results in many unit normal vectors n̂i pointing in a variety of different directions for different patches in the surface. • We can also understand surface orientation from a calculus/Taylor Series point of view. Consider z(x + δx, y + δy), i.e. an estimate for the surface height at (x, y) perturbed slightly by a small value δ. The Taylor Series expansion of this is given by: z(x + δx, y + δy) = z(x, y) + δx Δ ∂z Δ ∂z ∂x , q = ∂y ) • The surface orientation (p = ∂z ∂z + δy + · · · = z(x, y) + pδx + qδy ∂x ∂y Δ ∂z Δ ∂z Where p = ,q = ∂x ∂y captures the gradient of the height of the surface z. • Note that the surface unit normal n̂ is perpendicular to any two lines in the surface, e.g. tangent lines. We can use the cross product of the two following tangent lines of this surface to compute the the unit surface normal, which describes the orientation of the surface: 1. (δx, 0, pδx)T = δx(1, 0, p)T 2. (0, δy, qδx)T = δy(0, 1, q)T Since we seek to find the unit surface normal, we can remove the scaling factors out front (δx and δy). Then the cross product of these two vectors is: ⎡ ⎤ ⎡ ⎤ x̂ ŷ ẑ −p (1, 0, p)T × (0, 1, p)T = det ⎣ 1 0 p⎦ = n = ⎣−q ⎦ 0 1 q 1 We can now compute the unit surface normal by normalizing the vector n:  T −p −q 1 n n̂ = = p ||n||2 p2 + q 2 + 1 With this surface normal computed, we can also go the opposite way to extract the individual components of the surface orientation (p and q): p= −n · x̂ −n · ŷ , q= n · ẑ n · ẑ Note: Do the above equations look similar to something we have encountered before? Do they look at all like the coordinate-wise perspective projection equations? We will leverage these (p, q)-space plots a lot, using reflectance maps (which we will discuss later). Now, instead of plotting in velocity space, we are plotting in surface orientation/spatial derivative space. Individual points in this plane correspond to different surface orientations of the surface. 1.2.2 Surface Orientation Isophotes/Reflectance Maps for Lambertian Surfaces Recall the equation to derive isophotes of a Lambertian surface (since the brightness depends on the incident angle between the surface and the light source): n̂ · ŝ = E1 Where E1 is a constant, scalar level of brightness If we expand our light source vector ŝ, we get the following for this dot product: T  T  −p −q 1 −ps −qs 1 n̂ · ŝ = p · p = E1 p2s + qs2 + 1 p2 + q 2 + 1 3 Carrying out this dot product and squaring each side, we can derive a parametric form of these isophotes in (p, q) space (note that the light source orientation (ps , qs ) is fixed (at least for a single measurement), and thus we can treat it as constant: (1 + ps p + qs q)2 = c2 (1 + p2 + q 2 )(1 + ps2 + qs2 ) If we plot these isophotes in (p, q) space, we will see they become conic sections. Different isophotes will generate different curves. When we have multiple light sources, we can have intersections between these isophotes, which indicate solutions. 1.3 References [1] Manifolds, https://en.wikipedia.org/wiki/Manifold. 4