Lecture 8 Notes: Shape from Shading, Special Cases, Lunar Surface, Scanning Electron Microscope, Green’s Theorem in Photometric Stereo

6.801/6.866: Machine Vision, Lecture 8 1 Lecture 8: Shape from Shading, Special Cases, Lunar Surface, Scanning Electron Microscope, Green’s Theorem in Photometric Stereo In this lecture, we will continue our discussion of photometry/radiometry, beginning with the concepts covered last lecture. We will then explore Hapke surfaces and how they compare to Lambertian surfaces, and show how they can be used to solve our shape from shading problem. Finally, we will introduce two new types of lenses and some of their applications: “thick” lenses and telecentric lenses. 1.1 Review of Photometric and Radiometric Concepts Let us begin by reviewing a few key concepts from photometry and radiometry: • Photometry: Photometry is the science of measuring visible radiation, light, in units that are weighted according to the sensitivity of the human eye. It is a quantitative science based on a statistical model of the human visual perception of light (eye sensitivity curve) under carefully controlled conditions [3]. • Radiometry: Radiometry is the science of measuring radiation energy in any portion of the electromagnetic spectrum. In practice, the term is usually limited to the measurement of ultraviolet (UV), visible (VIS), and infrared (IR) radiation using optical instruments [3]. Δ 2 • Irradiance: E = δP δA (W/m ). This corresponds to light falling on a surface. When imaging an object, irradiance is converted to a grey level. Δ δP δW • Intensity: I = Δ • Radiance: L = (W/ster). This quantity applied to a point source and is often directionally-dependent. δ2 P δAδΩ (W/m2 × ster). This photometric quantity is a measure of how bright a surface appears in an image. e ,φe ) • BRDF (Bidirectional Reflectance Distribution): f (θi , θe , φi , φe ) = δL(θ δE(θi ,φi ) . This function captures the fact that oftentimes, we are only interested in light hitting the camera, as opposed to the total amount of light emitted from an object. Last time, we had the following equation to relate image irradiance with object/surface radiance: E= π  d 2 L cos4 α 4 f Where the irradiance of the image E is on the lefthand side and the radiance of the object/scene L is on the right. The BRDF must also satisfy Helmholtz reciprocity, otherwise we would be violating the 2nd Law of Thermodynamics. Mathematically, recall that Helmholtz reciprocity is given by: f (θi , θe , φi , φe ) = f (θe , θi , φe , φi ) ∀ θi , θe , φi , φe With this, we are now ready to discuss our first type of surfaces: ideal Lambertian surfaces. 1 1.2 Ideal Lambertian Surfaces Ideal Lambertian surfaces are of interest because their sensed brightness (irradiance) only depends on the cosine of the incident angles (θi , φi ). A few additional notes about these surfaces: • Ideal Lambertian surfaces are equally bright from all directions, i.e. f (θi , θe , φi , φe ) = f (θe , θi , φe , φi ) ∀ θi , θe , φi , φe AND f (θi , θe , φi , φe ) = K ∈ R with respect to θe , φe , i.e. ∂f (θi , θe , φi , φe ) ∂f (θi , θe , φi , φe ) = =0 ∂φe ∂θe • If the surface is ideal, the Lambertian surface reflects all incident light. We can use this to compute f . Suppose we take a small slice of the light reflected by a Lambertian surface, i.e. a hemisphere for all positive z. The area of this surface is given by: sin θe δθe δφe , which is the determinant of the coordinate transformation from euclidean to spherical coordinates. Then, we have that: Z π Z π2 (f E cos θi sin θe dθe )dφe = E cos θi (Integrate all reflected light) −π 0 Z π π 2 Z E cos θi (f sin θe dθe )dφe = E cos θi (Factor out E cos θi ) −π Z π 0 π 2 Z (f sin θe dθe )dφe = 1 (Cancel out on both sides) −π 0 π 2 Z 2π f sin θe cos θe dθe = 1 (No dependence on φ) 0 π 2 Z πf 2 sin θe cos θe dθe = 1 (Rearranging terms) 0 π 2 Z πf sin(2θe )dθe = 1 (Using identity 2 sin θ cos θ = sin(2θ)) 0 1 θ =π πf [− cos(2θe )]θee =02 = 1 2 1 πf = 1 =⇒ f = π 1.2.1 Foreshortening Effects in Lambertian Surfaces Now let us discuss a key issue in radiation reflectance and machine vision - foreshortening. Foreshortening is the visual effect or optical illusion that causes an object or distance to appear shorter than it actually is because it is angled toward the viewer [1]. Let us show how this is relevant for computing f . Suppose we have a point source radiating isotropically over the positive hemisphere in a 3-dimensional spherical coordinate system. Then, the solid angle spanned by this hemisphere is: Ω= 2πr2 Asurface = = 2π Steradians r2 r2 1 If the point source is radiating isotropically in all directions, then f = 2π . But we saw above that f = π1 for an ideal Lambertian surface. Why do we have this discrepancy? Even if brightness is the same/radiation emitted is the same in all directions, this does not mean that the power radiated in all directions is hte same. This is due to foreshortening, since the angle between the object’s surface normal and the camera/viewer changes. 1.2.2 Example: Distant Lambertian Point Source Here, we will have that L = f1 Es (where s is the location of the light source). Since we have to take foreshortening into account, the perceived area A of an object with world size A0 is given by: A = A0 cos θi Therefore, our expression for the the radiance (how bright the object appears in the image) is given by: L= 1 1 Es = E0 cos θi π π 2 1.3 Hapke Surfaces Hapke surfaces are another type of surface we study in this course. Formally, a Hapke surface is an object which has reflectance properties corresponding to Hapke parameters, which in turn are a set of parameters for an empirical model that is commonly used to describe the directional reflectance properties of the airless regolith surfaces of bodies in the solar system [2]. The BRDF of a Hapke surface is given by: 1 f (θi , φi , θe , φe ) = √ cos θe cos θi We can see that the Hapke BRDF also satisfies Helmholtz Reciprocity: 1 1 =√ = f (θe , φe , θi , φi ) f (θi , φi , θe , φe ) = √ cos θe cos θi cos θi cos θe Using the Hapke BRDF, we can use this to compute illumination (same as radiance, which is given by: L = E0 cos θi f (θi , φi , θe , φe ) 1 = E0 cos θi √ cos θe cos θi r cos θi = E0 cos θe Where L is the radiance/illumination of the object in the image, E0 cos θi is the irradiance of the object, accounting for foreshortening effects, and √cos θ1 cos θ is the BRDF of our Hapke surface. i 1.3.1 e Example Application: Imaging the Lunar Surface What do the isophotes of the moon look like, supposing the moon can fall under some of the different types of surfaces we have discussed? • Lambertian: We will see circles and ellipses of isophotes, depending on the angle made between the viewer and the the moon. • Hapke: Because of the BRDF behavior, isophotes will run longitudinally along the moon in the case in which it is a Hapke surface. Recall isophotes are where the brightness of the object in the image, given by: v u r u ||n̂||n̂·ŝ cos θi n̂ · ŝ 2 ||ŝ||2 = t n̂·v̂ = c =⇒ = c2 for some c ∈ R cos θe n̂ · v̂ ||n̂|| ||v̂|| 2 2 Where n̂ is the surface normal vector of the object being imaged, ŝ is the unit vector describing the position of the light source, and v̂ is the unit vector describing the position of the vertical. We can derive a relationship between n̂ and ŝ: n̂ · ŝ = cn̂ · v̂ =⇒ n̂ · (ŝ − cv̂) = 0 =⇒ n̂ ⊥ (ŝ − cv̂) Next, we will use spherical coordinates to show how the isophotes of this surface will be longitudinal: ⎡ ⎤ sin θ cos φ n̂ = ⎣ sin θ sin φ ⎦ ∈ R3 cos θ Since normal vectors n̂ ∈ R3 only have two degrees of freedom (since ||n̂||2 = 1), we can fully parameterize these two degrees of freedom by the azimuth and colatitude/polar angles φ and θ, respectively. Then we can express our coordinate system given by the orthogonal basis vectors n̂, ŝ, v̂ as:  n̂ = sin θ cos φ sin θ sin φ  T ŝ = cos θs sin θs 0  T v̂ = cos θv sin θv 0 3 cos θ T We can use our orthogonal derivations from above to now show our longitudinal isophotes: r cos θi n̂ · ŝ sin θ cos φ cos φs + sin θ sin φ sin φs cos(φ − φs ) = = = = c for some c ∈ R n̂ · v̂ sin θ cos φ cos φv + sin θ sin φ sin φv cos(φ − φv ) cos θe From the righthand side above, we can deduce that isophotes for Hapke surfaces correspond to points with the same azimuth having the same brightness, i.e. the isophotes of the imaged object (such as the moon) are along lines of constant longitude. 1.3.2 Surface Orientation and Reflectance Maps of Hapke Surfaces Next, we will take another look at our “shape from shading” problem, but this time using the surface normals of Hapke surfaces and relating this back to reflectance maps and our (p, q) space. From our previous derivations, we already have that the BRDF of a Hapke surface is nothing more than the square root of the dot product of the surface normal with the light source vector divided by the dot product of the surface normal with the vertical vector, i.e. r r cos θi n̂ · ŝ = cos θe n̂ · v̂ Next, if we rewrite our unit vectors as 3D vectors in (p, q) space, we have:  T −p −q 1 n̂ = p p2 + q 2 + 1  T −ps −qs 1 p ŝ = p2s + qs2 + 1  T v̂ = 0 0 1 Then, substituting these definitions, we have: n̂ · v̂ = p 1 1 + p2 + q 2 1 + ps p + q s q p n̂ · ŝ = p 1 + p2 + q 2 1 + ps2 + qs2 s p+qs q √ 1+p √ 1+p2 +q 2 1+ps2 +qs2 n̂ · ŝ = √ 12 2 n̂ · v̂ 1+p +q 1 + ps p + q s q =p 1 + ps2 + qs2 1 + p s P + qs q = Rs Δ p (Where Rs = 1 + p2s + qs2 .) Note that ps and qs are constants, since they only reflect the spatial derivatives of the light source. This ratio of dot products is therefore linear in (p, q) space! Then, what do the isophotes look like in gradient space? Recall that isophotes take the form: r n̂ · ŝ n̂ · ŝ 1 + ps P + qs q = c =⇒ = = c2 for some c ∈ R . n̂ · v̂ n̂ · v̂ Rs Therefore, isophotes of Hapke surfaces are linear in spatial gradient space! This is very useful for building reflectance maps because, as we will see shortly, a simple change of coordinate system in (p, q) space tells us a lot about the surface orientation in one of the two directions. For one thing, let us use our linear isophotes in gradient space to solve our photometric stereo problem, in this case with two measurements under different lighting conditions. Photometric stereo is substantially easier with Hapke surfaces than with Lambertian, because there is no ambiguity in where the solutions lie. Unlike Lambertian surfaces, because of the linearity in (p, q) space we are guaranteed by Bezout’s Theorem to have only one unique solution. 4 1.3.3 Rotated (p0 , q 0 ) Coordinate Systems for Brightness Measurements We can also derive/obtain a surface orientation in gradient space in a rotated coordinate system in gradient space, i.e. from (p, q) −→ (p0 , q 0 ). We can prove (see the synchronous lecture notes for this part of the course) that the transforming the points in (x, y) via a rotation matrix R is equivalent to rotating the gradient space (p, q) by the same matrix R. I.e.  0   x x R : (x, y) =⇒ (x0 , y 0 ), 0 = R y y  0   p p R : (p, q) =⇒ (p0 , q 0 ), 0 = R q q Where R is given according to:  cos α R= sin α  sinα ∈ SO(2) cos α (Where SO(2) is the Special Orthogonal Group [4].) Note that R−1 = RT since R is orthogonal and symmetric. By rotating our coordinate system from (p, q) −→ (p0 , q 0 ), we are able to uniquely specify p0 , but not q 0 , since our isophotes lie along a multitude/many q 0 values. Note that this rotation system is specified by having p0 lying along the gradient of our isophotes. Returning to our Lunar Surface application with Hapke surfaces, we can use this surface orientation estimation framework to take an iterative, incremental approach to get a profile of the lunar surface. This enables us to do depth profile shaping of the moon simply from brightness estimates! There are a few caveats to be mindful of for this methodology, however: • We can only recover absolute depth values provided we are given initial conditions (this makes sense, since we are effectively solving a differential equation in estimating the depth z from the surface gradient (p, q)T ). Even without initial conditions, however, we can still recover the general shape of profiling, simply without the absolute depth. • Additionally, we can get profiles for each cross-section of the moon/object we are imaging, but it is important to recognize that these profiles are effectively independent of one another, i.e. we do not recover any information about the relative surface orientation changes between/at the interfaces of different profiles we image. We can use heuristic-based approaches to get a topographic mapping estimate. We can then combine these stitched profiles together into a 3D surface. 1.4 “Thick” & Telecentric Lenses We will now shift gears somewhat to revisit lenses, namely “thick” and telecentric lenses. 1.4.1 “Thick” Lenses In practice, lenses are not thick, and are typically cascaded together to mitigate the specific aberrations and wavelength dependence of each individual lens. For thick lenses, we will leverage the following definitions: • Nodal/principal points: The center points of the outermost (ending and beginning lenses). • Principal planes: The distance from the object to the first imaging lens and the distance from the last imaging lens to the viewer or imaging plane. Abbreviated by b. • Focal length: This definition is the same as before, but in this case, the focal length is the distance from the first imaging lens to the last imaging lens along the optical axis (i.e. orthogonal to the radial unit vectors of each lens in the “thick” lens). From this, we have the formula: 1 1 1 + = a b f One “trick” we can do with this is achieve a long focal length f and a small field of view (FOV), i.e. as is done in a telephoto lens. 5 1.4.2 Telecentric Lenses These types of lenses are frequently used in machine vision, such as for circuit board inspection or barcode reading. Motivation for these types of lenses: When we have perspective projection, the degree of magnification depends on distance. How can we remove this unwanted distance-dependent magnification due to perspective projection? We can do so by effectively “moving” the Center of Projection (COP) to “−∞”. This requires using a lens with a lot of glass. A few notes about this: • To image with magnifying effects, the lens must be at least as wide in diameter as the width of the object. • a A double telecentric lens can be created by moving the other “modal point” (the imaging plane/viewer) to “+∞”. In this case, our magnifying term given by cos4 α goes to zero, because: lim COP→+∞ α=0 • We can generalize this idea even more with “lenselets”: These each concentrate light into a confined area, and can be used in an array fashion as we would with arrays of photodiodes. A useful term for these types of arrays is fill factor: The fill factor of an image sensor array is the ratio of a pixel’s light sensitive area to its total area [5]. Lenselet arrays can be useful because it helps us to avoid aliasing effects. We need to make sure that if we are sampling an object discretely, that we do not have extremely high-frequency signals without some low pass filtering. This sort of low-pass filtering can be achieved with using large pixels and averaging (a crude form of lowpass filtering). However, this averaging scheme does not work well when light comes in at off-90 degree angles. We want light to come into the sensors at near-90 degrees (DSLRs find a way to get a around this). • Recall that for an object space telecentric device, we no longer have a distance-based dependence. Effectively, we are taking perspective projection and making the focal length larger, resulting in approximately orthographic projection. Recall that for orthographic projection, projection becomes nearly independent in position, i.e. (x, y) in image space has a linear relationship with (X, Y ) in the world. This means that we can measure sizes of objects independently of how far away they are! Starting with perspective projection: x X y Y = , = f Z f Z Having |ΔZ| << |Z|, whereΔZ is the variation in Z of an object =⇒ x = Approximating 1.5 f = 1 =⇒ x = X, y = Y Z0 References 1. Foreshortening, https://en.wikipedia.org/wiki/Perspective (graphical) 2. Hapke Parameters, https://en.wikipedia.org/wiki/Hapke parameters 3. Understanding Radiance (Brightness), Irradiance and Radiant Flux 4. Orthogonal Group, https://en.wikipedia.org/wiki/Orthogonal group 5. https://en.wikipedia.org/wiki/Fill factor (image sensor) 6 f f X, y = Y Z0 Z0