Lecture 7 Notes: Gradient Space, Reflectance Map, Image Irradiance Equation, Gnomonic Projection

6.801/6.866: Machine Vision, Lecture 7 1 Lecture 7: Gradient space, Reflectance Map, Image Irradiance Equation, Gnomonic Projection This lecture continues our treatment of the photometric stereo problem, and how we can estimate motion graphically by finding intersections of curves in (p, q) (spatial derivative) space. Following this, we discuss concepts from photometry and introduce the use of lenses. 1.1 Surface Orientation Estimation (Cont.) & Reflectance Maps Let’s review a couple of concepts from the previous lecture: ∂z ∂z • Recall our spatial gradient space given by (p, q) = ( ∂x , ∂y ). • We also discussed Lambertian surfaces, or surfaces in which the viewing angle doesn’t change brightness, only the angle between the light source and the object affects the brightness. Additionally, the relation between the source/object angle is proportional to: E ∝ cos θi , where θi is the angle between the light source and the object. • Recall our derivation of the isophotes of Lambertian surfaces: T  T  −p −q 1 −ps −qs 1 n̂ · ŝ = p · p =c p2s + qs2 + 1 p2 + q 2 + 1 Where c is a constant brightness level. Carrying out this dot product: p p2 1 + p s p + qs q p =c + q 2 + 1 p2s + qs2 + 1 We can generate plots of these isophotes in surface orientation/(p, q) space. This is helpful in computer graphics, for instance, because it allows us to find not only surface heights, but also the local spatial orientation of the surface. In practice, we can use lookup tables for these reflectance maps to find different orientations based off of a measured brightness. 1.1.1 Forward and Inverse Problems As discussed previously, there are two problems of interest that relate surface orientation to brightness measurements: 1. Forward Problem: (p, q) → E (Brightness from surface orientation) 2. Inverse Problem: E → (p, q) (Surface orientation from brightness) In most settings, the second problem is of stronger interest, namely because it is oftentimes much easier to measure brightness than to directly measure surface orientation. To measure the surface orientation, we need many pieces of information about the local geometries of the solid, at perhaps many scales, and in practice this is hard to implement. But it is straightforward to simply find different ways to illuminate the object! A few notes about solving this inverse problem: 1 • We can use the reflectance maps to find intersections (either curves or points) between isophotes from different light source orientations. The orientations are therefore defined by the interesections of these isophote curves. • Intuitively, the reflectance map answers the question “How bright will a given orientation be?” • We can use an automated lookup table to help us solve this inverse problem, but depending on the discretization level of the reflectance map, this can become tedious. Instead, wwe can use a calibration object, such as a sphere, instead. • For a given pixel, we can take pictures with different lighting conditions and repeatedly take images to infer the surface orientation from the sets of brightness measurements. • This problem is known as photometric stereo: Photometric stereo is a technique in computer vision for estimating the surface normals of objects by observing that object under different lighting conditions [1]. 1.1.2 Reflectance Map Example: Determining the Surface Normals of a Sphere A few terms to set before we solve for p and q: • Center of the sphere being imaged: (x0 , y0 , z0 )T • Location of the light source: (x, y, z)T Then: • z − z0 = • p= x−x0 z−z0 • a= y−y0 z−z0 p r02 − (x − x0 )2 + (y − y0 )2 (Do the above equations carry a form similar to that of perspective projection?) 1.1.3 Computational Photometric Stereo Though the intersection of curves approach we used above can be used for finding solutions to surfaces with well-defined reflectance maps, in general we seek to leverage approaches that are as general and robust as possible. We can accomplish this, for instance, by using a 3D reflectance map in (E1 , E2 , E3 ) space, where each Ei is a brightness measurement. Each discretized (unit of graphic information that defines a point in three-dimensional space [2]) contains a specific surface orientation parameterized by (p, q). This approach allows us to obtain surface orientation estimates by using brightness measurements along with a lookup table (what is described above). While this approach has many advantages, it also has some drawbacks that we need to be mindful of: • Discretization levels often need to be made coarse, since the number of (p, q) voxel cells scales as a cubic in 3D, i.e. f (d) ∈ Ospace (d3 ), where d is the discretization level. • Not all voxels may be filled in - i.e. mapping (let us denote this as φ) from 3D to 2D means that some kind of surface will be created, and therefore the (E1 , E2 , E3 ) space may be sparse. One way we can solve this latter problem is to reincorporate albedo ρ! Recall that we leveraged albedo to transform a system with a quadratic constraint into a system of all linear equations. Now, we have a mapping from 3D space to 3D space: φ : (E1 , E2 , E3 ) −→ (p, q, ρ) i.e. φ : R3 −→ R3 It is worth noting, however, that even with this approach, that there will still exist brightness triples (E1 , E2 , E3 ) that do not map to any (p, q, ρ) triples. This could happen, for instance, when we have extreme brightness from measurement noise. Alternatively, an easy way to ensure that this map φ is injective is to use the reverse map (we can denote this by ψ): ψ : (p, q, ρ) −→ (E1 , E2 , E3 ) i.e. ψ : R3 −→ R3 2 1.2 Photometry & Radiometry Next, we will rigorously cover some concepts in photometry and radiommetry. Let us begin by rigorously defining these two fields: • Photometry: The science of the measurement of light, in terms of its perceived brightness to the human eye [3]. • Radiometry: The science of measurement of radiant energy (including light) in terms of absolute power [3]. Next, we will cover some concepts that form the cornerstone of these fields: • Irradiance: This quantity concerns with power emitted/reflected by the object when light from a light source is incident upon it. Note that for many objects/mediums, the radiation reflection is not isotropic - i.e. it is not emitted equally in different directions. Irradiance is given by: E = δP δA (Power/unit area, W/m2 ) • Intensity: Intensity can be thought of as the power per unit angle. In this case, we make use of solid angles, which have units of Steradians. Solid angles are a measure of the amount of the field of view from some particular point that a given object covers [4]. We can compute a solid angle Ω using: Ω = Asurface R2 . Intensity is given by: Intensity = δP δΩ . Radiance: Oftentimes, we are only interested in the power reflected from an object that actually reaches the light sensor. For this, we have radiance, we contract with irradiance defined above. We can use radiance and irradiance to contrast the “brightness” of a scene and the “brightness” of the scene perceived in the image plane. Radiance is given by: L = δ2 P δAδΩ These quantities provide motivation for our next section: lenses. Until now, we have considered only pinhole cameras, which are infinitisimally small, but to achieve nonzero irradiance in the image plane, we need to have lenses of finite size. 1.3 Lenses Lenses are used to achieve the same photometric properties of perspective projection as pinhole cameras, but do so using a finite “pinhole” size and thus by encountering a finite number of photons from light sources in scenes. One caveat with lenses that we will cover in greater detail: lenses only work for a finite focal length f . From Gauss, we have that it is impossible to make a perfect lens, but we can come close. 1.3.1 Thin Lenses - Introduction and Rules Thin lenses are the first type of lens we consider. These are often made from glass spheres, and obey the following three rules: • Central rays (rays that pass through the center of the lens) are undeflected - this allows us to preserve perspective projection as we had for pinhole cameras. • The ray from the focal center emerges parallel to the optical axis. • Any parallel rays go through the focal center. Thin lenses also obey the following law, which relates the focal centers (a and b) to the focal length of the lens. 1 1 1 + = a b f0 Thin lenses can also be thought of as analog computers, in that they enforce the aforementioned rules for all light rays, much like a computer executes a specific program with outlined rules and conditions. Here are some tradeoffs/issues to be aware of for thin lenses: • Radial distortion: In order to bring the entire angle into an image (e.g. for wide-angle lenses), we have the “squash” the edges of the solid angle, thus leading to distortion that is radially-dependent. Typically, other lens defects are mitigated at the cost of increased radial distortion. Some specific kinds of radial distortion [5]: – Barrel distortion – Mustache distortion – Pincushion distortion • Lens Defects: These occur frequently when manufacturing lenses, and can originate from a multitude of different issues. 3 1.3.2 Putting it All Together: Image Irradiance from Object Irradiance Here we show how we can relate object irradiance (amount of radiation emitted from an object) to image plane irradiance of that object (the amount of radiation emitted from an object that is incident on an image plane). To understand how these two quantities relate to each other, it is best to do so by thinking of this as a ratio of “Power in / Power out”. We can compute this ratio by matching solid angles: δI cos α δO cos θ δI cos α δO δO cos α  z 2 = =⇒ = =⇒ = (f sec α)2 (z sec α)2 f2 z2 δI cos θ f The equality on the far right-hand side can be interpreted as a unit ratio of “power out” to “power in”. Next, we’ll compute the subtended solid angle Ω: Ω= π 2 4d cos α π  d 2 = cos3 α 4 z (z sec α)2 Next, we will look at a unit change of power (δP ): δP = LδOΩ cos θ = LδO π d 2 ) cos3 α cos θ 4 z We can relate this quantity to the irradiance brightness in the image: δP δO π  d 2 =L cos3 α cos θ δI δI 4 z π  d 2 = L cos4 α 4 f E= A few notes about this above expression: • The quantity on the left, E is the irradiance brightness in the image. • The quantity L on the righthand side is the radiance brightness of the world. • The ratio fd is the inverse of the so-called “f-stop”, which describes how open the aperture is. Since these terms are squared, √ they typically come in multiples of 2. • The reason why we can be sloppy about the word brightness (i.e. radiance vs. irradiance) is because the two quantities are proportional to one another: E ∝ L. • When does the cos4 α term matter in the above equation? This term becomes non-negligible when we go off-axis from the optical axis - e.g. when we have a wide-angle axis. Part of the magic of DSLR lenses is to compensate for this effect. • Radiance in the world is determined by illumination, orientation, and reflectance. 1.3.3 BRDF: Bidirectional Reflectance Distribution Function The BRDF can be interpreted intuitively as a ratio of “Energy In” divided by “Energy Out” for a specific slice of incident and emitted angles. It depends on: • Incident and emitted azimuth angles, given by φi , φe , respectively. • Incident and emitted colatitude/polar angles, given by θi , θe , respectively. Mathematically, the BRDF is given by: f (θi , θe , φi , φe ) = δL(θe , φe ) “Energy In” = “Energy Out” δE(θi , φi ) Let’s think about this ratio for a second. Please take note of the following as a mnemonic to remember which angles go where: • The emitted angles (between the object and the camera/viewer) are parameters for radiance (L), which makes sense because radiance is measured in the image plane. • The incident angles (between the light source and the object) are parameters for irradiance (E), which makes sense because irradiance is measured in the scene with the object. 4 Practically, how do we measure this? • Put light source and camera in different positions • Note that we are exploring a 4D space comprised of [θi , θe , φi , φe ], so search/discretization routines/sub-routines will be computationally-expensive. • This procedure is carried out via goniometers (angle measurement devices). Automation can be carried out by providing these devices with mechanical trajectories and the ability to measure brightness. But we can also use multiple light sources! • Is the space of BRDF always in 4D? For most surfaces, what really matters are the difference in emitted and incident azimuth angles (φe − φi ). This is applicable for materials in which you can rotate the surface in the plane normal to the surface normal without changing the surface normal. • But this aforementioned dimensionality reduction procedure cannot always be done - materials for which there is directionallydependent microstructures, e.g. hummingbird’s neck, semi-precious stones, etc. 1.3.4 Helmholtz Reciprocity Somewhat formally, the Helmholtz reciprocity principle describes how a ray of light and its reverse ray encounter matched optical adventures, such as reflections, refractions, and absorptions in a passive medium, or at an interface [6]. It is a property of BRDF functions which mathematically states the following: f (θi , θe , φi , φe ) = f (θe , θi , φe , φi ) ∀ θi , θe , φi , φe In other words, if the ray were reversed, and the incident and emitted angles were consequently switched, we would obtain the same BRDF value. A few concluding notes/comments on this topic: • If there is not Helmholtz reciprocity/symmetry between a ray and its inverse in a BRDF, then there must be energy transfer. This is consistent with the 2nd Law of Thermodynamics. • Helmholtz reciprocity has good computational implications - since you have symmetry in your (possibly) 4D table across incident and emitted angles, you only need to collect and populate half of the entries in this table. Effectively, this is a tensor that is symmetric across some of its axes. 2 References 1. Photometric Stereo, https://en.wikipedia.org/wiki/Photometric stereo 2. Voxels, https://whatis.techtarget.com/definition/voxel 3. Photometry, https://en.wikipedia.org/wiki/Photometry (optics) 4. Solid Angles, https://en.wikipedia.org/wiki/Solid angle 5. Distortion, https://en.wikipedia.org/wiki/Distortion (optics) 6. Helmholtz Reciprocity, https://en.wikipedia.org/wiki/Helmholtz reciprocity 5