Imaging Through Lensless Cameras

Imaging through Lensless Cameras Methods Introduction Lensless Imaging Cameras are an inexpensive, compact way to take images of 2D and 3D objects. By placing a thin translucent film in front of a light sensor, we can create patterns (caustics) on the sensor that can be used to reconstruct an image. These caustic patterns are linear shift invariant (LSI) which means linear shifts in the object will cause proportional shifts in the pattern. Using this property, we can calibrate the film (diffuser) with a single point source, resulting in a Point Spread Function (PSF).1 Then, using various methods, we can reconstruct the real image with the sensor data and PSF. Sensor data PSF Actual Image 𝑓 𝑥 =ℎ ∗ 𝑣 convolution Analysis We can use this equation to guide our optimization and direct inverse algorithms Iterative Optimization Method The underlying equation behind this reconstruction is 1 ′ 𝑣 = 𝑎𝑟𝑔𝑚𝑖𝑛𝑣 2 | 𝐴𝑣 − 𝑏 |22 where Av is the convolution of the PSF and trained “real image” + crop, b is the sensor data. Since the system is LSI, a convolution of the PSF with the “real image” will result in the sensor data so the argument v that minimizes the mean squared error of Av and b will be the real image.1 Iterative optimization algorithms such as Gradient Descent use the above equation as the “cost function” and take the derivative with respect to v, obtaining gradients that it can use to adjust v until it results in a quality result, therefore minimizing the error. Direct Inverse Method A convolution can be expressed as a Fourier transform which decomposes signals into individual parts. 𝐹 𝐻 ∗ 𝑉 = 𝐹 𝐻 ∗ 𝐹 𝑉 →𝐹 𝐻 ∗ 𝐹 𝑉 = 𝐹(𝐷) Image reconstruction can be achieved through iterative optimization algorithms such as Gradient Descent and ADMM and a direct inverse solution. In addition, we compared this model-based reconstruction to a data-driven approach using Deep Learning with Unet and DenseNet architecture to learn a mapping between sensor data and the resulting image. All images were taken using a Raspberry Pi camera and reconstructed using the above algorithms, adjusting hyper-parameters as needed. 𝑉=𝐹 −1 𝐹 𝐻 (𝐹 𝐷 +𝜀) V is then the resulting “real image” Deep Learning Model Sensor Deep Learning Direct Inverse Actual Image Direct Inverse Optimization Deep Learning Optimization Actual Image Direct Inverse Actual Image Real Objects Sensor Sensor Given the model was only trained on MNIST, it didn’t work with the heart reconstruction, a definite drawback. In addition, it requires tons of data: with only 100 training set images, the following model could not accurately create a mapping between the measurement and the ground truth (binary cross entropy error ≈ 0.11 after 300 iterations). Collecting data to be the ground truth is certainly a challenge and would only be optimal if the camera is imaging specific objects. Conclusion Sensor Deep Learning Optimization Deep Learning Model Neural Networks operate by having layers of nodes that accept an input and return an output. Similar to optimization, it calculates a gradient and uses that to adjust each layer’s nodes. Our model uses a Unet architecture which is better than traditional CNNs at image to image mapping.3 We also employ DenseNet layers which improve on the traditional fully connected architecture by allowing layers access feature maps of all previous layers.2 Loss = Binary Cross Entropy Reconstructions Sensor The best image quality results came from the optimization algorithm, which is reasonable as the algorithm is very good at finding where the error function is at a minimum. Therefore, it is only limited by the quality of the PSF and the sensor data. The Deep Learning model and Direct Inverse solution were far faster than the optimization algorithm but the results were as good. Interestingly, however, with lower quality sensor data and a lower quality PSF, the Direct Inverse solution resulted in higher quality images than optimization. Images of real objects had sensor data that was more unclear so the Direct Inverse solution was used PSF Reconstruction Non Self Illuminating objects are much harder to image and get good results. The object must have good reflectivity or be light emitting. The measurements of real objects are of poorer quality so the direct inverse solution was used here. Other methods return little or no result. In this study, we analyzed the plausibility and effectiveness of a lensless imaging system and its algorithms. Each of the three methods had their advantageous and drawbacks but the conclusion is this: for highly reflective objects or light-emitting objects, the Raspberry Pi Camera with glass diffuser is a cheap, compact, and lightweight imaging system. It has potential to perform 3D imaging. Thin, translucent films such as tape or paper can substitute for a glass diffuser and suitable for builds in poorer regions. To obtain the best results, ensure that the PSF is clear and without any noise. Placing the film at a correct distance and photoshop can help. For higher quality images and PSFs, use the iterative optimization algorithms while for lower quality images, the direct inverse solution works best. A deep learning model is useful for imaging a specific set of images or imaging when a PSF is unavailable. However, if outside of those two situations, the other methods are better as the deep learning model requires too much data and time to be worth it.