Absolute depth using low-cost light field cameras

Digital cameras are increasingly used for measurement tasks within engineering scenarios, often being part of metrology platforms. Existing cameras are well equipped to provide 2D information about the fields of view (FOV) they observe, the objects within the FOV, and the accompanying environments. But for some applications these 2D results are not sufficient, specifically applications that require Z dimensional data (depth data) along with the X and Y dimensional data. New designs of camera systems have previously been developed by integrating multiple cameras to provide 3D data, ranging from 2 camera photogrammetry to multiple camera stereo systems. Many earlier attempts to record 3D data on 2D sensors have been completed, and likewise many research groups around the world are currently working on camera technology but from different perspectives; computer vision, algorithm development, metrology, etc. Plenoptic or Lightfield camera technology was defined as a technique over 100 years ago but has remained dormant as a potential metrology instrument. Lightfield cameras utilize an additional Micro Lens Array (MLA) in front of the imaging sensor, to create multiple viewpoints of the same scene and allow encoding of depth information. A small number of companies have explored the potential of lightfield cameras, but in the majority, these have been aimed at domestic consumer photography, only ever recording scenes as relative scale greyscale images. This research considers the potential for lightfield cameras to be used for world scene metrology applications, specifically to record absolute coordinate data. Specific interest has been paid to a range of low cost lightfield cameras to; understand the functional/behavioural characteristics of the optics, identify potential need for optical and/or algorithm development, define sensitivity, repeatability and accuracy characteristics and limiting thresholds of use, and allow quantified 3D absolute scale coordinate data to be extracted from the images. The novel output of this work is; an analysis of lightfield camera system sensitivity leading to the definition of Active Zones (linear data generation good data) and In-active Zones (non-linear data generation poor data), development of bespoke calibration algorithms that remove radial/tangential distortion from the data captured using any MLA based camera, and, a light field camera independent algorithm that allows the delivery of 3D coordinate data in absolute units within a well-defined measurable range from a given camera

Software simulation of depth of field effects in video from small aperture cameras

Thesis (M. Eng.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2010.Cataloged from PDF version of thesis.Includes bibliographical references (p. 71-73).This thesis proposes a technique for post processing digital video to introduce a simulated depth of field effect. Because the technique is implemented in software, it affords the user greater control over the parameters of the effect (such as the amount of defocus, aperture shape, and defocus plane) and allows the effect to be used even on hardware which would not typically allow for depth of field. In addition, because it is a completely post processing technique and requires no change in capture method or hardware, it can be used on any video and introduces no new costs. This thesis describes the technique, evaluates its performance on example videos, and proposes further work to improve the technique's reliability.by Jordan Sorensen.M.Eng

From Calibration to Large-Scale Structure from Motion with Light Fields

Classic pinhole cameras project the multi-dimensional information of the light flowing through a scene onto a single 2D snapshot. This projection limits the information that can be reconstructed from the 2D acquisition. Plenoptic (or light field) cameras, on the other hand, capture a 4D slice of the plenoptic function, termed the “light field”. These cameras provide both spatial and angular information on the light flowing through a scene; multiple views are captured in a single photographic exposure facilitating various applications. This thesis is concerned with the modelling of light field (or plenoptic) cameras and the development of structure from motion pipelines using such cameras. Specifically, we develop a geometric model for a multi-focus plenoptic camera, followed by a complete pipeline for the calibration of the suggested model. Given a calibrated light field camera, we then remap the captured light field to a grid of pinhole images. We use these images to obtain metric 3D reconstruction through a novel framework for structure from motion with light fields. Finally, we suggest a linear and efficient approach for absolute pose estimation for light fields

Coded aperture imaging

This thesis studies the coded aperture camera, a device consisting of a conventional camera with a modified aperture mask, that enables the recovery of both depth map and all-in-focus image from a single 2D input image. Key contributions of this work are the modeling of the statistics of natural images and the design of efficient blur identification methods in a Bayesian framework. Two cases are distinguished: 1) when the aperture can be decomposed in a small set of identical holes, and 2) when the aperture has a more general configuration. In the first case, the formulation of the problem incorporates priors about the statistical variation of the texture to avoid ambiguities in the solution. This allows to bypass the recovery of the sharp image and concentrate only on estimating depth. In the second case, the depth reconstruction is addressed via convolutions with a bank of linear filters. Key advantages over competing methods are the higher numerical stability and the ability to deal with large blur. The all-in-focus image can then be recovered by using a deconvolution step with the estimated depth map. Furthermore, for the purpose of depth estimation alone, the proposed algorithm does not require information about the mask in use. The comparison with existing algorithms in the literature shows that the proposed methods achieve state-of-the-art performance. This solution is also extended for the first time to images affected by both defocus and motion blur and, finally, to video sequences with moving and deformable objects

Contribution to ground-based and UAV SAR systems for Earth observation

Mankind's way of life is the main driver of a planetary-scale change that is marked by the growing of human population's demand of energy, food, goods, services and information. As a result, it have emerged new ecological, economical, social and geopolitical concerns. In this scenario, SAR remote sensing is a potential tool that provides unique information about the Earth's properties and processes that can be used to solve societal challenges of local and global dimension. SARs, which are coherent systems that are able to provide high resolution images with weather independence, represent a suitable alternative for EO with diverse applications. Some examples of SAR application areas are topography (DEM generation with interferometry), agriculture (crop classification or soil moisture), or geology (monitoring surface deformation). In this framework, the encompassing objective of the present doctoral work has been part of the implementation and the subsequent evaluation of capabilities of two X-band SAR sensors. On the one hand, the RISKSAR-X radar designed to be operated at ground to monitor small-scale areas of observation and, on the other, the ARBRES-X sensor designed to be integrated into small UAVs. Despite its inherently dissimilar conception, the concurrence of both sensors has been evidenced along this manuscript. By taking advantage of the similarities between them, it has been possible to analogously assess both sensors to obtain conclusions. In this context, the common link has been the development of the polarimetric OtF operation mode of the RISKSAR-X, allowing this sensor to be operated equivalently to the ARBRES-X. Regarding the RISKSAR-X SAR sensor, several hardware contributions have been developed during part of this Ph.D. with the aim of improving the system performance. By endowing the system with the capability to operate in the fully polarimetric OtF acquisition mode, the relative long scanning time has been reduced. It is of great interest since the measured scatterers that present a short term variable reflectivity during the scanning time, such as moving vegetation, may degrade the extracted parameters from the retrieved data and the SAR image reconstruction. During this doctoral activity, it has been studied the image blurring, the decorrelation and the coherence degradation introduced by this effect. Furthermore, a new term in the differential interferometric coherence that takes into account the image blurring has been introduced. Concerning the ARBRES-X SAR system, one of the main objectives pursued during this Ph.D. has been the integration of the sensor into a small UAV MP overcoming restrictions of weight, size and aerodynamics of the platform. The use of this type of platforms is expected to open up new possibilities in airborne SAR remote sensing, since it offers much more versatility than the commonly used fixed wings UAVs. Different innovative flight strategies with this type of platforms have been assessed and some preliminary results have been obtained with the use of the ARBRES-X SAR system. During the course of the present doctoral work, much effort has been devoted to achieve the first experimental repeat-pass interfereometric results obtained with the UAV MP together with the ARBRES-X. Moreover, the sensor has been endowed with fully polarimetric capabilities by applying the improvements developed to the RISKSAR-X radar, which is another example of the duality between both systems. Finally, a vertical and a semicircular aperture have been successfully performed obtaining SLC images of the scenario, which envisages the capability of the UAV MP to perform tomographic images and complete circular apertures in the future. In conclusion, the UAV MP is a promising platform that opens new potentials for several applications, such as repeat-pass interferometry or differential tomography imaging with the realization of almost arbitrary trajectories.El mode de viure de la humanitat és el principal motor d'un canvi a escala planetària que està marcat per la creixent demanda d'energia, d'aliment, de béns, de serveis i d'informació de les poblacions humanes. Com a resultat, han sorgit noves inquietuds ecològiques, econòmiques, socials i geopolítiques. En aquest escenari, la detecció remota SAR és una eina potencial que proporciona informació única sobre les propietats i processos de la Terra que es pot utilitzar per resoldre reptes socials de dimensió local i global. Els SARs, que són sistemes coherents que poden proporcionar imatges d'alta resolució amb independència del temps, representen una alternativa adequada per a l'observació de la Terra. Alguns exemples d'àrees d'aplicació SAR són la topografia (generació de DEM amb interferometria), l'agricultura (classificació de cultius o humitat del sòl) o la geologia (monitoratge de deformació superficial). En aquest context, l'objectiu general del present doctorat ha estat part de la implementació i posterior avaluació de les capacitats de dos sensors SAR de banda X. D'una banda, el radar RISKSAR-X dissenyat per funcionar a terra i monitoritzar àrees d'observació a petita escala i, d'altra, el sensor ARBRES-X dissenyat per ser integrat en petits UAVs. Malgrat la seva concepció inherentment diferent, la concurrència d'ambdós sensors s'ha evidenciat al llarg d'aquest manuscrit. Aprofitant les similituds entre ells, s'han pogut avaluar de forma anàloga els dos sensors per obtenir conclusions. En aquest sentit, el vincle comú ha estat el desenvolupament del mode de funcionament polimètric OtF del RISKSAR-X, permetent que aquest sensor operi de forma equivalent a l'ARBRES-X. Pel que fa al sensor RISKSAR-X, s'han desenvolupat diverses contribucions hardware durant part d'aquest doctorat amb l'objectiu de millorar el rendiment del sistema. En dotar el sistema de la possibilitat d'operar en el mode d'adquisició totalment polarimètric OtF, s'ha reduït el relatiu llarg temps d'escaneig. Això és de gran interès ja que els blancs mesurats que presenten una reflectivitat variable a curt termini, com ara la vegetació en moviment, poden degradar els paràmetres extrets de les dades recuperades i la reconstrucció d'imatges SAR. Durant aquesta activitat doctoral s'ha estudiat el desenfocat de la imatge, la decorrelació i la degradació de la coherència introduïts per aquest efecte. A més, s'ha introduït un nou terme en la coherència interferomètrica diferencial que té en compte el desenfocat de la imatge. Pel que fa al sistema ARBRES-X, un dels principals objectius perseguits durant aquest doctorat ha estat la integració del sensor en un petit UAV MP superant les restriccions de pes, grandària i aerodinàmica de la plataforma. S'espera que l'ús d'aquest tipus de plataformes obri noves possibilitats en la detecció remota SAR aerotransportada, ja que ofereix molta més versatilitat que els UAV d'ales fixes habituals. S'han avaluat diferents estratègies de vol innovadores amb aquest tipus de plataformes i s'han obtingut resultats preliminars amb l'ús del sistema ARBRES-X. Durant el transcurs del present treball, s'ha dedicat molt esforç a assolir els primers resultats experimentals d'interferometria de múltiple passada obtinguts amb l'UAV MP conjuntament amb l'ARBRES-X. A més, el sensor ha estat dotat de capacitats totalment polarimètriques aplicant les millores desenvolupades al radar RISKSAR-X, el qual constitueix un altre exemple de la dualitat entre ambdós sistemes. Finalment, s'han realitzat amb èxit una apertura vertical i semicircular obtenint imatges SLC de l'escenari, el qual permet preveure la capacitat de l'UAV MP per a realitzar imatges tomogràfiques i apertures circulars completes en el futur. En conclusió, l'UAV MP és una plataforma prometedora que obre nous potencials per a diverses aplicacions, com ara la interferometria de múltiple passada o la tomografia diferencial amb la realització de trajectòries gairebé arbitràries

Contribution to ground-based and UAV SAR systems for Earth observation

Mankind's way of life is the main driver of a planetary-scale change that is marked by the growing of human population's demand of energy, food, goods, services and information. As a result, it have emerged new ecological, economical, social and geopolitical concerns. In this scenario, SAR remote sensing is a potential tool that provides unique information about the Earth's properties and processes that can be used to solve societal challenges of local and global dimension. SARs, which are coherent systems that are able to provide high resolution images with weather independence, represent a suitable alternative for EO with diverse applications. Some examples of SAR application areas are topography (DEM generation with interferometry), agriculture (crop classification or soil moisture), or geology (monitoring surface deformation). In this framework, the encompassing objective of the present doctoral work has been part of the implementation and the subsequent evaluation of capabilities of two X-band SAR sensors. On the one hand, the RISKSAR-X radar designed to be operated at ground to monitor small-scale areas of observation and, on the other, the ARBRES-X sensor designed to be integrated into small UAVs. Despite its inherently dissimilar conception, the concurrence of both sensors has been evidenced along this manuscript. By taking advantage of the similarities between them, it has been possible to analogously assess both sensors to obtain conclusions. In this context, the common link has been the development of the polarimetric OtF operation mode of the RISKSAR-X, allowing this sensor to be operated equivalently to the ARBRES-X. Regarding the RISKSAR-X SAR sensor, several hardware contributions have been developed during part of this Ph.D. with the aim of improving the system performance. By endowing the system with the capability to operate in the fully polarimetric OtF acquisition mode, the relative long scanning time has been reduced. It is of great interest since the measured scatterers that present a short term variable reflectivity during the scanning time, such as moving vegetation, may degrade the extracted parameters from the retrieved data and the SAR image reconstruction. During this doctoral activity, it has been studied the image blurring, the decorrelation and the coherence degradation introduced by this effect. Furthermore, a new term in the differential interferometric coherence that takes into account the image blurring has been introduced. Concerning the ARBRES-X SAR system, one of the main objectives pursued during this Ph.D. has been the integration of the sensor into a small UAV MP overcoming restrictions of weight, size and aerodynamics of the platform. The use of this type of platforms is expected to open up new possibilities in airborne SAR remote sensing, since it offers much more versatility than the commonly used fixed wings UAVs. Different innovative flight strategies with this type of platforms have been assessed and some preliminary results have been obtained with the use of the ARBRES-X SAR system. During the course of the present doctoral work, much effort has been devoted to achieve the first experimental repeat-pass interfereometric results obtained with the UAV MP together with the ARBRES-X. Moreover, the sensor has been endowed with fully polarimetric capabilities by applying the improvements developed to the RISKSAR-X radar, which is another example of the duality between both systems. Finally, a vertical and a semicircular aperture have been successfully performed obtaining SLC images of the scenario, which envisages the capability of the UAV MP to perform tomographic images and complete circular apertures in the future. In conclusion, the UAV MP is a promising platform that opens new potentials for several applications, such as repeat-pass interferometry or differential tomography imaging with the realization of almost arbitrary trajectories.El mode de viure de la humanitat és el principal motor d'un canvi a escala planetària que està marcat per la creixent demanda d'energia, d'aliment, de béns, de serveis i d'informació de les poblacions humanes. Com a resultat, han sorgit noves inquietuds ecològiques, econòmiques, socials i geopolítiques. En aquest escenari, la detecció remota SAR és una eina potencial que proporciona informació única sobre les propietats i processos de la Terra que es pot utilitzar per resoldre reptes socials de dimensió local i global. Els SARs, que són sistemes coherents que poden proporcionar imatges d'alta resolució amb independència del temps, representen una alternativa adequada per a l'observació de la Terra. Alguns exemples d'àrees d'aplicació SAR són la topografia (generació de DEM amb interferometria), l'agricultura (classificació de cultius o humitat del sòl) o la geologia (monitoratge de deformació superficial). En aquest context, l'objectiu general del present doctorat ha estat part de la implementació i posterior avaluació de les capacitats de dos sensors SAR de banda X. D'una banda, el radar RISKSAR-X dissenyat per funcionar a terra i monitoritzar àrees d'observació a petita escala i, d'altra, el sensor ARBRES-X dissenyat per ser integrat en petits UAVs. Malgrat la seva concepció inherentment diferent, la concurrència d'ambdós sensors s'ha evidenciat al llarg d'aquest manuscrit. Aprofitant les similituds entre ells, s'han pogut avaluar de forma anàloga els dos sensors per obtenir conclusions. En aquest sentit, el vincle comú ha estat el desenvolupament del mode de funcionament polimètric OtF del RISKSAR-X, permetent que aquest sensor operi de forma equivalent a l'ARBRES-X. Pel que fa al sensor RISKSAR-X, s'han desenvolupat diverses contribucions hardware durant part d'aquest doctorat amb l'objectiu de millorar el rendiment del sistema. En dotar el sistema de la possibilitat d'operar en el mode d'adquisició totalment polarimètric OtF, s'ha reduït el relatiu llarg temps d'escaneig. Això és de gran interès ja que els blancs mesurats que presenten una reflectivitat variable a curt termini, com ara la vegetació en moviment, poden degradar els paràmetres extrets de les dades recuperades i la reconstrucció d'imatges SAR. Durant aquesta activitat doctoral s'ha estudiat el desenfocat de la imatge, la decorrelació i la degradació de la coherència introduïts per aquest efecte. A més, s'ha introduït un nou terme en la coherència interferomètrica diferencial que té en compte el desenfocat de la imatge. Pel que fa al sistema ARBRES-X, un dels principals objectius perseguits durant aquest doctorat ha estat la integració del sensor en un petit UAV MP superant les restriccions de pes, grandària i aerodinàmica de la plataforma. S'espera que l'ús d'aquest tipus de plataformes obri noves possibilitats en la detecció remota SAR aerotransportada, ja que ofereix molta més versatilitat que els UAV d'ales fixes habituals. S'han avaluat diferents estratègies de vol innovadores amb aquest tipus de plataformes i s'han obtingut resultats preliminars amb l'ús del sistema ARBRES-X. Durant el transcurs del present treball, s'ha dedicat molt esforç a assolir els primers resultats experimentals d'interferometria de múltiple passada obtinguts amb l'UAV MP conjuntament amb l'ARBRES-X. A més, el sensor ha estat dotat de capacitats totalment polarimètriques aplicant les millores desenvolupades al radar RISKSAR-X, el qual constitueix un altre exemple de la dualitat entre ambdós sistemes. Finalment, s'han realitzat amb èxit una apertura vertical i semicircular obtenint imatges SLC de l'escenari, el qual permet preveure la capacitat de l'UAV MP per a realitzar imatges tomogràfiques i apertures circulars completes en el futur. En conclusió, l'UAV MP és una plataforma prometedora que obre nous potencials per a diverses aplicacions, com ara la interferometria de múltiple passada o la tomografia diferencial amb la realització de trajectòries gairebé arbitràries.Postprint (published version

Orbital Determination Feasibility of LEO Nanosatellites Using Small Aperture Telescopes

This thesis is directed toward the feasibility of observing satellites on the nano scale and determining an accurate propagated orbit using a Meade LX600-ACF 14” diameter aperture telescope currently located on the California Polytechnic State University campus. The optical telescope is fitted with an f/6.3 focal reducer, SBIG ST-10XME CCD camera and Optec TCF-S Focuser. This instrumentation allowed for a 22’ X 15’ arcminute FOV in order to accurately image passing LEO satellites. Through the use of the Double-r and Gauss Initial Orbit Determination methods as well as Least Squared Differential Correction and Extended Kalman Filter Orbit Determination methods, an accurate predicted orbit can be determined. These calculated values from observational data of satellites within the Globalstar system are compared against the most updated TLEs for each satellite at the time of observation. The determined differential errors from the well-defined TLEs acquired via online database were used to verify the feasibility of the accuracy which can be obtained from independent observations. Through minimization of error caused from imaging noise, pointing error, and timing error, the main determination of accurate orbital determination lies in the instrumentation mechanical capabilities itself. With the ability to acquire up to 7 individual satellite observations during a single transit, the use of both IOD and OD methods, and the recently acquired Cal Poly telescope with an increased 14” aperture, the feasibility of imaging and orbital determination of nanosatellites is greatly improved

Enhanced processing methods for light field imaging

The light field camera provides rich textural and geometric information, but it is still challenging to use it efficiently and accurately to solve computer vision problems. Light field image processing is divided into multiple levels. First, low-level processing technology mainly includes the acquisition of light field images and their preprocessing. Second, the middle-level process consists of the depth estimation, light field encoding, and the extraction of cues from the light field. Third, high-level processing involves 3D reconstruction, target recognition, visual odometry, image reconstruction, and other advanced applications. We propose a series of improved algorithms for each of these levels. The light field signal contains rich angular information. By contrast, traditional computer vision methods, as used for 2D images, often cannot make full use of the high-frequency part of the light field angular information. We propose a fast pre-estimation algorithm to enhance the light field feature to improve its speed and accuracy when keeping full use of the angular information.Light field filtering and refocusing are essential cues in light field signal processing. Modern frequency domain filtering technology and wavelet technology have effectively improved light field filtering accuracy but may fail at object edges. We adapted the sub-window filtering with the light field to improve the reconstruction of object edges. Light field images can analyze the effects of scattering and refraction phenomena, and there are still insufficient metrics to evaluate the results. Therefore, we propose a physical rendering-based light field dataset that simulates the distorted light field image through a transparent medium, such as atmospheric turbulence or water surface. The neural network is an essential method to process complex light field data. We propose an efficient 3D convolutional autoencoder network for the light field structure. This network overcomes the severe distortion caused by high-intensity turbulence with limited angular resolution and solves the difficulty of pixel matching between distorted images. This work emphasizes the application and usefulness of light field imaging in computer vision whilst improving light field image processing speed and accuracy through signal processing, computer graphics, computer vision, and artificial neural networks

Onboard Science Instrument Autonomy for the Detection of Microscopy Biosignatures on the Ocean Worlds Life Surveyor

The quest to find extraterrestrial life is a critical scientific endeavor with civilization-level implications. Icy moons in our solar system are promising targets for exploration because their liquid oceans make them potential habitats for microscopic life. However, the lack of a precise definition of life poses a fundamental challenge to formulating detection strategies. To increase the chances of unambiguous detection, a suite of complementary instruments must sample multiple independent biosignatures (e.g., composition, motility/behavior, and visible structure). Such an instrument suite could generate 10,000x more raw data than is possible to transmit from distant ocean worlds like Enceladus or Europa. To address this bandwidth limitation, Onboard Science Instrument Autonomy (OSIA) is an emerging discipline of flight systems capable of evaluating, summarizing, and prioritizing observational instrument data to maximize science return. We describe two OSIA implementations developed as part of the Ocean Worlds Life Surveyor (OWLS) prototype instrument suite at the Jet Propulsion Laboratory. The first identifies life-like motion in digital holographic microscopy videos, and the second identifies cellular structure and composition via innate and dye-induced fluorescence. Flight-like requirements and computational constraints were used to lower barriers to infusion, similar to those available on the Mars helicopter, "Ingenuity." We evaluated the OSIA's performance using simulated and laboratory data and conducted a live field test at the hypersaline Mono Lake planetary analog site. Our study demonstrates the potential of OSIA for enabling biosignature detection and provides insights and lessons learned for future mission concepts aimed at exploring the outer solar system.Comment: 49 pages, 18 figures, submitted to The Planetary Science Journal on 2023-04-2