Illumination Invariant Outdoor Perception

This thesis proposes the use of a multi-modal sensor approach to achieve illumination invariance in images taken in outdoor environments. The approach is automatic in that it does not require user input for initialisation, and is not reliant on the input of atmospheric radiative transfer models. While it is common to use pixel colour and intensity as features in high level vision algorithms, their performance is severely limited by the uncontrolled lighting and complex geometric structure of outdoor scenes. The appearance of a material is dependent on the incident illumination, which can vary due to spatial and temporal factors. This variability causes identical materials to appear differently depending on their location. Illumination invariant representations of the scene can potentially improve the performance of high level vision algorithms as they allow discrimination between pixels to occur based on the underlying material characteristics. The proposed approach to obtaining illumination invariance utilises fused image and geometric data. An approximation of the outdoor illumination is used to derive per-pixel scaling factors. This has the effect of relighting the entire scene using a single illuminant that is common in terms of colour and intensity for all pixels. The approach is extended to radiometric normalisation and the multi-image scenario, meaning that the resultant dataset is both spatially and temporally illumination invariant. The proposed illumination invariance approach is evaluated on several datasets and shows that spatial and temporal invariance can be achieved without loss of spectral dimensionality. The system requires very few tuning parameters, meaning that expert knowledge is not required in order for its operation. This has potential implications for robotics and remote sensing applications where perception systems play an integral role in developing a rich understanding of the scene

Data-Driven Radiometric Photo-Linearization

In computer vision and computer graphics, a photograph is often considered a photometric representation of a scene. However, for most camera models, the relation between recorded pixel value and the amount of light received on the sensor is not linear. This non-linear relationship is modeled by the camera response function which maps the scene radiance to the image brightness. This non-linear transformation is unknown, and it can only be recovered via a rigorous radiometric calibration process. Classic radiometric calibration methods typically estimate a camera response function from an exposure stack (i.e., an image sequence captured with different exposures from the same viewpoint and time). However, for photographs in large image collections for which we do not have control over the capture process, traditional radiometric calibration methods cannot be applied. This thesis details two novel data-driven radiometric photo-linearization methods suit- able for photographs captured with unknown camera settings and under uncontrolled conditions. First, a novel example-based radiometric linearization method is pro- posed, that takes as input a radiometrically linear photograph of a scene (i.e., exemplar), and a standard (radiometrically uncalibrated) image of the same scene potentially from a different viewpoint and/or under different lighting, and which produces a radiometrically linear version of the latter. Key to this method is the observation that for many patches, their change in appearance (from different viewpoints and lighting) forms a 1D linear subspace. This observation allows the problem to be reformulated in a form similar to classic radiometric calibration from an exposure stack. In addition, practical solutions are proposed to automatically select and align the best matching patches/correspondences between the two photographs, and to robustly reject outliers/unreliable matches. Second, CRF-net (or Camera Response Function net), a robust single image radiometric calibration method based on convolutional neural net- works (CNNs) is presented. The proposed network takes as input a single photograph, and outputs an estimate of the camera response function in the form of the 11 PCA coefficients for the EMoR camera response model. CRF-net is able to accurately recover the camera response function from a single photograph under a wide range of conditions

Scene Monitoring With A Forest Of Cooperative Sensors

In this dissertation, we present vision based scene interpretation methods for monitoring of people and vehicles, in real-time, within a busy environment using a forest of co-operative electro-optical (EO) sensors. We have developed novel video understanding algorithms with learning capability, to detect and categorize people and vehicles, track them with in a camera and hand-off this information across multiple networked cameras for multi-camera tracking. The ability to learn prevents the need for extensive manual intervention, site models and camera calibration, and provides adaptability to changing environmental conditions. For object detection and categorization in the video stream, a two step detection procedure is used. First, regions of interest are determined using a novel hierarchical background subtraction algorithm that uses color and gradient information for interest region detection. Second, objects are located and classified from within these regions using a weakly supervised learning mechanism based on co-training that employs motion and appearance features. The main contribution of this approach is that it is an online procedure in which separate views (features) of the data are used for co-training, while the combined view (all features) is used to make classification decisions in a single boosted framework. The advantage of this approach is that it requires only a few initial training samples and can automatically adjust its parameters online to improve the detection and classification performance. Once objects are detected and classified they are tracked in individual cameras. Single camera tracking is performed using a voting based approach that utilizes color and shape cues to establish correspondence in individual cameras. The tracker has the capability to handle multiple occluded objects. Next, the objects are tracked across a forest of cameras with non-overlapping views. This is a hard problem because of two reasons. First, the observations of an object are often widely separated in time and space when viewed from non-overlapping cameras. Secondly, the appearance of an object in one camera view might be very different from its appearance in another camera view due to the differences in illumination, pose and camera properties. To deal with the first problem, the system learns the inter-camera relationships to constrain track correspondences. These relationships are learned in the form of multivariate probability density of space-time variables (object entry and exit locations, velocities, and inter-camera transition times) using Parzen windows. To handle the appearance change of an object as it moves from one camera to another, we show that all color transfer functions from a given camera to another camera lie in a low dimensional subspace. The tracking algorithm learns this subspace by using probabilistic principal component analysis and uses it for appearance matching. The proposed system learns the camera topology and subspace of inter-camera color transfer functions during a training phase. Once the training is complete, correspondences are assigned using the maximum a posteriori (MAP) estimation framework using both the location and appearance cues. Extensive experiments and deployment of this system in realistic scenarios has demonstrated the robustness of the proposed methods. The proposed system was able to detect and classify targets, and seamlessly tracked them across multiple cameras. It also generated a summary in terms of key frames and textual description of trajectories to a monitoring officer for final analysis and response decision. This level of interpretation was the goal of our research effort, and we believe that it is a significant step forward in the development of intelligent systems that can deal with the complexities of real world scenarios

Radiometric calibration methods from image sequences

In many computer vision systems, an image of a scene is assumed to directly reflect the scene radiance. However, this is not the case for most cameras as the radiometric response function which is a mapping from the scene radiance to the image brightness is nonlinear. In addition, the exposure settings of the camera are adjusted (often in the auto-exposure mode) according to the dynamic range of the scene changing the appearance of the scene in the images. Vignetting effect which refers to the gradual fading-out of an image at points near its periphery also contributes in changing the scene appearance in images. In this dissertation, I present several algorithms to compute the radiometric properties of a camera which enable us to find the relationship between the image brightness and the scene radiance. First, I introduce an algorithm to compute the vignetting function, the response function, and the exposure values that fully explain the radiometric image formation process from a set of images of a scene taken with different and unknown exposure values. One of the key features of the proposed method is that the movement of the camera is not limited when taking the pictures whereas most existing methods limit the motion of the camera. Then I present a joint feature tracking and radiometric calibration scheme which performs an integrated radiometric calibration in contrast to previous radiometric calibration techniques which require the correspondences as an input which leads to a chicken-and-egg problem as precise tracking requires accurate radiometric calibration. By combining both into an integrated approach we solve this chicken-and-egg problem. Finally, I propose a radiometric calibration method suited for a set of images of an outdoor scene taken at a regular interval over a period of time. This type of data is a challenging problem because the illumination for each image is changing causing the exposure of the camera to change and the conventional radiometric calibration framework cannot be used for this type of data. The proposed methods are applied to radiometrically align images for seamless mosaics and 3D model textures, to create high dynamic range mosaics, and to build an adaptive stereo system

Measuring atmospheric scattering from digital images of urban scenery using temporal polarization-based vision

Suspended atmospheric particles (particulate matter) are a form of air pollution that visually degrades urban scenery and is hazardous to human health and the environment. Current environmental monitoring devices are limited in their capability of measuring average particulate matter (PM) over large areas. Quantifying the visual effects of haze in digital images of urban scenery and correlating these effects to PM levels is a vital step in more practically monitoring our environment. Current image haze extraction algorithms remove all the haze from the scene and hence produce unnatural scenes for the sole purpose of enhancing vision. We present two algorithms which bridge the gap between image haze extraction and environmental monitoring. We provide a means of measuring atmospheric scattering from images of urban scenery by incorporating temporal knowledge. In doing so, we also present a method of recovering an accurate depthmap of the scene and recovering the scene without the visual effects of haze. We compare our algorithm to three known haze removal methods from the perspective of measuring atmospheric scattering, measuring depth and dehazing. The algorithms are composed of an optimization over a model of haze formation in images and an optimization using the constraint of constant depth over a sequence of images taken over time. These algorithms not only measure atmospheric scattering, but also recover a more accurate depthmap and dehazed image. The measurements of atmospheric scattering this research produces, can be directly correlated to PM levels and therefore pave the way to monitoring the health of the environment by visual means. Accurate atmospheric sensing from digital images is a challenging and under-researched problem. This work provides an important step towards a more practical and accurate visual means of measuring PM from digital images

Service robotics and machine learning for close-range remote sensing

L'abstract è presente nell'allegato / the abstract is in the attachmen

Multispectral Image Analysis of Remotely Sensed Crops

The range in topography, biodiversity, and agricultural technology has led to the emergence of precision agriculture. Precision agriculture is a farming management concept based on monitoring, measuring, and responding to crop variability. Computer vision, image analysis, and image processing are gaining considerable traction. For this paper, image analysis involves recognizing individual objects and providing insights from vegetation indices. The data acquired was remote-sensed multispectral images from blueberry, maguey, and pineapple. After computing vegetation indices, histograms were analyzed to choose thresholds. The masking of vegetation indices with threshold allowed the removal of areas with shadows and soil. The four leading vegetation indices used were the Normalized Difference Vegetation Index (NDVI), the Normalized Difference Red Edge (NDRE), the Simple Ratio, the Red Edge Chlorophyll Index, and the Visible Atmospherically Resistant Index (SAVI). This research reviews literature for acquiring, preprocessing, and analyzing remote-sensed multispectral images in precision agriculture. It compiles the theoretical framework for analyzing multispectral data. Also, it describes and implements radiometric calibration and image alignment using the custom code from the MicaSense repository. As a result, it was possible to segment the blueberry, tequila agave, and pineapple plants from the background regardless of the noisy images. Non-plant pixels were excluded and shown as transparent by masking areas with shadows and low NDVI pixels, which sometimes removed plant pixels. The NDVI and NDRE helped identify crop pixels. On the other hand, it was possible to identify the pineapple fruits from the agave plantation using the SAVI vegetation index and the thresholding method. Finally, the work identifies the problems associated with an incorrect data acquisition methodology and provides suggestions.ITESO, A. C

An Experimental Campaign to Collect Multiresolution Hyperspectral Data with Ground Truth

Advancements in hyperspectral imaging systems, data processing, and collection methods, have allowed for comprehensive spectral assessment of terrestrial objects to assist in a vast array of imaging problems previously deemed too complex. As the field evolves and more research is completed, the boundaries of what is capable are constantly pushed to new thresholds. At this junction, there exists an opportunity for the capabilities of these new creations to exceed our ability to prove their efficiency and efficacy. One area where this is true, is in the assessment of new hyperspectral panchromatic sharpening (HPS) algorithms. This is due to the lack of a true multi-resolution hyperspectral data focused on the same scene, which is needed to properly test these algorithms. Typically, the assessment process awaiting a HPS algorithm starts by taking a high-resolution Hyperspectral Image (HSI), which is then spatially degraded so that after the sharpening process, the result can be compared to the original and analyzed for accuracy. Presentations featuring the results of such algorithms often lead to immediate questions about quantitative assessments based solely on simulated low-resolution data. To address this problem, a collection experiment seeking to create a multi-resolution hyperspectral data set was designed and completed in Henrietta, NY on July 24th, 2020. Imagery of 48 felt targets, ranging in size from 5 cm to 30 cm and in six different colors, was collected in three resolutions by the Rochester Institute of Technology (RIT) MX-1 sUAS imaging system. In addition to a high resolution 5-band frame camera, the MX-1 utilizes a 272-band hyperspectral line imager that features a silicon detector, 1.88cm GSD at 30m flight altitude, and is responsive from approximately 400nm to 1000nm. To create the desired multi-resolution imagery, three flights were performed, and images of the target scene were collected at flight heights of 30m, 60m, and 120m. The resulting imagery possesses ratios of 2:1 and 4:1 spatial resolution relative to the lowest altitude flight. Use of this data in the evaluation process of HPS algorithms ensures that the radiometric accuracy of the result and the effectiveness of the sharpening method employed can be better ascertained through quantitative analysis of the inputs and output after being applied to this non-simulated multiresolution hyperspectral data set. During planning and after collection, care was taken to rigorously document the data set, its structure, and features, as the intention exists for the data to be used for various purposes long exceeding the extent of its immediate usefulness to researchers at RIT. Additional documentation provided as ancillary metadata describes the spectral signatures of all non-environmental materials present in the HSI target scene, the process of creating both the targets and their layout, as well as the collection plan, the data itself, post processing steps, and preliminary experiments undertaken to improve end-user understanding and ease of use. This documentation also includes an assessment surmised through understanding the flaws present within the data, and portions of the collection plan which did not lead to the intended outcome. Because of the efforts described in this document, there now exists a multi-resolution hyperspectral data set of a known target scene with highly documented ground truth. This data and report will prove useful in the assessment of HPS algorithms currently under development, but also has the potential to assist as useful test data for various applications within the fields of hyperspectral imaging and remote sensing