The emerging versatility of a scannerless range imager

Sandia National Laboratories is nearing the completion of the initial development of a unique type of range imaging sensor. This innovative imaging optical radar is based on an active flood-light scene illuminator and an image intensified CCD camera receiver. It is an all solid-state device (no moving parts) and offers significant size, performance, reliability, simplicity, and affordability advantages over other types of 3-D sensor technologies, including: scanned laser radar, stereo vision, and structured lighting. The sensor is based on low cost, commercially available hardware, and is very well suited for affordable application to a wide variety of military and commercial uses, including: munition guidance, target recognition, robotic vision, automated inspection, driver enhanced vision, collision avoidance, site security and monitoring, terrain mapping, and facility surveying. This paper reviews the sensor technology and its development for the advanced conventional munition guidance application, and discusses a few of the many other emerging applications for this new innovative sensor technology

3D indoor scene modeling from RGB-D data: a survey

3D scene modeling has long been a fundamental problem in computer graphics and computer vision. With the popularity of consumer-level RGB-D cameras, there is a growing interest in digitizing real-world indoor 3D scenes. However, modeling indoor 3D scenes remains a challenging problem because of the complex structure of interior objects and poor quality of RGB-D data acquired by consumer-level sensors. Various methods have been proposed to tackle these challenges. In this survey, we provide an overview of recent advances in indoor scene modeling techniques, as well as public datasets and code libraries which can facilitate experiments and evaluation

Orientation and integration of images and image blocks with laser scanning data

Laser scanning and photogrammetry are methods for effective and accurate measurement and classification of urban and forest areas. Because these methods complement each other, then integration or integrated use brings additional benefits to real-life applications. However, finding tie features between data sets is a challenging task since laser scanning and imagery are far from each other in nature. The aim of this thesis was to create methods for solving relative orientations between laser scanning data and imagery that would assist in near-future applications integrating laser scanning and photogrammetry. Moreover, a further goal was to create methods enabling the use of data acquired from very different perspectives, such as terrestrial and airborne data. To meet these aims, an interactive orientation method enabling the use of single images, stereo images or larger image blocks was developed and tested. The multi-view approach usually has a significant advantage over the use of a single image. After accurate orientation of laser scanning data and imagery, versatile applications become available. Such applications include, e.g., automatic object recognition, accurate classification of individual trees, point cloud densification, automatic classification of land use, system calibration, and generation of photorealistic 3D models. Besides the orientation part, another aim of the research was to investigate how to fuse or use these two data types together in applications. As a result, examples that evaluated the behavior of laser point clouds in both urban and forestry areas, detection and visualization of temporal changes, enhanced data understanding, stereo visualization, multi-source and multi-angle data fusion, point cloud colorizing, and detailed examination of full waveform laser scanning data were given

An overview of depth cameras and range scanners based on time-of-flight technologies

“The final publication is available at Springer via http://dx.doi.10.1007/s00138-016-0784-4.This work has received funding from the French Agence Nationale de la Recherche (ANR) under the MIXCAM project ANR-13-BS02-0010-01, and from the European Research Council (ERC) under the Advanced Grant VHIA Project 340113

Fruit Detection and Tree Segmentation for Yield Mapping in Orchards

Accurate information gathering and processing is critical for precision horticulture, as growers aim to optimise their farm management practices. An accurate inventory of the crop that details its spatial distribution along with health and maturity, can help farmers efficiently target processes such as chemical and fertiliser spraying, crop thinning, harvest management, labour planning and marketing. Growers have traditionally obtained this information by using manual sampling techniques, which tend to be labour intensive, spatially sparse, expensive, inaccurate and prone to subjective biases. Recent advances in sensing and automation for field robotics allow for key measurements to be made for individual plants throughout an orchard in a timely and accurate manner. Farmer operated machines or unmanned robotic platforms can be equipped with a range of sensors to capture a detailed representation over large areas. Robust and accurate data processing techniques are therefore required to extract high level information needed by the grower to support precision farming. This thesis focuses on yield mapping in orchards using image and light detection and ranging (LiDAR) data captured using an unmanned ground vehicle (UGV). The contribution is the framework and algorithmic components for orchard mapping and yield estimation that is applicable to different fruit types and orchard configurations. The framework includes detection of fruits in individual images and tracking them over subsequent frames. The fruit counts are then associated to individual trees, which are segmented from image and LiDAR data, resulting in a structured spatial representation of yield. The first contribution of this thesis is the development of a generic and robust fruit detection algorithm. Images captured in the outdoor environment are susceptible to highly variable external factors that lead to significant appearance variations. Specifically in orchards, variability is caused by changes in illumination, target pose, tree types, etc. The proposed techniques address these issues by using state-of-the-art feature learning approaches for image classification, while investigating the utility of orchard domain knowledge for fruit detection. Detection is performed using both pixel-wise classification of images followed instance segmentation, and bounding-box regression approaches. The experimental results illustrate the versatility of complex deep learning approaches over a multitude of fruit types. The second contribution of this thesis is a tree segmentation approach to detect the individual trees that serve as a standard unit for structured orchard information systems. The work focuses on trellised trees, which present unique challenges for segmentation algorithms due to their intertwined nature. LiDAR data are used to segment the trellis face, and to generate proposals for individual trees trunks. Additional trunk proposals are provided using pixel-wise classification of the image data. The multi-modal observations are fine-tuned by modelling trunk locations using a hidden semi-Markov model (HSMM), within which prior knowledge of tree spacing is incorporated. The final component of this thesis addresses the visual occlusion of fruit within geometrically complex canopies by using a multi-view detection and tracking approach. Single image fruit detections are tracked over a sequence of images, and associated to individual trees or farm rows, with the spatial distribution of the fruit counting forming a yield map over the farm. The results show the advantage of using multi-view imagery (instead of single view analysis) for fruit counting and yield mapping. This thesis includes extensive experimentation in almond, apple and mango orchards, with data captured by a UGV spanning a total of 5 hectares of farm area, over 30 km of vehicle traversal and more than 7,000 trees. The validation of the different processes is performed using manual annotations, which includes fruit and tree locations in image and LiDAR data respectively. Additional evaluation of yield mapping is performed by comparison against fruit counts on trees at the farm and counts made by the growers post-harvest. The framework developed in this thesis is demonstrated to be accurate compared to ground truth at all scales of the pipeline, including fruit detection and tree mapping, leading to accurate yield estimation, per tree and per row, for the different crops. Through the multitude of field experiments conducted over multiple seasons and years, the thesis presents key practical insights necessary for commercial development of an information gathering system in orchards

Perceptually Optimized Visualization on Autostereoscopic 3D Displays

The family of displays, which aims to visualize a 3D scene with realistic depth, are known as "3D displays". Due to technical limitations and design decisions, such displays create visible distortions, which are interpreted by the human vision as artefacts. In absence of visual reference (e.g. the original scene is not available for comparison) one can improve the perceived quality of the representations by making the distortions less visible. This thesis proposes a number of signal processing techniques for decreasing the visibility of artefacts on 3D displays. The visual perception of depth is discussed, and the properties (depth cues) of a scene which the brain uses for assessing an image in 3D are identified. Following the physiology of vision, a taxonomy of 3D artefacts is proposed. The taxonomy classifies the artefacts based on their origin and on the way they are interpreted by the human visual system. The principles of operation of the most popular types of 3D displays are explained. Based on the display operation principles, 3D displays are modelled as a signal processing channel. The model is used to explain the process of introducing distortions. It also allows one to identify which optical properties of a display are most relevant to the creation of artefacts. A set of optical properties for dual-view and multiview 3D displays are identified, and a methodology for measuring them is introduced. The measurement methodology allows one to derive the angular visibility and crosstalk of each display element without the need for precision measurement equipment. Based on the measurements, a methodology for creating a quality profile of 3D displays is proposed. The quality profile can be either simulated using the angular brightness function or directly measured from a series of photographs. A comparative study introducing the measurement results on the visual quality and position of the sweet-spots of eleven 3D displays of different types is presented. Knowing the sweet-spot position and the quality profile allows for easy comparison between 3D displays. The shape and size of the passband allows depth and textures of a 3D content to be optimized for a given 3D display. Based on knowledge of 3D artefact visibility and an understanding of distortions introduced by 3D displays, a number of signal processing techniques for artefact mitigation are created. A methodology for creating anti-aliasing filters for 3D displays is proposed. For multiview displays, the methodology is extended towards so-called passband optimization which addresses Moiré, fixed-pattern-noise and ghosting artefacts, which are characteristic for such displays. Additionally, design of tuneable anti-aliasing filters is presented, along with a framework which allows the user to select the so-called 3d sharpness parameter according to his or her preferences. Finally, a set of real-time algorithms for view-point-based optimization are presented. These algorithms require active user-tracking, which is implemented as a combination of face and eye-tracking. Once the observer position is known, the image on a stereoscopic display is optimised for the derived observation angle and distance. For multiview displays, the combination of precise light re-direction and less-precise face-tracking is used for extending the head parallax. For some user-tracking algorithms, implementation details are given, regarding execution of the algorithm on a mobile device or on desktop computer with graphical accelerator

Investigation of jitter on full-field amplitude modulated continuous wave time-of-flight range imaging cameras

The time-of-flight (ToF) range imaging cameras indirectly measure the time taken from the modulation light source to the scene and back to the camera and it is this principle that is used in depth cameras to perform depth measurements. This thesis is focused on ToF cameras that are based on the amplitude modulated continuous wave (AMCW) lidar techniques which measure the phase difference between the emitted and reflected light signals. Due to their portable size, feasible design, low weight and low energy consumption, these cameras have high demand in many applications. Commercially available AMCW ToF cameras have relatively high noise levels due to electronic sources such as shot noise, reset noise, amplifier noise, crosstalk, analogue to digital converters quantization and multipath light interference. Many noise sources in these cameras such as harmonic contamination, non-linearity, multipath interferences and light scattering are well investigated. In contrast, the effect of electronic jitter as a noise source in ranging cameras is barely studied. Jitter is defined to be any timing movement with reference to an ideal signal. An investigation of the effect of jitter on range imaging is important because timing errors potentially could cause errors in measuring phase, thus in range. The purpose of this research is to investigate the effect of jitter on range measurement in AMCW ToF range imaging. This is achieved by three main contributions: a development of a common algorithm for measurement of the jitter present in signals from depth cameras, secondly the proposal of a cost effective alternative method to measure jitter by using a software defined radio receiver, and finally an analysis of the influence of jitter on range measurement. Among the three contributions of this thesis, first, an algorithm for jitter extraction of a signal without access to a reference clock signal is proposed. The proposed algorithm is based upon Fourier analysis with signal processing techniques and it can be used for real time jitter extraction on a modulated signal with any kind of shape (sinusoidal, triangular, rectangular). The method is used to measure the amount of jitter in the light signals of two AMCW ToF range imaging cameras, namely, MESA Imaging SwissRanger 4000 and SoftKinetic DepthSense 325. Periodic and random jitter were found to be present in the light sources of both cameras with the MESA camera notably worse with random jitter of (159.6 +/- 0.1) ps RMS in amplitude. Next, in a novel approach, an inexpensive software defined radio (SDR) USB dongle is used with the proposed algorithm to extract the jitter in the light signal of the above two ToF cameras. This is a cost effective alternative to the expensive real-time medium speed digital oscilloscope. However, it is shown that this method has some significant limitations, (1) it can measure the jitter only up to half of the intermediate-frequency obtained from the down shift of the amplified radio frequency with the local oscillator which is less than the Nyquist frequency of the dongle and (2) if the number of samples per cycle captured from this dongle is not sufficient then the jitter extraction does not succeed since the signal is not properly (smoothly) represented. Finally, the periodic and random jitter influence on range measurements made with AMCW range imaging cameras are studied. An analytical model for the periodic jitter on the range measurements under the heterodyne and homodyne operations in AMCW ToF range imaging cameras is obtained in the frequency domain. The analytical model is tested through simulated data with various parameters in the system. The product of angular modulation frequency of the camera and the amplitude of the periodic jitter is a characteristic parameter for the phase error due to the presence of periodic jitter. We found that for currently available AMCW cameras (modulation frequency less than 100 MHz), neither periodic nor random jitter has a measurable effect on range measurement. But with modulation frequency increases and integration period decreases likely in the near future, periodic jitter may have a measurable detection affect on ranging. The influence of random jitter is also investigated by obtaining an analytical model based on stochastic calculus by using fundamental statistics and Fourier analysis. It is assumed that the random jitter follows the Gaussian distribution. Monte Carlo simulation is performed on the model obtained for a 1 ms integration period. We found increasing the modulation frequency above approximately 400 MHz with random jitter of 140 ps has a measurable affect on ranging

Capacitive sensing with a fluorescent lamp

Thesis (M. Eng.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2007.Includes bibliographical references (p. 163-167).This work presents a modified fluorescent lamp that can be used as a capacitive sensing system. The lamp sensor measures changes in the electric fields emitted from the fluorescent bulbs in order to deduce the presence and motion of conductive and dielectric objects below the lamp. The prototype lamp sensor demonstrated a detection range of 10 feet for the presence and motion of a human below the lamp. Potential applications of the lamp sensor include proximity detection, motion sensing, security monitoring, activity level monitoring for power management and control and metal or dangerous substance detection.by John Jacob Cooley.M.Eng