Real-time spatial modeling to detect and track resources on construction sites

For more than 10 years the U.S. construction industry has experienced over 1,000 fatalities annually. Many fatalities may have been prevented had the individuals and equipment involved been more aware of and alert to the physical state of the environment around them. Awareness may be improved by automatic 3D (three-dimensional) sensing and modeling of the job site environment in real-time. Existing 3D modeling approaches based on range scanning techniques are capable of modeling static objects only, and thus cannot model in real-time dynamic objects in an environment comprised of moving humans, equipment, and materials. Emerging prototype 3D video range cameras offer another alternative by facilitating affordable, wide field of view, automated static and dynamic object detection and tracking at frame rates better than 1Hz (real-time). This dissertation presents an imperical work and methodology to rapidly create a spatial model of construction sites and in particular to detect, model, and track the position, dimension, direction, and velocity of static and moving project resources in real-time, based on range data obtained from a three-dimensional video range camera in a static or moving position. Existing construction site 3D modeling approaches based on optical range sensing technologies (laser scanners, rangefinders, etc.) and 3D modeling approaches (dense, sparse, etc.) that offered potential solutions for this research are reviewed. The choice of an emerging sensing tool and preliminary experiments with this prototype sensing technology are discussed. These findings led to the development of a range data processing algorithm based on three-dimensional occupancy grids which is demonstrated in detail. Testing and validation of the proposed algorithms have been conducted to quantify the performance of sensor and algorithm through extensive experimentation involving static and moving objects. Experiments in indoor laboratory and outdoor construction environments have been conducted with construction resources such as humans, equipment, materials, or structures to verify the accuracy of the occupancy grid modeling approach. Results show that modeling objects and measuring their position, dimension, direction, and speed had an accuracy level compatible to the requirements of active safety features for construction. Results demonstrate that video rate 3D data acquisition and analysis of construction environments can support effective detection, tracking, and convex hull modeling of objects. Exploiting rapidly generated three-dimensional models for improved visualization, communications, and process control has inherent value, broad application, and potential impact, e.g. as-built vs. as-planned comparison, condition assessment, maintenance, operations, and construction activities control. In combination with effective management practices, this sensing approach has the potential to assist equipment operators to avoid incidents that result in reduce human injury, death, or collateral damage on construction sites.Civil, Architectural, and Environmental Engineerin

Analysis of the performance of a polarized LiDAR imager in fog

This paper focuses on exploring ways to improve the performance of LiDAR imagers through fog. One of the known weaknesses of LiDAR technology is the lack of tolerance to adverse environmental conditions, such as the presence of fog, which hampers the future development of LiDAR in several markets. Within this paper, a LiDAR unit is designed and constructed to be able to apply temporal and polarimetric discrimination for detecting the number of signal photons received with detailed control of its temporal and spatial distribution under co-polarized and cross-polarized configurations. The system is evaluated using different experiments in a macro-scale fog chamber under controlled fog conditions. Using the complete digitization of the acquired signals, we analyze the natural light media response, to see that due to its characteristics it could be directly filtered out. Moreover, we confirm that there exists a polarization memory effect, which, by using a polarimetric cross-configuration detector, allows improvement of object detection in point clouds. These results are useful for applications related to computer vision, in fields like autonomous vehicles or outdoor surveillance where many variable types of environmental conditions may be present.Agència de Gestió d’Ajuts Universitaris i de Recerca (2021FI_B2 00068, 2021FI_B2 00077); DSTL (DSTLX1000145661); Ministerio de Ciencia e Innovación (PDC2021-121038-I00, PID2020-119484RB-I00).Peer ReviewedPostprint (author's final draft

Pixel-Accurate Depth Evaluation in Realistic Driving Scenarios

This work introduces an evaluation benchmark for depth estimation and completion using high-resolution depth measurements with angular resolution of up to 25" (arcsecond), akin to a 50 megapixel camera with per-pixel depth available. Existing datasets, such as the KITTI benchmark, provide only sparse reference measurements with an order of magnitude lower angular resolution - these sparse measurements are treated as ground truth by existing depth estimation methods. We propose an evaluation methodology in four characteristic automotive scenarios recorded in varying weather conditions (day, night, fog, rain). As a result, our benchmark allows us to evaluate the robustness of depth sensing methods in adverse weather and different driving conditions. Using the proposed evaluation data, we demonstrate that current stereo approaches provide significantly more stable depth estimates than monocular methods and lidar completion in adverse weather. Data and code are available at https://github.com/gruberto/PixelAccurateDepthBenchmark.git.Comment: 3DV 201

Long-range imaging LiDAR with multiple denoising technologies

The ability to capture and record high-resolution images over long distances is essential for a wide range of applications, including connected and autonomous vehicles, defense and security operations, as well as agriculture and mining industries. Here, we demonstrate a self-assembled bistatic long-range imaging LiDAR system. Importantly, to achieve high signal-to-noise ratio (SNR) data, we employed a comprehensive suite of denoising methods including temporal, spatial, spectral, and polarization filtering. With the aid of these denoising technologies, our system has been validated to possess the capability of imaging under various complex usage conditions. In terms of distance performance, the test results achieved ranges of over 4000 m during daylight with clear weather, 19,200 m at night, 6700 m during daylight with haze, and 2000 m during daylight with rain. Additionally, it offers an angular resolution of 0.01 mrad. These findings demonstrate the potential to offer comprehensive construction strategies and operational methodologies to individuals seeking long-range LiDAR data

Resilient Perception for Outdoor Unmanned Ground Vehicles

This thesis promotes the development of resilience for perception systems with a focus on Unmanned Ground Vehicles (UGVs) in adverse environmental conditions. Perception is the interpretation of sensor data to produce a representation of the environment that is necessary for subsequent decision making. Long-term autonomy requires perception systems that correctly function in unusual but realistic conditions that will eventually occur during extended missions. State-of-the-art UGV systems can fail when the sensor data are beyond the operational capacity of the perception models. The key to resilient perception system lies in the use of multiple sensor modalities and the pre-selection of appropriate sensor data to minimise the chance of failure. This thesis proposes a framework based on diagnostic principles to evaluate and preselect sensor data prior to interpretation by the perception system. Image-based quality metrics are explored and evaluated experimentally using infrared (IR) and visual cameras onboard a UGV in the presence of smoke and airborne dust. A novel quality metric, Spatial Entropy (SE), is introduced and evaluated. The proposed framework is applied to a state-of-the-art Visual-SLAM algorithm combining visual and IR imaging as a real-world example. An extensive experimental evaluation demonstrates that the framework allows for camera-based localisation that is resilient to a range of low-visibility conditions when compared to other methods that use a single sensor or combine sensor data without selection. The proposed framework allows for a resilient localisation in adverse conditions using image data but also has significant potential to benefit many perception applications. Employing multiple sensing modalities along with pre-selection of appropriate data is a powerful method to create resilient perception systems by anticipating and mitigating errors. The development of such resilient perception systems is a requirement for next-generation outdoor UGVs

Fruit Detection and Tree Segmentation for Yield Mapping in Orchards

Accurate information gathering and processing is critical for precision horticulture, as growers aim to optimise their farm management practices. An accurate inventory of the crop that details its spatial distribution along with health and maturity, can help farmers efficiently target processes such as chemical and fertiliser spraying, crop thinning, harvest management, labour planning and marketing. Growers have traditionally obtained this information by using manual sampling techniques, which tend to be labour intensive, spatially sparse, expensive, inaccurate and prone to subjective biases. Recent advances in sensing and automation for field robotics allow for key measurements to be made for individual plants throughout an orchard in a timely and accurate manner. Farmer operated machines or unmanned robotic platforms can be equipped with a range of sensors to capture a detailed representation over large areas. Robust and accurate data processing techniques are therefore required to extract high level information needed by the grower to support precision farming. This thesis focuses on yield mapping in orchards using image and light detection and ranging (LiDAR) data captured using an unmanned ground vehicle (UGV). The contribution is the framework and algorithmic components for orchard mapping and yield estimation that is applicable to different fruit types and orchard configurations. The framework includes detection of fruits in individual images and tracking them over subsequent frames. The fruit counts are then associated to individual trees, which are segmented from image and LiDAR data, resulting in a structured spatial representation of yield. The first contribution of this thesis is the development of a generic and robust fruit detection algorithm. Images captured in the outdoor environment are susceptible to highly variable external factors that lead to significant appearance variations. Specifically in orchards, variability is caused by changes in illumination, target pose, tree types, etc. The proposed techniques address these issues by using state-of-the-art feature learning approaches for image classification, while investigating the utility of orchard domain knowledge for fruit detection. Detection is performed using both pixel-wise classification of images followed instance segmentation, and bounding-box regression approaches. The experimental results illustrate the versatility of complex deep learning approaches over a multitude of fruit types. The second contribution of this thesis is a tree segmentation approach to detect the individual trees that serve as a standard unit for structured orchard information systems. The work focuses on trellised trees, which present unique challenges for segmentation algorithms due to their intertwined nature. LiDAR data are used to segment the trellis face, and to generate proposals for individual trees trunks. Additional trunk proposals are provided using pixel-wise classification of the image data. The multi-modal observations are fine-tuned by modelling trunk locations using a hidden semi-Markov model (HSMM), within which prior knowledge of tree spacing is incorporated. The final component of this thesis addresses the visual occlusion of fruit within geometrically complex canopies by using a multi-view detection and tracking approach. Single image fruit detections are tracked over a sequence of images, and associated to individual trees or farm rows, with the spatial distribution of the fruit counting forming a yield map over the farm. The results show the advantage of using multi-view imagery (instead of single view analysis) for fruit counting and yield mapping. This thesis includes extensive experimentation in almond, apple and mango orchards, with data captured by a UGV spanning a total of 5 hectares of farm area, over 30 km of vehicle traversal and more than 7,000 trees. The validation of the different processes is performed using manual annotations, which includes fruit and tree locations in image and LiDAR data respectively. Additional evaluation of yield mapping is performed by comparison against fruit counts on trees at the farm and counts made by the growers post-harvest. The framework developed in this thesis is demonstrated to be accurate compared to ground truth at all scales of the pipeline, including fruit detection and tree mapping, leading to accurate yield estimation, per tree and per row, for the different crops. Through the multitude of field experiments conducted over multiple seasons and years, the thesis presents key practical insights necessary for commercial development of an information gathering system in orchards

Advanced photon counting techniques for long-range depth imaging

The Time-Correlated Single-Photon Counting (TCSPC) technique has emerged as a candidate approach for Light Detection and Ranging (LiDAR) and active depth imaging applications. The work of this Thesis concentrates on the development and investigation of functional TCSPC-based long-range scanning time-of-flight (TOF) depth imaging systems. Although these systems have several different configurations and functions, all can facilitate depth profiling of remote targets at low light levels and with good surface-to-surface depth resolution. Firstly, a Superconducting Nanowire Single-Photon Detector (SNSPD) and an InGaAs/InP Single-Photon Avalanche Diode (SPAD) module were employed for developing kilometre-range TOF depth imaging systems at wavelengths of ~1550 nm. Secondly, a TOF depth imaging system at a wavelength of 817 nm that incorporated a Complementary Metal-Oxide-Semiconductor (CMOS) 32×32 Si-SPAD detector array was developed. This system was used with structured illumination to examine the potential for covert, eye-safe and high-speed depth imaging. In order to improve the light coupling efficiency onto the detectors, the arrayed CMOS Si-SPAD detector chips were integrated with microlens arrays using flip-chip bonding technology. This approach led to the improvement in the fill factor by up to a factor of 15. Thirdly, a multispectral TCSPC-based full-waveform LiDAR system was developed using a tunable broadband pulsed supercontinuum laser source which can provide simultaneous multispectral illumination, at wavelengths of 531, 570, 670 and ~780 nm. The investigated multispectral reflectance data on a tree was used to provide the determination of physiological parameters as a function of the tree depth profile relating to biomass and foliage photosynthetic efficiency. Fourthly, depth images were estimated using spatial correlation techniques in order to reduce the aggregate number of photon required for depth reconstruction with low error. A depth imaging system was characterised and re-configured to reduce the effects of scintillation due to atmospheric turbulence. In addition, depth images were analysed in terms of spatial and depth resolution

An Outlook into the Future of Egocentric Vision

What will the future be? We wonder! In this survey, we explore the gap between current research in egocentric vision and the ever-anticipated future, where wearable computing, with outward facing cameras and digital overlays, is expected to be integrated in our every day lives. To understand this gap, the article starts by envisaging the future through character-based stories, showcasing through examples the limitations of current technology. We then provide a mapping between this future and previously defined research tasks. For each task, we survey its seminal works, current state-of-the-art methodologies and available datasets, then reflect on shortcomings that limit its applicability to future research. Note that this survey focuses on software models for egocentric vision, independent of any specific hardware. The paper concludes with recommendations for areas of immediate explorations so as to unlock our path to the future always-on, personalised and life-enhancing egocentric vision.Comment: We invite comments, suggestions and corrections here: https://openreview.net/forum?id=V3974SUk1