Virtual Reality to Simulate Visual Tasks for Robotic Systems

Virtual reality (VR) can be used as a tool to analyze the interactions between the visual system of a robotic agent and the environment, with the aim of designing the algorithms to solve the visual tasks necessary to properly behave into the 3D world. The novelty of our approach lies in the use of the VR as a tool to simulate the behavior of vision systems. The visual system of a robot (e.g., an autonomous vehicle, an active vision system, or a driving assistance system) and its interplay with the environment can be modeled through the geometrical relationships between the virtual stereo cameras and the virtual 3D world. Differently from conventional applications, where VR is used for the perceptual rendering of the visual information to a human observer, in the proposed approach, a virtual world is rendered to simulate the actual projections on the cameras of a robotic system. In this way, machine vision algorithms can be quantitatively validated by using the ground truth data provided by the knowledge of both the structure of the environment and the vision system

High-quality dense stereo vision for whole body imaging and obesity assessment

textThe prevalence of obesity has necessitated developing safe and convenient tools for timely assessing and monitoring this condition for a broad range of population. Three-dimensional (3D) body imaging has become a new mean for obesity assessment. Moreover, it generates body shape information that is meaningful for fitness, ergonomics, and personalized clothing. In the previous work of our lab, we developed a prototype active stereo vision system that demonstrated a potential to fulfill this goal. But the prototype required four computer projectors to cast artificial textures on the body which facilitate the stereo-matching on texture-deficient images (e.g., skin). This decreases the mobility of the system when used to collect a large population data. In addition, the resolution of the generated 3D~images is limited by both cameras and projectors available during the project. The study reported in this dissertation highlights our continued effort in improving the capability of 3Dbody imaging through simplified hardware for passive stereo and advanced computation techniques. The system utilizes high-resolution single-lens reflex (SLR) cameras, which became widely available lately, and is configured in a two-stance design to image the front and back surfaces of a person. A total of eight cameras are used to form four pairs of stereo units. Each unit covers a quarter of the body surface. The stereo units are individually calibrated with a specific pattern to determine cameras' intrinsic and extrinsic parameters for stereo matching. The global orientation and position of each stereo unit within a common world coordinate system is calculated through a 3Dregistration step. The stereo calibration and 3Dregistration procedures do not need to be repeated for a deployed system if the cameras' relative positions have not changed. This property contributes to the portability of the system, and tremendously alleviates the maintenance task. The image acquisition time is around two seconds for a whole-body capture. The system works in an indoor environment with a moderate ambient light. Advanced stereo computation algorithms are developed by taking advantage of high-resolution images and by tackling the ambiguity problem in stereo matching. A multi-scale, coarse-to-fine matching framework is proposed to match large-scale textures at a low resolution and refine the matched results over higher resolutions. This matching strategy reduces the complexity of the computation and avoids ambiguous matching at the native resolution. The pixel-to-pixel stereo matching algorithm follows a classic, four-step strategy which consists of matching cost computation, cost aggregation, disparity computation and disparity refinement. The system performance has been evaluated on mannequins and human subjects in comparison with other measurement methods. It was found that the geometrical measurements from reconstructed 3Dbody models, including body circumferences and whole volume, are highly repeatable and consistent with manual and other instrumental measurements (CV 0.99). The agreement of percent body fat (%BF) estimation on human subjects between stereo and dual-energy X-ray absorptiometry (DEXA) was found to be improved over the previous active stereo system, and the limits of agreement with 95% confidence were reduced by half. Our achieved %BF estimation agreement is among the lowest ones of other comparative studies with commercialized air displacement plethysmography (ADP) and DEXA. In practice, %BF estimation through a two-component model is sensitive to body volume measurement, and the estimation of lung volume could be a source of variation. Protocols for this type of measurement should still be created with an awareness of this factor.Biomedical Engineerin

Sensor development for estimation of biomass yield applied to Miscanthus Giganteus

Precision Agriculture technologies such as yield monitoring have been available for traditional field crops for decades. However, there are currently none available for energy crops such as Miscanthus Giganteus (MxG), switch grass, and sugar cane. The availability of yield monitors would allow better organization and scheduling of harvesting operations. In addition, the real-time yield data would allow adaptive speed control of a harvester to optimize performance. A yield monitor estimates a total amount of biomass per coverage area in kg/m2 as a function of location. However, for herbaceous type crops such as MxG and switchgrass, directly measuring the biomass entering a harvester in the field is complicated and impractical. Therefore, a novel yield monitoring system was proposed. The approach taken was to employ an indirect measure by determining a volume of biomass entering the harvester as a function of time. The volume can be obtained by multiplying the diameter related cross-sectional area, the height and the crop density of MxG. Subsequently, this volume is multiplied by an assumed constant, material density of the crop, which results in a mass flow per unit of time. To determine the coverage area, typically the width of the cutting device is multiplied by the machine speed to give the coverage area per unit of time. The ratio between the mass flow and coverage area is now the yield per area, and adding GPS geo-references the yield. To measure the height of MxG stems, a light detection and ranging (LIDAR) sensor based height measurement approach was developed. The LIDAR was applied to scan to the MxG vertically. Two measurement modes: static and dynamic, were designed and tested. A geometrical MxG height measurement model was developed and analyzed to obtain the resolution of the height measurement. An inclination correction method was proposed to correct errors caused by the uneven ground surface. The relationship between yield and stem height was discussed and analyzed, resulting in a linear relationship. To estimate the MxG stem diameter, two types of sensors were developed and evaluated. Firstly, a LIDAR based diameter sensor was designed and tested. The LIDAR was applied to scan MxG stems horizontally. A measurement geometry model of the LIDAR was developed to determine the region of interest. An angle continuity based pre-grouping algorithm was applied to group the raw data from the LIDAR. Based on the analysis of the presentation of MxG stems in the LIDAR data, a fuzzy clustering technique was developed to identify the MxG stems within the clusters. The diameter was estimated based on the clustering result. Four types of clustering techniques were compared. Based on their performances, the Gustafson - Kessel Clustering algorithm was selected. A drawback of the LIDAR based diameter sensor was that it could only be used for static diameter measurement. An alternative system based on a machine vision based diameter sensor, which supported the dynamic measurement, was applied. A binocular stereo vision based diameter sensor and a structured lighting-based monocular vision diameter estimation system were developed and evaluated in sequence. Both systems worked with structured lighting provided by a downward slanted laser sheet to provide detectable features in the images. An image segmentation based algorithm was developed to detect these features. These features were used to identify the MxG stems in both the binocular and monocular based systems. A horizontally covered length per pixel model was built and validated to extract the diameter information from images. The key difference between the binocular and monocular stereo vision systems was the approach to estimate the depth. For the binocular system, the depth information was obtained based on disparities of matched features in image pairs. The features were matched based on a pixel similarity in both one dimensional and two dimensional based image matching algorithm. In the monocular system, the depth was obtained by a geometry perspective model of the diameter sensor unit. The relationship between yield and stem diameter was discussed and analyzed. The result showed that the yield was more strongly dependent upon the stem height than diameter, and the relationship between yield and stem volume was linear. The crop density estimation was also based on the monocular stereo vision system. To predict the crop density, the geometry perspective model of the sensor unit was further analyzed to calculate the coverage area of the sensor. A Monte Carlo model based method was designed to predict the number of occluded MxG stems based on the number of visible MxG stems in images. The results indicated that the yield has a linear relationship with the number of stems with a zero intercept and the average individual mass as the coefficient. All sensors were evaluated in the field during the growing seasons of 2009, 2010 and 2011 using manually measured parameters (height, diameter and crop density) as references. The results showed that the LIDAR based height sensor achieved an accuracy of 92% (0.3m error) to 98.2% (0.06m error) in static height measurements and accuracy of 93.5% (0.22m error) to 98.5% (0.05m error) in dynamic height measurements. For the diameter measurements, the machine vision based sensors showed a more accurate result than the LIDAR based sensor. The binocular stereo vision based and monocular vision based diameter measurement achieved an accuracy of 93.1% and 93.5% for individual stem diameter estimation, and 99.8% and 99.9% for average stem diameter estimation, while the achieved accuracy of LIDAR based sensor for average stem diameter estimation was 92.5%. Among three stem diameter sensors, the monocular vision based sensor was recommended due to its higher accuracy and lower cost in both device and computation. The achieved accuracy of machine vision based crop density measurement was 92.2%

Measurement of crosstalk in stereoscopic display systems used for vision research

Studying binocular vision requires precise control over the stimuli presented to the left and right eyes. A popular technique is to segregate signals either temporally (frame interleaving), spectrally (using coloured filters) or through light polarization. None of these segregation methods achieves perfect isolation, and so a degree of ‘crosstalk’ is usually apparent in which signals intended for one eye are faintly visible to the other eye. Previous studies have reported crosstalk values mostly for consumer-grade systems. Here we measure crosstalk for eight systems, many of which are intended for use in vision research. We provide benchmark crosstalk values, report a negative crosstalk effect in some LCD-based systems, and give guidelines for dealing with crosstalk in different experimental paradigms

Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery

One of the main challenges for computer-assisted surgery (CAS) is to determine the intra-opera- tive morphology and motion of soft-tissues. This information is prerequisite to the registration of multi-modal patient-specific data for enhancing the surgeon’s navigation capabilites by observ- ing beyond exposed tissue surfaces and for providing intelligent control of robotic-assisted in- struments. In minimally invasive surgery (MIS), optical techniques are an increasingly attractive approach for in vivo 3D reconstruction of the soft-tissue surface geometry. This paper reviews the state-of-the-art methods for optical intra-operative 3D reconstruction in laparoscopic surgery and discusses the technical challenges and future perspectives towards clinical translation. With the recent paradigm shift of surgical practice towards MIS and new developments in 3D opti- cal imaging, this is a timely discussion about technologies that could facilitate complex CAS procedures in dynamic and deformable anatomical regions