Methods for Augmented Reality E-commerce

A new type of e-commerce system and related techniques are presented in this dissertation that customers of this type of e-commerce could visually bring product into their physical environment for interaction. The development and user study of this e-commerce system are provided. A new modeling method, which recovers 3D model directly from 2D photos without knowing camera information, is also presented to reduce the modeling cost of this new type of e-commerce. Also an immersive AR environment with GPU based occlusion is also presented to improve the rendering and usability of AR applications. Experiment results and data show the validity of these new technologies

INTERFACE DESIGN FOR A VIRTUAL REALITY-ENHANCED IMAGE-GUIDED SURGERY PLATFORM USING SURGEON-CONTROLLED VIEWING TECHNIQUES

Initiative has been taken to develop a VR-guided cardiac interface that will display and deliver information without affecting the surgeons’ natural workflow while yielding better accuracy and task completion time than the existing setup. This paper discusses the design process, the development of comparable user interface prototypes as well as an evaluation methodology that can measure user performance and workload for each of the suggested display concepts. User-based studies and expert recommendations are used in conjunction to es­ tablish design guidelines for our VR-guided surgical platform. As a result, a better understanding of autonomous view control, depth display, and use of virtual context, is attained. In addition, three proposed interfaces have been developed to allow a surgeon to control the view of the virtual environment intra-operatively. Comparative evaluation of the three implemented interface prototypes in a simulated surgical task scenario, revealed performance advantages for stereoscopic and monoscopic biplanar display conditions, as well as the differences between three types of control modalities. One particular interface prototype demonstrated significant improvement in task performance. Design recommendations are made for this interface as well as the others as we prepare for prospective development iterations

On the Interplay between Data Overlay and Real-World Context using See-through Displays

The recent availability of affordable see-through wearable displays has fostered the development of several new interfaces and applications. Some of them take the augmented reality path, by seeking the blending of physical objects with overlaid 3D models or textual information. Some, on the other hand, are much simpler and follow a rather basic paradigm where the spatial integration between real world and data overlay is dropped. This is the case, for instance, with most applications based on Google Glass hardware, where textual data and images partially share the field of view of the user, but are not pinpointed to physical features. This is a rather important difference, since it marks the shift from a cooperative see-through mode, that characterizes proper augmented reality, to a competitive overlay, where the user attention is actually contended between real objects and displayed data. To this end, the user focus must continuously shift from one context to the other, possibly leading to both reduced productivity and usage strain. With this paper we are addressing exactly this issue. Specifically, we are assessing the role of different properties of the overlay, including the level of occlusion, the depth of the data layer, the position of the view frustum and the impact of stereo vision. Such study has been implemented by mean of a real-world evaluation which has been performed using a general purpose see-through device in a practical application scenario.The recent availability of affordable see-through wearable displays has fostered the development of several new interfaces and applications. Some of them take the augmented reality path, by seeking the blending of physical objects with overlaid 3D models or textual information. Some, on the other hand, are much simpler and follow a rather basic paradigm where the spatial integration between real world and data overlay is dropped. This is the case, for instance, with most applications based on Google Glass hardware, where textual data and images partially share the field of view of the user, but are not pinpointed to physical features. This is a rather important difference, since it marks the shift from a cooperative see-through mode, that characterizes proper augmented reality, to a competitive overlay, where the user attention is actually contended between real objects and displayed data. To this end, the user focus must continuously shift from one context to the other, possibly leading to both reduced productivity and usage strain. With this paper we are addressing exactly this issue. Specifically, we are assessing the role of different properties of the overlay, including the level of occlusion, the depth of the data layer, the position of the view frustum and the impact of stereo vision. Such study has been implemented by mean of a real-world evaluation which has been performed using a general purpose see-through device in a practical application scenario

Development of teat sensing system for automated milking

Robotic application of milking cups to the udder of a cow in a rotary high capacity group milking system is a major challenge in automated milking. Application time and reliability are the main constraints. Manual application by an operator of a rotary carousel is of the order of 10 seconds and 100% reliable. In existing non-rotary milking machines, the cups are applied to each teat individually and the process can take up to two minutes. In order to achieve a more rapid simultaneous application of the four cups, the three dimensional locations of the four teats must be known in real time. In this thesis, a multimodal vision system combining optical stereovision and thermal imaging is developed. The overall system is evaluated from the point of view of accuracy and robustness. Laboratory tests have shown that stereovision can rapidly locate teat three dimensional position coordinates, however robust identification of the teats is required. It is shown that this may be achieved using thermal imaging to isolate teats from background objects due to their elevated temperature profile. Further development is necessary to overcome specific situations such as overlapping teats

Moving object detection and segmentation in urban environments from a moving platform

This paper proposes an effective approach to detect and segment moving objects from two time-consecutive stereo frames, which leverages the uncertainties in camera motion estimation and in disparity computation. First, the relative camera motion and its uncertainty are computed by tracking and matching sparse features in four images. Then, the motion likelihood at each pixel is estimated by taking into account the ego-motion uncertainty and disparity in computation procedure. Finally, the motion likelihood, color and depth cues are combined in the graph-cut framework for moving object segmentation. The efficiency of the proposed method is evaluated on the KITTI benchmarking datasets, and our experiments show that the proposed approach is robust against both global (camera motion) and local (optical flow) noise. Moreover, the approach is dense as it applies to all pixels in an image, and even partially occluded moving objects can be detected successfully. Without dedicated tracking strategy, our approach achieves high recall and comparable precision on the KITTI benchmarking sequences.This work was carried out within the framework of the Equipex ROBOTEX (ANR-10- EQPX-44-01). Dingfu Zhou was sponsored by the China Scholarship Council for 3.5 year’s PhD study at HEUDIASYC laboratory in University of Technology of Compiegne

Anomaly detection in urban drainage with stereovision

This work introduces RADIUS, a framework for anomaly detection in sewer pipes using stereovision. The framework employs three-dimensional geometry reconstruction from stereo vision, followed by statistical modeling of the geometry with a generic pipe model. The framework is designed to be compatible with existing workflows for sewer pipe defect detection, as well as to provide opportunities for machine learning implementations in the future. We test the framework on 48 image sets of 26 sewer pipes in different conditions collected in the lab. Of these 48 image sets, 5 could not be properly reconstructed in three dimensions due to insufficient stereo matching. The surface fitting and anomaly detection performed well: a human-graded defect severity score had a moderate, positive Pearson correlation of 0.65 with our calculated anomaly scores, making this a promising approach to automated defect detection in urban drainage

Virtual Reality to Simulate Visual Tasks for Robotic Systems

Virtual reality (VR) can be used as a tool to analyze the interactions between the visual system of a robotic agent and the environment, with the aim of designing the algorithms to solve the visual tasks necessary to properly behave into the 3D world. The novelty of our approach lies in the use of the VR as a tool to simulate the behavior of vision systems. The visual system of a robot (e.g., an autonomous vehicle, an active vision system, or a driving assistance system) and its interplay with the environment can be modeled through the geometrical relationships between the virtual stereo cameras and the virtual 3D world. Differently from conventional applications, where VR is used for the perceptual rendering of the visual information to a human observer, in the proposed approach, a virtual world is rendered to simulate the actual projections on the cameras of a robotic system. In this way, machine vision algorithms can be quantitatively validated by using the ground truth data provided by the knowledge of both the structure of the environment and the vision system