Towards Intelligent Telerobotics: Visualization and Control of Remote Robot

Human-machine cooperative or co-robotics has been recognized as the next generation of robotics. In contrast to current systems that use limited-reasoning strategies or address problems in narrow contexts, new co-robot systems will be characterized by their flexibility, resourcefulness, varied modeling or reasoning approaches, and use of real-world data in real time, demonstrating a level of intelligence and adaptability seen in humans and animals. The research I focused is in the two sub-field of co-robotics: teleoperation and telepresence. We firstly explore the ways of teleoperation using mixed reality techniques. I proposed a new type of display: hybrid-reality display (HRD) system, which utilizes commodity projection device to project captured video frame onto 3D replica of the actual target surface. It provides a direct alignment between the frame of reference for the human subject and that of the displayed image. The advantage of this approach lies in the fact that no wearing device needed for the users, providing minimal intrusiveness and accommodating users eyes during focusing. The field-of-view is also significantly increased. From a user-centered design standpoint, the HRD is motivated by teleoperation accidents, incidents, and user research in military reconnaissance etc. Teleoperation in these environments is compromised by the Keyhole Effect, which results from the limited field of view of reference. The technique contribution of the proposed HRD system is the multi-system calibration which mainly involves motion sensor, projector, cameras and robotic arm. Due to the purpose of the system, the accuracy of calibration should also be restricted within millimeter level. The followed up research of HRD is focused on high accuracy 3D reconstruction of the replica via commodity devices for better alignment of video frame. Conventional 3D scanner lacks either depth resolution or be very expensive. We proposed a structured light scanning based 3D sensing system with accuracy within 1 millimeter while robust to global illumination and surface reflection. Extensive user study prove the performance of our proposed algorithm. In order to compensate the unsynchronization between the local station and remote station due to latency introduced during data sensing and communication, 1-step-ahead predictive control algorithm is presented. The latency between human control and robot movement can be formulated as a linear equation group with a smooth coefficient ranging from 0 to 1. This predictive control algorithm can be further formulated by optimizing a cost function. We then explore the aspect of telepresence. Many hardware designs have been developed to allow a camera to be placed optically directly behind the screen. The purpose of such setups is to enable two-way video teleconferencing that maintains eye-contact. However, the image from the see-through camera usually exhibits a number of imaging artifacts such as low signal to noise ratio, incorrect color balance, and lost of details. Thus we develop a novel image enhancement framework that utilizes an auxiliary color+depth camera that is mounted on the side of the screen. By fusing the information from both cameras, we are able to significantly improve the quality of the see-through image. Experimental results have demonstrated that our fusion method compares favorably against traditional image enhancement/warping methods that uses only a single image

Improving SLI Performance in Optically Challenging Environments

The construction of 3D models of real-world scenes using non-contact methods is an important problem in computer vision. Some of the more successful methods belong to a class of techniques called structured light illumination (SLI). While SLI methods are generally very successful, there are cases where their performance is poor. Examples include scenes with a high dynamic range in albedo or scenes with strong interreflections. These scenes are referred to as optically challenging environments. The work in this dissertation is aimed at improving SLI performance in optically challenging environments. A new method of high dynamic range imaging (HDRI) based on pixel-by-pixel Kalman filtering is developed. Using objective metrics, it is show to achieve as much as a 9.4 dB improvement in signal-to-noise ratio and as much as a 29% improvement in radiometric accuracy over a classic method. Quality checks are developed to detect and quantify multipath interference and other quality defects using phase measuring profilometry (PMP). Techniques are established to improve SLI performance in the presence of strong interreflections. Approaches in compressed sensing are applied to SLI, and interreflections in a scene are modeled using SLI. Several different applications of this research are also discussed

Clasificación automática de anomalías asociadas con ausencia de información en superficies tridimensionales de objetos de forma libre

En este trabajo se propone un método computacional para clasificar anomalías relacionadas con ausencia de información sobre modelos tridimensionales de forma libre. Para ello, se hizo una exploración descriptiva de las propiedades geométricas globales y locales de las anomalías y una evaluación posterior de distintos métodos de clasificación utilizados en visión artificial y aplicaciones de reconstrucción tridimensional. El método propuesto logra un nivel de clasificación cercano al 90% y un tiempo de ejecución de alrededor de 100 milisegundos. Restringir la clasificación de acuerdo a la aplicación en específico se propone como trabajo futuro./Abstract. In this work it is proposed a computational method to classify anomalies related with information absence over free-form tridimensional models. For that, it was made a descriptive exploration of global and local geometric properties of anomalies and a posterior evaluation of different classification methods widely used in artificial vision and tridimensional reconstruction applications. The proposed method achieved a classification level near to 90% and an execution time near to 100 miliseconds. Constrain classification according to the specific application is suggested as future work.Maestrí

Dense RGB-D SLAM and object localisation for robotics and industrial applications

Dense reconstruction and object localisation are two critical steps in robotic and industrial applications. The former entails a joint estimation of camera egomotion and the structure of the surrounding environment, also known as Simultaneous Localisation and Mapping (SLAM), and the latter aims to locate the object in the reconstructed scenes. This thesis addresses the challenges of dense SLAM with RGB-D cameras and object localisation towards robotic and industrial applications. Camera drift is an essential issue in camera egomotion estimation. Due to the accumulated error in camera pose estimation, the estimated camera trajectory is inaccurate, and the reconstruction of the environment is inconsistent. This thesis analyses camera drift in SLAM under the probabilistic inference framework and proposes an online map fusion strategy with standard deviation estimation based on frame-to-model camera tracking. The camera pose is estimated by aligning the input image with the global map model, and the global map merges the information in the images by weighted fusion with standard deviation modelling. In addition, a pre-screening step is applied before map fusion to preclude the adverse effect of accumulated errors and noises on camera egomotion estimation. Experimental results indicated that the proposed method mitigates camera drift and improves the global consistency of camera trajectories. Another critical challenge for dense RGB-D SLAM in industrial scenarios is to handle mechanical and plastic components that usually have reflective and shiny surfaces. Photometric alignment in frame-to-model camera tracking tends to fail on such objects due to the inconsistency in intensity patterns of the images and the global map model. This thesis addresses this problem and proposes RSO-SLAM, namely a SLAM approach to reflective and shiny object reconstruction. RSO-SLAM adopts frame-to-model camera tracking and combines local photometric alignment and global geometric registration. This study revealed the effectiveness and excellent performance of the proposed RSO-SLAM on both plastic and metallic objects. In addition, a case study involving the cover of a electric vehicle battery with metallic surface demonstrated the superior performance of the RSO-SLAM approach in the reconstruction of a common industrial product. With the reconstructed point cloud model of the object, the problem of object localisation is tackled as point cloud registration in the thesis. Iterative Closest Point (ICP) is arguably the best-known method for point cloud registration, but it is susceptible to sub-optimal convergence due to the multimodal solution space. This thesis proposes the Bees Algorithm (BA) enhanced with the Singular Value Decomposition (SVD) procedure for point cloud registration. SVD accelerates the speed of the local search of the BA, helping the algorithm to rapidly identify the local optima. It also enhances the precision of the obtained solutions. At the same time, the global outlook of the BA ensures adequate exploration of the whole solution space. Experimental results demonstrated the remarkable performance of the SVD-enhanced BA in terms of consistency and precision. Additional tests on noisy datasets demonstrated the robustness of the proposed procedure to imprecision in the models

Metodología para la corrección de huecos en imágenes de rango basada en conocimiento del dominio

El proceso de reconstrucción de formas implica estimar una representación matemática de la geometría de un objeto a partir de un conjunto de medidas conocidas y adquiridas de dicho objeto. En relación con las muestras adquiridas, existen numerosas dificultades relacionadas con el proceso de adquisición; la topología del objeto, la estructura del sensor, las características físicas del material del objeto, las condiciones de iluminación, entre otras dificultades. Éstas constituyen la fuente principal de generación de anomalías dentro de los datos muestreados, que afectan la estimación de representaciones precisas de los objetos. En esta investigación, dichas anomalías se clasifican en tres grupos, los cuales son: Presencia de Ruido, Formación de Huecos y Redundancia de Información. Para tratar con estas dificultades, se aplica típicamente una etapa de corrección denominada Etapa de Integración. La importancia de esta etapa consiste en disminuir el efecto de estas anomalías en el nivel de precisión de la representación final. Cualquiera que sea el tipo de anomalía, corresponde a un área amplia de estudios, con diversas y numerosas técnicas propuestas. A pesar de esto, el tratamiento de estas anomalías, es un área de continuo mejoramiento y se considera aún un problema abierto para la comunidad científica. La dificultad radica fundamentalmente, en algunos casos en que la naturaleza exacta de la fuente de estas anomalías es desconocida, compleja de modelar, o sencillamente porque una solución a estos problemas necesariamente tendrá un nivel de incertidumbre. Lo anterior, ha generado la necesidad de desarrollar procedimientos de corrección asistidos por el usuario. Aunque diferentes enfoques geométricos y matemáticos han sido propuestos, sus debilidades radican principalmente en que sus aplicaciones son de dominio limitado debido a su poca flexibilidad de adaptación a objetos con diferentes topologías. En este documento, se propone una aproximación metodológica para la corrección de anomalías asociadas con la ausencia de información. Cada discontinuidad es clasificada como reparable o no, de acuerdo a una aproximación estimada de su irregularidad, y reparadas mediante el uso de un modelo de corrección bayesiano./Abstract. The representation of the geometry of an object from a set of measures from the object. The samples have numerous difficulties associated with the acquisition process, the topology of the object, the sensor structure, the physical characteristics of the material, illumination conditions, among other difficulties. These are the main sources of defect generation in the sampled data, which affect the estimate of accurate representations of the objects. In this research, these anomalies are classified into three groups, which are: Presence of Noise, Redundancy and Holes. To deal with these difficulties, is typically applied correction stage called stage of Integration. The importance of this stage is to reduce the effect of these anomalies in the precision of the final representation. Whatever the type of anomaly, it corresponds to a broad area of study with various and numerous proposed techniques. Despite this, treatment of these anomalies is an area of continuous improvement and is still considered an open problem for the scientific community. The difficulty lies mainly, that in some cases where the exact nature of the source of these anomalies is unknown, modeling complex, or simply, because one solution to these problems necessarily has a level of uncertainty. This has generated the need to develop procedures for user-assisted correction. Different approaches and mathematical geometry have been proposed. Their main weakness lies in their domain applications are limited due to little flexibility to adapt to objects with no restrictions on the topologies. In this paper, we propose a methodological approach for the correction of anomalies associated with the loss of information. Each discontinuity is classified as repairable or not, according to an estimate approximation of its irregularity, and repaired by using a Bayesian correction model.Doctorad

3D modeling of optically challenging objects

We present a system for constructing 3D models of real-world objects with optically challenging surfaces. The system utilizes a new range imaging concept called multipeak range imaging, which stores multiple candidates of range measurements for each point on the object surface. The multiple measurements include the erroneous range data caused by various surface properties that are not ideal for structured-light range sensing. False measurements generated by spurious reflections are eliminated by applying a series of constraint tests. The constraint tests based on local surface and local sensor visibility are applied first to individual range images. The constraint tests based on global consistency of coordinates and visibility are then applied to all range images acquired from different viewpoints. We show the effectiveness of our method by constructing 3D models of five different optically challenging objects. To evaluate the performance of the constraint tests and to examine the effects of the parameters used in the constraint tests, we acquired the ground-truth data by painting those objects to suppress the surface-related properties that cause difficulties in range sensing. Experimental results indicate that our method significantly improves upon the traditional methods for constructing reliable 3D models of optically challenging objects