Automatic facial expression tracking for 4D range scans

This paper presents a fully automatic approach of spatio-temporal facial expression tracking for 4D range scans without any manual interventions (such as specifying landmarks). The approach consists of three steps: rigid registration, facial model reconstruction, and facial expression tracking. A Scaling Iterative Closest Points (SICP) algorithm is introduced to compute the optimal rigid registration between a template facial model and a range scan with consideration of the scale problem. A deformable model, physically based on thin shells, is proposed to faithfully reconstruct the facial surface and texture from that range data. And then the reconstructed facial model is used to track facial expressions presented in a sequence of range scans by the deformable model

A comparison of hole-filling methods in 3D

This paper presents a review of the most relevant current techniques that deal with hole-filling in 3D models. Contrary to earlier reports, which approach mesh repairing in a sparse and global manner, the objective of this review is twofold. First, a specific and comprehensive review of hole-filling techniques (as a relevant part in the field of mesh repairing) is carried out. We present a brief summary of each technique with attention paid to its algorithmic essence, main contributions and limitations. Second, a solid comparison between 34 methods is established. To do this, we define 19 possible meaningful features and properties that can be found in a generic hole-filling process. Then, we use these features to assess the virtues and deficiencies of the method and to build comparative tables. The purpose of this review is to make a comparative hole-filling state-of-the-art available to researchers, showing pros and cons in a common framework.• Ministerio de Economía y Competitividad: Proyecto DPI2013-43344-R (I+D+i) • Gobierno de Castilla-La Mancha: Proyecto PEII-2014-017-PpeerReviewe

Evaluation of Word Representations in Grounding Natural Language Instructions through Computational Human-Robot Interaction

International audienceIn order to interact with people in a natural way, a robot must be able to link words to objects and actions. Although previous studies in the literature have investigated grounding, they did not consider grounding of unknown synonyms. In this paper, we introduce a probabilistic model for grounding unknown synonymous object and action names using cross-situational learning. The proposed Bayesian learning model uses four different word representations to determine synonymous words. Afterwards, they are grounded through geometric characteristics of objects and kinematic features of the robot joints during action execution. The proposed model is evaluated through an interaction experiment between a human tutor and HSR robot. The results show that semantic and syntactic information enable grounding of unknown synonyms and that the combination of both achieves the best grounding

Restitution of Sculptural Groups Using 3D Scanners

Imagine for a moment that you have to solve a 3D jigsaw of which you have lost several pieces. You have also lost the original box-top showing the final picture, and as if that were not enough, some of the pieces you do have may belong to some other jigsaw. This is in essence the sort of challenge that we faced in the novel project that we shall be describing in this paper. The final aim of the project was, with the help of 3D scanners, to digitalize and reconstruct multi-piece classical sculptures. Particularly, we tackle the restitution of the so-called “Aeneas Group”, a famous iconographic reference during the Roman Empire. We have undertaken this ambitious project in collaboration with the research department of the Spanish National Museum of Roman Art (MNAR). This paper summarizes the real problems that arose and had to be solved, the innovations, and the main results of the work that we have carried out over these recent years

A geometrical-based approach to recognise structure of complex interiors

3D modelling of building interiors has gained a lot of interest recently, specifically since the rise of Building Information Modeling (BIM). A number of methods have been developed in the past, however most of them are limited to modelling non-complex interiors. 3D laser scanners are the preferred sensor to collect the 3D data, however the cost of state-of-the-art laser scanners are prohibitive to many. Other types of sensors could also be used to generate the 3D data but they have limitations especially when dealing with clutter and occlusions. This research has developed a platform to produce 3D modelling of building interiors while adapting a low-cost, low-level laser scanner to generate the 3D interior data. The PreSuRe algorithm developed here, which introduces a new pipeline in modelling building interiors, combines both novel methods and adapts existing approaches to produce the 3D modelling of various interiors, from sparse room to complex interiors with non-ideal geometrical structure, highly cluttered and occluded. This approach has successfully reconstructed the structure of interiors, with above 96% accuracy, even with high amount of noise data and clutter. The time taken to produce the resulting model is almost real-time, compared to existing techniques which may take hours to generate the reconstruction. The produced model is also equipped with semantic information which differentiates the model from a regular 3D CAD drawing and can be use to assist professionals and experts in related fields

LOAM: Lidar Odometry and Mapping in Real-time

Abstract — We propose a real-time method for odometry and mapping using range measurements from a 2-axis lidar moving in 6-DOF. The problem is hard because the range measurements are received at different times, and errors in motion estimation can cause mis-registration of the resulting point cloud. To date, coherent 3D maps can be built by off-line batch methods, often using loop closure to correct for drift over time. Our method achieves both low-drift and low-computational complexity with-out the need for high accuracy ranging or inertial measurements. The key idea in obtaining this level of performance is the division of the complex problem of simultaneous localization and mapping, which seeks to optimize a large number of variables simultaneously, by two algorithms. One algorithm performs odometry at a high frequency but low fidelity to estimate velocity of the lidar. Another algorithm runs at a frequency of an order of magnitude lower for fine matching and registration of the point cloud. Combination of the two algorithms allows the method to map in real-time. The method has been evaluated by a large set of experiments as well as on the KITTI odometry benchmark. The results indicate that the method can achieve accuracy at the level of state of the art offline batch methods. I

Pipe and Ductwork Progress Tracking using 3D Sensing Technologies

Automated construction progress tracking is becoming critical to efficient and effective construction management. More and more construction companies are putting aside the old way of tracking progress, which was mainly based on foremen daily reports and visual inspections, and are adopting 3D sensing technologies as a new and modern way of tracking progress. Technologies such as 3D laser scanners (LADARs) are investigated as a means to acquire comprehensive 3D point-cloud data which can then be studied by management to determine the progress of construction. Although being much more accurate and efficient than visual inspections, this new progress tracking approach can be improved by applying object recognition algorithms that enable an automated progress tracking. This new approach has been investigated by other researchers, but only for progress tracking of structural elements. This study focuses on mechanical objects such as pipes and ducts, which would give the progress tracking a better level of detail and a wider scope. The investigation is carried out on a field database acquired during the construction of the Engineering VI Building at the University of Waterloo. It was found that the laser scanning technology is a suitable method for acquiring point-clouds of pipes and ductwork, and also that the object recognition algorithm used in this study allows a progress tracking as well as a quality tracking of the HVAC system installation

Visual SLAM with RGB-D cameras based on pose graph optimization

En este trabajo abordamos el problema de localización y mapeo simultáneo (SLAM) utilizando únicamente información obtenida mediante una cámara RGB-D. El objetivo principal es desarrollar un sistema SLAM capaz de estimar la trayectoria completa del sensor y generar una representación 3D consistente del entorno en tiempo real. Para lograr este objetivo, el sistema se basa en un método de estimación del movimiento del sensor a partir de información de profundidad densa y en técnicas de reconocimiento de lugares a partir de características visuales. A partir de estos algoritmos, se extraen restricciones espaciales entre fotogramas cuidadosamente seleccionados. Con estas restricciones espaciales se construye un grafo de poses, empleado para inferir la trayectoria más verosímil. El sistema se ha diseñado para ejecutarse en dos hilos paralelos: uno para el seguimiento y el otro para la construcción de la representación consistente. El sistema se evalúa en conjuntos de datos públicamente accesible, alcanzando una precisión comparable a sistemas de SLAM del estado del arte. Además, el hilo de seguimiento se ejecuta a una frecuencia de 60 Hz en un ordenador portátil de prestaciones modestas. También se realizan pruebas en situaciones más realistas, procesando observaciones adquiridas mientras se movía el sensor por dos entornos de interiores distintos

Orientation and integration of images and image blocks with laser scanning data

Laser scanning and photogrammetry are methods for effective and accurate measurement and classification of urban and forest areas. Because these methods complement each other, then integration or integrated use brings additional benefits to real-life applications. However, finding tie features between data sets is a challenging task since laser scanning and imagery are far from each other in nature. The aim of this thesis was to create methods for solving relative orientations between laser scanning data and imagery that would assist in near-future applications integrating laser scanning and photogrammetry. Moreover, a further goal was to create methods enabling the use of data acquired from very different perspectives, such as terrestrial and airborne data. To meet these aims, an interactive orientation method enabling the use of single images, stereo images or larger image blocks was developed and tested. The multi-view approach usually has a significant advantage over the use of a single image. After accurate orientation of laser scanning data and imagery, versatile applications become available. Such applications include, e.g., automatic object recognition, accurate classification of individual trees, point cloud densification, automatic classification of land use, system calibration, and generation of photorealistic 3D models. Besides the orientation part, another aim of the research was to investigate how to fuse or use these two data types together in applications. As a result, examples that evaluated the behavior of laser point clouds in both urban and forestry areas, detection and visualization of temporal changes, enhanced data understanding, stereo visualization, multi-source and multi-angle data fusion, point cloud colorizing, and detailed examination of full waveform laser scanning data were given

Automatic 3D facial modelling with deformable models.

Facial modelling and animation has been an active research subject in computer graphics since the 1970s. Due to extremely complex biomechanical structures of human faces and peoples visual familiarity with human faces, modelling and animating realistic human faces is still one of greatest challenges in computer graphics. Since we are so familiar with human faces and very sensitive to unnatural subtle changes in human faces, it usually requires a tremendous amount of artistry and manual work to create a convincing facial model and animation. There is a clear need of developing automatic techniques for facial modelling in order to reduce manual labouring. In order to obtain a realistic facial model of an individual, it is now common to make use of 3D scanners to capture range scans from the individual and then fit a template to the range scans. However, most existing template-fitting methods require manually selected landmarks to warp the template to the range scans. It would be tedious to select landmarks by hand over a large set of range scans. Another way to reduce repeated work is synthesis by reusing existing data. One example is expression cloning, which copies facial expression from one face to another instead of creating them from scratch. This aim of this study is to develop a fully automatic framework for template-based facial modelling, facial expression transferring and facial expression tracking from range scans. In this thesis, the author developed an extension of the iterative closest points (ICP) algorithm, which is able to match a template with range scans in different scales, and a deformable model, which can be used to recover the shapes of range scans and to establish correspondences between facial models. With the registration method and the deformable model, the author proposed a fully automatic approach to reconstructing facial models and textures from range scans without re-quiring any manual interventions. In order to reuse existing data for facial modelling, the author formulated and solved the problem of facial expression transferring in the framework of discrete differential geometry. The author also applied his methods to face tracking for 4D range scans. The results demonstrated the robustness of the registration method and the capabilities of the deformable model. A number of possible directions for future work were pointed out