DATA-DRIVEN FACIAL IMAGE SYNTHESIS FROM POOR QUALITY LOW RESOLUTION IMAGE

Ph.DDOCTOR OF PHILOSOPH

Facial Texture Super-Resolution by Fitting 3D Face Models

This book proposes to solve the low-resolution (LR) facial analysis problem with 3D face super-resolution (FSR). A complete processing chain is presented towards effective 3D FSR in real world. To deal with the extreme challenges of incorporating 3D modeling under the ill-posed LR condition, a novel workflow coupling automatic localization of 2D facial feature points and 3D shape reconstruction is developed, leading to a robust pipeline for pose-invariant hallucination of the 3D facial texture

Rekonstruktion, Analyse und Editierung dynamisch deformierter 3D-Oberflächen

Dynamically deforming 3D surfaces play a major role in computer graphics. However, producing time-varying dynamic geometry at ever increasing detail is a time-consuming and costly process, and so a recent trend is to capture geometry data directly from the real world. In the first part of this thesis, I propose novel approaches for this research area. These approaches capture dense dynamic 3D surfaces from multi-camera systems in a particularly robust and accurate way. This provides highly realistic dynamic surface models for phenomena like moving garments and bulging muscles. However, re-using, editing, or otherwise analyzing dynamic 3D surface data is not yet conveniently possible. To close this gap, the second part of this dissertation develops novel data-driven modeling and animation approaches. I first show a supervised data-driven approach for modeling human muscle deformations that scales to huge datasets and provides fine-scale, anatomically realistic deformations at high quality not attainable by previous methods. I then extend data-driven modeling to the unsupervised case, providing editing tools for a wider set of input data ranging from facial performance capture and full-body motion to muscle and cloth deformation. To this end, I introduce the concepts of sparsity and locality within a mathematical optimization framework. I also explore these concepts for constructing shape-aware functions that are useful for static geometry processing, registration, and localized editing.Dynamisch deformierbare 3D-Oberflächen spielen in der Computergrafik eine zentrale Rolle. Die Erstellung der für Computergrafik-Anwendungen benötigten, hochaufgelösten und zeitlich veränderlichen Oberflächengeometrien ist allerdings äußerst arbeitsintensiv. Aus dieser Problematik heraus hat sich der Trend entwickelt, Oberflächendaten direkt aus Aufnahmen der echten Welt zu erfassen. Dazu nötige 3D-Rekonstruktionsverfahren werden im ersten Teil der Arbeit entwickelt. Die vorgestellten, neuartigen Verfahren erlauben die Erfassung dynamischer 3D-Oberflächen aus Mehrkamera-Aufnahmen bei hoher Verlässlichkeit und Präzision. Auf diese Weise können detaillierte Oberflächenmodelle von Phänomenen wie in Bewegung befindliche Kleidung oder sich anspannende Muskeln erfasst werden. Aber auch die Wiederverwendung, Bearbeitung und Analyse derlei gewonnener 3D-Oberflächendaten ist aktuell noch nicht auf eine einfache Art und Weise möglich. Um diese Lücke zu schließen beschäftigt sich der zweite Teil der Arbeit mit der datengetriebenen Modellierung und Animation. Zunächst wird ein Ansatz für das überwachte Lernen menschlicher Muskel-Deformationen vorgestellt. Dieses neuartige Verfahren ermöglicht eine datengetriebene Modellierung mit besonders umfangreichen Datensätzen und liefert anatomisch-realistische Deformationseffekte. Es übertrifft damit die Genauigkeit früherer Methoden. Im nächsten Teil beschäftigt sich die Dissertation mit dem unüberwachten Lernen aus 3D-Oberflächendaten. Es werden neuartige Werkzeuge vorgestellt, die eine weitreichende Menge an Eingabedaten verarbeiten können, von aufgenommenen Gesichtsanimationen über Ganzkörperbewegungen bis hin zu Muskel- und Kleidungsdeformationen. Um diese Anwendungsbreite zu erreichen stützt sich die Arbeit auf die allgemeinen Konzepte der Spärlichkeit und Lokalität und bettet diese in einen mathematischen Optimierungsansatz ein. Abschließend zeigt die vorliegende Arbeit, wie diese Konzepte auch für die Konstruktion von oberflächen-adaptiven Basisfunktionen übertragen werden können. Dadurch können Anwendungen für die Verarbeitung, Registrierung und Bearbeitung statischer Oberflächenmodelle erschlossen werden

Super-resolution of 3-dimensional scenes

Super-resolution is an image enhancement method that increases the resolution of images and video. Previously this technique could only be applied to 2D scenes. The super-resolution algorithm developed in this thesis creates high-resolution views of 3-dimensional scenes, using low-resolution images captured from varying, unknown positions

Computational Imaging Approach to Recovery of Target Coordinates Using Orbital Sensor Data

This dissertation addresses the components necessary for simulation of an image-based recovery of the position of a target using orbital image sensors. Each component is considered in detail, focusing on the effect that design choices and system parameters have on the accuracy of the position estimate. Changes in sensor resolution, varying amounts of blur, differences in image noise level, selection of algorithms used for each component, and lag introduced by excessive processing time all contribute to the accuracy of the result regarding recovery of target coordinates using orbital sensor data. Using physical targets and sensors in this scenario would be cost-prohibitive in the exploratory setting posed, therefore a simulated target path is generated using Bezier curves which approximate representative paths followed by the targets of interest. Orbital trajectories for the sensors are designed on an elliptical model representative of the motion of physical orbital sensors. Images from each sensor are simulated based on the position and orientation of the sensor, the position of the target, and the imaging parameters selected for the experiment (resolution, noise level, blur level, etc.). Post-processing of the simulated imagery seeks to reduce noise and blur and increase resolution. The only information available for calculating the target position by a fully implemented system are the sensor position and orientation vectors and the images from each sensor. From these data we develop a reliable method of recovering the target position and analyze the impact on near-realtime processing. We also discuss the influence of adjustments to system components on overall capabilities and address the potential system size, weight, and power requirements from realistic implementation approaches

Pixel-level Image Fusion Algorithms for Multi-camera Imaging System

This thesis work is motivated by the potential and promise of image fusion technologies in the multi sensor image fusion system and applications. With specific focus on pixel level image fusion, the process after the image registration is processed, we develop graphic user interface for multi-sensor image fusion software using Microsoft visual studio and Microsoft Foundation Class library. In this thesis, we proposed and presented some image fusion algorithms with low computational cost, based upon spatial mixture analysis. The segment weighted average image fusion combines several low spatial resolution data source from different sensors to create high resolution and large size of fused image. This research includes developing a segment-based step, based upon stepwise divide and combine process. In the second stage of the process, the linear interpolation optimization is used to sharpen the image resolution. Implementation of these image fusion algorithms are completed based on the graphic user interface we developed. Multiple sensor image fusion is easily accommodated by the algorithm, and the results are demonstrated at multiple scales. By using quantitative estimation such as mutual information, we obtain the experiment quantifiable results. We also use the image morphing technique to generate fused image sequence, to simulate the results of image fusion. While deploying our pixel level image fusion algorithm approaches, we observe several challenges from the popular image fusion methods. While high computational cost and complex processing steps of image fusion algorithms provide accurate fused results, they also makes it hard to become deployed in system and applications that require real-time feedback, high flexibility and low computation abilit

On some interactive mesh deformations

Techniques devoted to deform 3D models are an important research field in Computer Graphics. They can be used in differentstages: the modelling phase, the animation process and also during some special simulations. Additionally, some applications may require the manipulation of 3D models under certain restrictions to preserve the volume of the modified object. Hence, thepresent PhD Dissertation explores new algorithms to perform flexible, robust and efficient 3D deformations. Apart from this, it also researches on a new methodology to restrict these deformations so that the volume of the manipulated model remains constant. Some of the most used methods to achieve smooth deformations are those included in the Cage-Based Deformation paradigm. Cage-based deformations enclose the model to be deformed in a coarse polyhedron, the cage. Then, they usually rely on Generalized Barycentric Coordinates to relate the model with the vertices, and other geometric elements, of this cage, which are the control points or the deformation handles. Finally, every time that one of these handles is dragged, the model is deformed accordingly. Although this paradigm is simple, elegant and performs efficient deformations, some cage-free space deformation techniques have recently appeared. They increase the flexibility of the deformation handles, which do not need to be connected, and define powerful tools that make the deformation process more versatile and intuitive. In this context, the Dissertation introduces new Generalized Barycentric Coordinate systems specially designed to be used in a cage-free environment. Any user who wants to use the presented schemes only needs to locate a set of control points in the vicinity of the model that he or she wants to deform. These handles can be placed wherever he or she considers mode suitable and the only requirement is that the model has to be enclosed in their convex hull. Up to now, there are few techniques to produce volume-preserving space deformations. However, in recent years there has been a growing interest in performing constrained deformations due to their more realistic and physically plausible results. Our contribution to this research line consists in a deformation framework that preserves the volume of the 3D models by means of its gradient and a control surface to restrict the movement of the handles. Moreover, the proposed methodology is not restricted to the cage-based schemes, but it can also be used in a cage-free environment. Finally, our research can be specially useful for spatial deformations of biological and medical models. This kind of models represent real organs and tissues, which are often soft and lack an internal rigid structure. In addition, they are elastic and incompressible. Any application designed to deal with this group of models and to train or assist doctors must be flexible, robust, efficient and user-friendly. The combination of the proposed cage-free systems with the presented volume-preserving deformation framework satisfiesLes deformacions de models 3D s'utilitzen en diverses etapes de la generació de continguts digitals: durant la fase de modelatge, durant el procés d'animació i en alguns tipus de simulacions. A més a més, hi ha aplicacions que necessiten que la manipulació dels models 3D es faci tenint en compte certes restriccions que permeten la conservació del volum de l'objecte modificat. Tot plegat fa que les tècniques de deformació 3D siguin un camp d'estudi molt important dins del món dels Gràfics. Per aquesta raó, aquesta Tesi Doctoral estudia nous algorismes que permetin realitzar deformacions 3D de manera flexible, robusta i eficient i que, a més a més, permetin conservar el volum dels objectes modificats. Un dels paradigmes més utilitzats per tal de realitzar deformacions suaus és el conegut amb el nom de Deformacions Basades en un Poliedre Englobant. Aquesta família de mètodes embolcalla el model que es vol deformar, normalment representat com una malla de triangles, dins d'un poliedre simple, amb poques cares. Un cop fet això, estableix un sistema de Coordenades Baricèntriques Generalitzades per tal de definir els vèrtexs del model a partir dels vèrtexs del poliedre englobant, els quals s'anomenen punts de control o controls de la deformació. D'aquesta manera, cada cop que s'arrossega o es modifica un d'aquests punts de control, el model que es troba dins del poliedre englobant es deforma segons el sistema de coordenades que s'ha definit. Tot i que aquest paradigma és simple, elegant i eficient, des de fa ja uns anys han començat a aparèixer noves tècniques que no necessiten el poliedre englobant per tal de realitzar la deformació. El seu principal objectiu és augmentar la flexibilitat dels controls de la deformació i definir eines que facin que el procés de deformació sigui més versàtil i intuïtiu. Tenint en compte aquest factor, aquesta Tesi també estudia sistemes de Coordenades Baricèntriques Generalitzades dissenyats per realitzar deformacions sense la necessitat de definir el poliedre englobant. D'aquesta manera, qualsevol usuari que vulgui utilitzar els mètodes que es presenten en aquesta Dissertació només s'ha d'encarregar de definir un conjunt de punts de control al voltant del model que vol deformar, podent-los posar allí on consideri més oportú segons la deformació que vulgui obtenir. L'únic requeriment necessari és que el model ha de quedar dins de l'envolupant convexa d'aquests punts de control. Actualment existeixen pocs mètodes que realitzin deformacions 3D amb preservació del volum. No obstant això, d'un temps ençà ha augmentat l'interès per realitzar deformacions subjectes a certes restriccions que fan que el resultat sigui més realista i físicament versemblant. La contribució d'aquesta Tesi dins d'aquesta línia de recerca consisteix en un sistema de deformació que preserva el volum dels objectes 3D gràcies a còmput del seu gradient i a una superfície de control que restringeix el moviment dels punts de control. Aquest mètode es pot aplicar tant als sistemes de deformació que necessiten un poliedre englobant com als que no el necessiten. Finalment, i ja per acabar, la recerca realitzada pot ser especialment útil per tal de realitzar deformacions de models mèdics i biològics. Aquests tipus de models poden representar òrgans i teixits reals, els quals, normalment, són tous, mancats d'una estructura rígida interna, elàstics i incompressibles. Qualsevol aplicació dissenyada per treballar amb aquest tipus de models i per entrenar i donar assistència a usuaris mèdics hauria de ser flexible, robusta, eficient i fàcil d'utilitzar. La combinació dels mètodes de deformació proposats conjuntament amb el sistema de preservació de volum satisfà totes aquestes condicions. Per aquesta raó es creu que les contribucions realitzades poden esdevenir eines importants per produir deformacions mèdiques.Postprint (published version