Quality Enhancement of 3D Models Reconstructed By RGB-D Camera Systems

Low-cost RGB-D cameras like Microsoft\u27s Kinect capture RGB data for each vertex while reconstructing 3D models from objects with obvious drawbacks of poor mesh and texture qualities due to their hardware limitations. In this thesis we propose a combined method that enhances geometrically and chromatically 3D models reconstructed by RGB-D camera systems. Our approach utilizes Butterfly Subdivision and Surface Fitting techniques to generate smoother triangle surface meshes, where sharp features can be well preserved or minimized by different Surface Fitting algorithms. Additionally the global contrast of mesh textures is enhanced by using a modified Histogram Equalization algorithm, in which the new intensity of each vertex is obtained by applying cumulative distribution function and calculating the accumulated normalized histogram of the texture. A number of experimental results and comparisons demonstrate that our method efficiently and effectively improves the geometric and chromatic quality of 3D models reconstructed from RGB-D cameras

High-quality dense stereo vision for whole body imaging and obesity assessment

textThe prevalence of obesity has necessitated developing safe and convenient tools for timely assessing and monitoring this condition for a broad range of population. Three-dimensional (3D) body imaging has become a new mean for obesity assessment. Moreover, it generates body shape information that is meaningful for fitness, ergonomics, and personalized clothing. In the previous work of our lab, we developed a prototype active stereo vision system that demonstrated a potential to fulfill this goal. But the prototype required four computer projectors to cast artificial textures on the body which facilitate the stereo-matching on texture-deficient images (e.g., skin). This decreases the mobility of the system when used to collect a large population data. In addition, the resolution of the generated 3D~images is limited by both cameras and projectors available during the project. The study reported in this dissertation highlights our continued effort in improving the capability of 3Dbody imaging through simplified hardware for passive stereo and advanced computation techniques. The system utilizes high-resolution single-lens reflex (SLR) cameras, which became widely available lately, and is configured in a two-stance design to image the front and back surfaces of a person. A total of eight cameras are used to form four pairs of stereo units. Each unit covers a quarter of the body surface. The stereo units are individually calibrated with a specific pattern to determine cameras' intrinsic and extrinsic parameters for stereo matching. The global orientation and position of each stereo unit within a common world coordinate system is calculated through a 3Dregistration step. The stereo calibration and 3Dregistration procedures do not need to be repeated for a deployed system if the cameras' relative positions have not changed. This property contributes to the portability of the system, and tremendously alleviates the maintenance task. The image acquisition time is around two seconds for a whole-body capture. The system works in an indoor environment with a moderate ambient light. Advanced stereo computation algorithms are developed by taking advantage of high-resolution images and by tackling the ambiguity problem in stereo matching. A multi-scale, coarse-to-fine matching framework is proposed to match large-scale textures at a low resolution and refine the matched results over higher resolutions. This matching strategy reduces the complexity of the computation and avoids ambiguous matching at the native resolution. The pixel-to-pixel stereo matching algorithm follows a classic, four-step strategy which consists of matching cost computation, cost aggregation, disparity computation and disparity refinement. The system performance has been evaluated on mannequins and human subjects in comparison with other measurement methods. It was found that the geometrical measurements from reconstructed 3Dbody models, including body circumferences and whole volume, are highly repeatable and consistent with manual and other instrumental measurements (CV 0.99). The agreement of percent body fat (%BF) estimation on human subjects between stereo and dual-energy X-ray absorptiometry (DEXA) was found to be improved over the previous active stereo system, and the limits of agreement with 95% confidence were reduced by half. Our achieved %BF estimation agreement is among the lowest ones of other comparative studies with commercialized air displacement plethysmography (ADP) and DEXA. In practice, %BF estimation through a two-component model is sensitive to body volume measurement, and the estimation of lung volume could be a source of variation. Protocols for this type of measurement should still be created with an awareness of this factor.Biomedical Engineerin

Vermessung, Modellierung und Verifizierung von Licht-Masse-Interaktions-Phänomenen

The photo-realistic rendering of scenes showing natural phenomena requires skilled graphic designers not only to produce a convincingly good-looking image but also to convey physical plausibility. This is especially important in industrial context, where a modelled scene showcasing a product has to approximate the actual environment of a product as closely as possible, e.g. in automotive industries. In this thesis, new techniques to measure natural phenomena are presented in order to provide new or verify existing models for rendering the physically plausible image. In contrast to other approaches, the measurement is performed using nonconventional methods: an ellipsometer is employed to capture the specular reflectance with respect to the polarisation behaviour, a transmissive screen attached to a glass tank is imaged to capture underwater reflectances, and the Microsoft Kinect, a motion capturing device, is used to detect the gas flows around objects. The results are the verification of existing, physically plausible models for commodity metals, an enhanced reflectance model for materials immersed in transparent media with known refractive index, and the reconstruction of two-phase gas flows around occluding objects.Das Erzeugen von Szenen mit natürlichen Phänomenen in fotorealistischer Qualität ist aufwändig, weil nicht nur ein realistisches Bild erstellt werden soll, sondern auch physikalische Plausibilität in Bezug auf das modellierte Phänomen verlangt wird. Besonders in der Industrie, z.B. in der Automobilindustrie, sollte die modellierte Szene, in der ein Produkt eingesetzt wird, der tatsächlichen Einsatzumgebung so naturgetreu wie möglich ähneln. In dieser Dissertation werden neue Ansätze zum Messen von natürlichen Phänomenen präsentiert, die es ermöglichen, für bestimmte Phänomene neue Modelle zu erstellen oder bestehende Modelle erschöpfender zu verifizieren, um damit physikalische Plausibilität für Szenen, die am Computer ereugt werden, zu gewährleisten. Im Unterschied zu anderen Verfahren, werden unkonventionelle Methoden zur Messung umgesetzt: Mit Hilfe eines Ellipsometers wird die Oberflächenreflektanz von Metallen so vermessen, dass auch Änderungen im Polarisationszustand des Lichtes erfasst werden. Unterwasserreflektanzen von Materialien werden mit Hilfe eines lichtdurchlässigen Diffusers abgebildet, der an einen Glasbecher angebracht wird, und der Bewegungssensor Kinect von Microsoft wird verwendet, um Gasströmungen um Objekte zu detektieren. Die Ergebnisse sind die Verifikation von bestehenden Modellen für handelsübliche Metallflächen, ein erweitertes Reflektanzmodell für Oberflächen, die in refraktive Medien eingetaucht werden und die Rekonstruktion von Gasströmungen um Objekte

Handbook of best practice and standards for 2D+ and 3D imaging of natural history collections

peer reviewedDigitising a collection is key to make it last even after the physical objects are no longer available. Almost all of the techniques currently available to digitise a natural history collection in 2D+ and 3D are listed herein. The techniques are explained in a way that even one without any knowledge on the subject may understand their principle. The strong and weak points of the techniques are discussed, and an overview of suitable collections and specimens are given for each one of them. Also, plenty of examples already digitised with each technique are provided together with the links to visualise them in 3D. After explaining all the different digitisation options, the subsequent chapters provide information on how to improve the 2D+ and 3D digital twins of the specimens and techniques are compared to each other by means of test specimens. These give a fast overview of the capabilities of the digitisation techniques. Possible solutions to avoid digitisation errors are equally provided. Lastly, the dissemination of the results and the data management of the 3D models are briefly discussed in the final chapters. Also, a large chapter is provided with several workflows that can be followed to get the best possible results

From scans to models: Registration of 3D human shapes exploiting texture information

New scanning technologies are increasing the importance of 3D mesh data, and of algorithms that can reliably register meshes obtained from multiple scans. Surface registration is important e.g. for building full 3D models from partial scans, identifying and tracking objects in a 3D scene, creating statistical shape models. Human body registration is particularly important for many applications, ranging from biomedicine and robotics to the production of movies and video games; but obtaining accurate and reliable registrations is challenging, given the articulated, non-rigidly deformable structure of the human body. In this thesis, we tackle the problem of 3D human body registration. We start by analyzing the current state of the art, and find that: a) most registration techniques rely only on geometric information, which is ambiguous on flat surface areas; b) there is a lack of adequate datasets and benchmarks in the field. We address both issues. Our contribution is threefold. First, we present a model-based registration technique for human meshes that combines geometry and surface texture information to provide highly accurate mesh-to-mesh correspondences. Our approach estimates scene lighting and surface albedo, and uses the albedo to construct a high-resolution textured 3D body model that is brought into registration with multi-camera image data using a robust matching term. Second, by leveraging our technique, we present FAUST (Fine Alignment Using Scan Texture), a novel dataset collecting 300 high-resolution scans of 10 people in a wide range of poses. FAUST is the first dataset providing both real scans and automatically computed, reliable ground-truth correspondences between them. Third, we explore possible uses of our approach in dermatology. By combining our registration technique with a melanocytic lesion segmentation algorithm, we propose a system that automatically detects new or evolving lesions over almost the entire body surface, thus helping dermatologists identify potential melanomas. We conclude this thesis investigating the benefits of using texture information to establish frame-to-frame correspondences in dynamic monocular sequences captured with consumer depth cameras. We outline a novel approach to reconstruct realistic body shape and appearance models from dynamic human performances, and show preliminary results on challenging sequences captured with a Kinect

Monen näkymän stereon rekonstruktio tietokonegrafiikan sisällön taltiointiin

Rendering of photorealistic models is becoming increasingly feasible and important in computer graphics. Due to the high amount of work in creating such models by hand, required by the need of high level of detail in geometry, colors, and animation, it is desirable to automate the model creation. This task is realised by recovering content from photographs that describe the modeled subject from multiple viewpoints. In this thesis, elementary theory on image acquisition and photograph-based reconstruction of geometry and colors is studied and recent research and software for graphics content reconstruction are reviewed. Based on the studied background, a rig is built as a grid of nine off-the-shelf digital cameras, a custom remote shutter trigger, and supporting software. The purpose of the rig is to capture high-quality photographs and video in a reliable and practical manner, to be processed in multi-view reconstruction programs. The camera rig was configured to shoot small static subjects for experiments. The resulting photos and video were then processed directly for the subject's geometry and color, with little or no image enhancements done. Two typical reconstruction software pipelines were described and tested. The rig was found to perform well for several subjects with little subject-specific tuning; issues in the setup were analyzed and further improvements suggested based on the captured test models. Image quality of the cameras was found to be excellent for the task, and most problems arose from uneven lighting and practical issues. The developed rig was found to produce sub-millimeter scale accuracy in geometry and texture of subjects such as human faces. Future work was suggested for lighting, video synchronization and study of state-of-the-art image processing and reconstruction algorithms.Fotorealististen mallien renderöinti on yhä tärkeämpää ja mahdollisempaa tietokonegrafiikassa. Näiden mallien luominen käsityönä on työlästä vaaditun korkean tarkkuuden takia geometrian, värien ja animaation osalta, ja onkin haluttavaa korvata käsityö automatiikalla. Automatisointi voidaan suorittaa taltioimalla sisältö valokuvista, jotka kuvastavat samaa kohdetta useammasta näkymästä. Tässä diplomityössä tarkastellaan kuvantamista ja valokuvapohjaisen geometrian rekonstruktiota geometrian ja värien kannalta ja katsastetaan viimeaikainen tutkimus ja ohjelmistot grafiikkasisällön rekonstruointiin. Taustan pohjalta rakennetaan laitteisto, joka koostuu yhdeksästä valmiina saatavilla olevasta digikamerasta aseteltuna hilaksi, itse tehdystä etälaukaisimesta sekä ohjelmistoista. Laitteiston tarkoituksena on taltioida luotettavalla ja käytännöllisellä tavalla korkealaatuisia valokuvia ja videokuvaa, joita voi käsitellä monen näkymän stereon rekonstruktion tietokonesovelluksissa. Laitteisto säädettiin kuvaamaan pieniä staattisia kohteita kokeita varten. Tuloksena saadusta kuvamateriaalista laskettiin kohteen geometria ja värit ilman mainittavaa kuvanparannuksen käyttöä. Käytiin läpi kaksi tyypillistä rekonstruktio-ohjelmistoa testiksi ja laitteiston havaittiin soveltuvan hyvin useisiin kohteisiin ilman erityistä säätämistä. Kameroiden kuvanlaatu todettiin tehtävään erinomaiseksi ja useimmat haasteet johtuivat epätasaisesta valaistuksesta ja käytännön pulmista. Laitteiston todettiin tuottavan alle millimetriskaalan geometriaa ja pintavärikuvaa ihmiskasvojen kaltaisista kohteista. Jatkotyötä ehdotettiin valaistukseen, videon synkronointiin ja viimeisimpien kuvankäsittely- ja rekonstruktioalgoritmien tutkimiseen

Physics-based Reconstruction and Animation of Humans

Creating digital representations of humans is of utmost importance for applications ranging from entertainment (video games, movies) to human-computer interaction and even psychiatrical treatments. What makes building credible digital doubles difficult is the fact that the human vision system is very sensitive to perceiving the complex expressivity and potential anomalies in body structures and motion. This thesis will present several projects that tackle these problems from two different perspectives: lightweight acquisition and physics-based simulation. It starts by describing a complete pipeline that allows users to reconstruct fully rigged 3D facial avatars using video data coming from a handheld device (e.g., smartphone). The avatars use a novel two-scale representation composed of blendshapes and dynamic detail maps. They are constructed through an optimization that integrates feature tracking, optical flow, and shape from shading. Continuing along the lines of accessible acquisition systems, we discuss a framework for simultaneous tracking and modeling of articulated human bodies from RGB-D data. We show how semantic information can be extracted from the scanned body shapes. In the second half of the thesis, we will deviate from using standard linear reconstruction and animation models, and rather focus on exploiting physics-based techniques that are able to incorporate complex phenomena such as dynamics, collision response and incompressibility of the materials. The first approach we propose assumes that each 3D scan of an actor records his body in a physical steady state and uses a process called inverse physics to extract a volumetric physics-ready anatomical model of him. By using biologically-inspired growth models for the bones, muscles and fat, our method can obtain realistic anatomical reconstructions that can be later on animated using external tracking data such as the one resulting from tracking motion capture markers. This is then extended to a novel physics-based approach for facial reconstruction and animation. We propose a facial animation model which simulates biomechanical muscle contractions in a volumetric head model in order to create the facial expressions seen in the input scans. We then show how this approach allows for new avenues of dynamic artistic control, simulation of corrective facial surgery, and interaction with external forces and objects

Creating a Virtual Mirror for Motor Learning in Virtual Reality

Waltemate T. Creating a Virtual Mirror for Motor Learning in Virtual Reality. Bielefeld: Universität Bielefeld; 2018

Detecção de ataques de apresentação por faces em dispositivos móveis

Orientadores: Anderson de Rezende Rocha, Fernanda Alcântara AndalóDissertação (mestrado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Com o crescimento e popularização de tecnologias de autenticação biométrica, tais como aquelas baseadas em reconhecimento facial, aumenta-se também a motivação para se explorar ataques em nível de sensor de captura ameaçando a eficácia dessas aplicações em cenários reais. Um desses ataques se dá quando um impostor, desejando destravar um celular alheio, busca enganar o sistema de reconhecimento facial desse dispositivo apresentando a ele uma foto do usuário alvo. Neste trabalho, estuda-se o problema de detecção automática de ataques de apresentação ao reconhecimento facial em dispositivos móveis, considerando o caso de uso de destravamento rápido e as limitações desses dispositivos. Não se assume o uso de sensores adicionais, ou intervenção consciente do usuário, dependendo apenas da imagem capturada pela câmera frontal em todos os processos de decisão. Contribuições foram feitas em relação a diferentes aspectos do problema. Primeiro, foi coletada uma base de dados de ataques de apresentação chamada RECOD-MPAD, que foi especificamente projetada para o cenário alvo, possuindo variações realistas de iluminação, incluindo sessões ao ar livre e de baixa luminosidade, ao contrário das bases públicas disponíveis atualmente. Em seguida, para enriquecer o entendimento do que se pode esperar de métodos baseados puramente em software, adota-se uma abordagem em que as características determinantes para o problema são aprendidas diretamente dos dados a partir de redes convolucionais, diferenciando-se de abordagens tradicionais baseadas em conhecimentos específicos de aspectos do problema. São propostas três diferentes formas de treinamento da rede convolucional profunda desenvolvida para detectar ataques de apresentação: treinamento com faces inteiras e alinhadas, treinamento com patches (regiões de interesse) de resolução variável, e treinamento com uma função objetivo projetada especificamente para o problema. Usando uma arquitetura leve como núcleo da nossa rede, certifica-se que a solução desenvolvida pode ser executada diretamente em celulares disponíveis no mercado no ano de 2017. Adicionalmente, é feita uma análise que considera protocolos inter-fatores e disjuntos de usuário, destacando-se alguns dos problemas com bases de dados e abordagens atuais. Experimentos no benchmark OULU-NPU, proposto recentemente e usado em uma competição internacional, sugerem que os métodos propostos se comparam favoravelmente ao estado da arte, e estariam entre os melhores na competição, mesmo com a condição de pouco uso de memória e recursos computacionais limitados. Finalmente, para melhor adaptar a solução a cada usuário, propõe-se uma forma efetiva de usar uma galeria de dados do usuário para adaptar os modelos ao usuário e ao dispositivo usado, aumentando sua eficácia no cenário operacionalAbstract: With the widespread use of biometric authentication systems, such as those based on face recognition, comes the exploitation of simple attacks at the sensor level that can undermine the effectiveness of these technologies in real-world setups. One example of such attack takes place when an impostor, aiming at unlocking someone else's smartphone, deceives the device¿s built-in face recognition system by presenting a printed image of the genuine user's face. In this work, we study the problem of automatically detecting presentation attacks against face authentication methods in mobile devices, considering the use-case of fast device unlocking and hardware constraints of such devices. We do not assume the existence of any extra sensors or user intervention, relying only on the image captured by the device¿s frontal camera. Our contributions lie on multiple aspects of the problem. Firstly, we collect RECOD-MPAD, a new presentation-attack dataset that is tailored to the mobile-device setup, and is built to have real-world variations in lighting, including outdoors and low-light sessions, in contrast to existing public datasets. Secondly, to enrich the understanding of how far we can go with purely software-based methods when tackling this problem, we adopt a solely data-driven approach ¿ differently from handcrafted methods in prior art that focus on specific aspects of the problem ¿ and propose three different ways of training a deep convolutional neural network to detect presentation attacks: training with aligned faces, training with multi-resolution patches, and training with a multi-objective loss function crafted specifically to the problem. By using a lightweight architecture as the core of our network, we ensure that our solution can be efficiently embedded in modern smartphones in the market at the year of 2017. Additionally, we provide a careful analysis that considers several user-disjoint and cross-factor protocols, highlighting some of the problems with current datasets and approaches. Experiments with the OULU-NPU benchmark, which was used recently in an international competition, suggest that our methods are among the top performing ones. Finally, to further enhance the model's efficacy and discriminability in the target setup of user authentication for mobile devices, we propose a method that leverages the available gallery of user data in the device and adapts the method decision-making process to the user's and device¿s own characteristicsMestradoCiência da ComputaçãoMestre em Ciência da Computaçã

Generation of Virtual Humans for Virtual Reality, Medicine, and Domestic Assistance

Achenbach J. Generation of Virtual Humans for Virtual Reality, Medicine, and Domestic Assistance. Bielefeld: Universität Bielefeld; 2019.Virtual humans are employed in various applications including computer games, special effects in movies, virtual try-ons, medical surgery planning, and virtual assistance. This thesis deals with virtual humans and their computer-aided generation for different purposes. In a first step, we derive a technique to digitally clone the face of a scanned person. Fitting a facial template model to 3D-scanner data is a powerful technique for generating face avatars, in particular in the presence of noisy and incomplete measurements. Consequently, there are many approaches for the underlying non-rigid registration task, and these are typically composed from very similar algorithmic building blocks. By providing a thorough analysis of the different design choices, we derive a face matching technique tailored to high-quality reconstructions from high-resolution scanner data. We then extend this approach in two ways: An anisotropic bending model allows us to more accurately reconstruct facial details. A simultaneous constrained fitting of eyes and eyelids improves the reconstruction of the eye region considerably. Next, we extend this work to full bodies and present a complete pipeline to create animatable virtual humans by fitting a holistic template character. Due to the careful selection of techniques and technology, our reconstructed humans are quite realistic in terms of both geometry and texture. Since we represent our models as single-layer triangle meshes and animate them through standard skeleton-based skinning and facial blendshapes, our characters can be used in standard VR engines out of the box. By optimizing computation time and minimizing manual intervention, our reconstruction pipeline is capable of processing entire characters in less than ten minutes. In a following part of this thesis, we build on our template fitting method and deal with the problem of inferring the skin surface of a head from a given skull and vice versa. Starting with a method for automated estimation of a human face from a given skull remain, we extend this approach to bidirectional facial reconstruction in order to also estimate the skull from a given scan of the skin surface. This is based on a multilinear model that describes the correlation between the skull and the facial soft tissue thickness on the one hand and the head/face surface geometry on the other hand. We demonstrate the versatility of our novel multilinear model by estimating faces from given skulls as well as skulls from given faces within just a couple of seconds. To foster further research in this direction, we made our multilinear model publicly available. In a last part, we generate assistive virtual humans that are employed as stimuli for an interdisciplinary study. In the study, we shed light on user preferences for visual attributes of virtual assistants in a variety of smart home contexts