Deep learning-based simultaneous multi-phase deformable image registration of sparse 4D-CBCT

Purpose: Respiratory gated 4D-CBCT suffers from sparseness artefacts caused by the limited number of projections available for each respiratory phase/amplitude. These artefacts severely impact deformable image registration methods used to extract motion information. We use deep learning-based methods to predict displacement vector-fields (DVF) from sparse 4D-CBCT images to alleviate the impacts of sparseness artefacts. Methods: We trained U-Net-type convolutional neural network models to predict multiple (10) DVFs in a single forward pass given multiple sparse, gated CBCT and an optional artefact-free reference image as inputs. The predicted DVFs are used to warp the reference image to the different motion states, resulting in an artefact-free image for each state. The supervised training uses data generated by a motion simulation framework. The training dataset consists of 560 simulated 4D-CBCT images of 56 different patients; the generated data include fully sampled ground-truth images that are used to train the network. We compare the results of our method to pairwise image registration (reference image to single sparse image) using a) the deeds algorithm and b) VoxelMorph with image pair inputs. Results: We show that our method clearly outperforms pairwise registration using the deeds algorithm alone. PSNR improved from 25.8 to 46.4, SSIM from 0.9296 to 0.9999. In addition, the runtime of our learning-based method is orders of magnitude shorter (2 seconds instead of 10 minutes). Our results also indicate slightly improved performance compared to pairwise registration (delta-PSNR=1.2). We also trained a model that does not require the artefact-free reference image (which is usually not available) during inference demonstrating only marginally compromised results (delta-PSNR=-0.8). Conclusion: To the best of our knowledge, this is the first time CNNs are used to predict multi-phase DVFs in a single forward pass. This enables novel applications such as 4D-auto-segmentation, motion compensated image reconstruction, motion analyses, and patient motion modeling

Mitigation of motion-induced artifacts in cone beam computed tomography using deep convolutional neural networks

Background: Cone beam computed tomography (CBCT) is often employed on radiation therapy treatment devices (linear accelerators) used in image-guided radiation therapy (IGRT). For each treatment session, it is necessary to obtain the image of the day in order to accurately position the patient, and to enable adaptive treatment capabilities including auto-segmentation and dose calculation. Reconstructed CBCT images often suffer from artifacts, in particular those induced by patient motion. Deep-learning based approaches promise ways to mitigate such artifacts. Purpose: We propose a novel deep-learning based approach with the goal to reduce motion induced artifacts in CBCT images and improve image quality. It is based on supervised learning and includes neural network architectures employed as pre- and/or post-processing steps during CBCT reconstruction. Methods: Our approach is based on deep convolutional neural networks which complement the standard CBCT reconstruction, which is performed either with the analytical Feldkamp-Davis-Kress (FDK) method, or with an iterative algebraic reconstruction technique (SART-TV). The neural networks, which are based on refined U-net architectures, are trained end-to-end in a supervised learning setup. Labeled training data are obtained by means of a motion simulation, which uses the two extreme phases of 4D CT scans, their deformation vector fields, as well as time-dependent amplitude signals as input. The trained networks are validated against ground truth using quantitative metrics, as well as by using real patient CBCT scans for a qualitative evaluation by clinical experts. Results: The presented novel approach is able to generalize to unseen data and yields significant reductions in motion induced artifacts as well as improvements in image quality compared with existing state-of-the-art CBCT reconstruction algorithms (up to +6.3 dB and +0.19 improvements in peak signal-to-noise ratio, PSNR, and structural similarity index measure, SSIM, respectively), as evidenced by validation with an unseen test dataset, and confirmed by a clincal evaluation on real patient scans (up to 74% preference for motion artifact reduction over standard reconstruction). Conclusions: For the first time, it is demonstrated, also by means of clinical evaluation, that inserting deep neural networks as pre- and post-processing plugins in the existing 3D CBCT reconstruction and trained end-to-end yield significant improvements in image quality and reduction of motion artifacts

Statistical modeling of facial aging based on 3D scans

This thesis presents an approach to the modeling of facial aging using and extending the Morphable Model technique. For modeling the face variation across individuals, facial expressions, and physical attributes, we collected 3D face scans of 298 persons. The 3D face scans where acquired with a structured light 3D scanner, which we improved in collaboration with the manufacturer to achieve superior geometry and texture quality. Moreover, we developed an efficient way to measure fine skin structure and reflection properties with the scanner. The collected face scans have been used to build the Basel Face Model, a new publicly available Morphable Model. Using the 3D scans we learn the correlation between physical attributes such as weight, height, and especially age and faces. With the learned correlation, we present a novel way to simultaneously manipulate different attributes and demonstrate the capability to model changes caused by aging. Using the attributes of the face model in conjunction with a skull model developed in the same research group, we present a method to reconstruct faces from skull shapes which considers physical attributes, as the body weight, age etc. The most important aspect of facial aging that can not be simulated with the Morphable Model is the appearance of facial wrinkles. In this work we present a novel approach to synthesize age wrinkles based on statistics. Our wrinkle synthesis consists of two main parts: The learning of a generative model of wrinkle constellations, and the modeling of their visual appearance. For learning the constellations we use kernel density estimation of manually labeled wrinkles to estimate the wrinkle occurrence probability. To learn the visual appearance of wrinkles we use the fine scale skin structure captured with our improved scanning method. Our results show that the combination of the attribute fitting based aging and the wrinkle synthesis, facilitate a simulation of visually convincing progressive aging. The method is without restrictions applicable to any face that can be represented by the Morphable Model

Weight, Sex, and Facial Expressions : on the Manipulation of Attributes in Generative 3D Face Models

Generative 3D Face Models are expressive models with applicationsin modelling and editing. They are learned from example faces, and offer a compactrepresentation of the continuous space of faces. While they have proven tobe useful as strong priors in face reconstruction they remain to be difficult to usein artistic editing tasks. We describe a way to navigate face space by changingmeaningful parameters learned from the training data. This makes it possible tofix attributes such as height, weight, age, expression or ‘lack of sleep’ while lettingthe infinity of unfixed other attributes vary in a statistically meaningful way.We propose an inverse approach based on learning the distribution of faces inattribute space. Given a set of target attributes we then find the face which has thetarget attributes with high probability, and is as similar as possible to the inputface

Facial Normal Map Capture using Four Lights : an Effective and Inexpensive Method of Capturing the Fine Scale Detail of Human Faces using Four Point Lights

Obtaining photorealistic scans of human faces is both challenging and expensive. Capturing thehigh-frequency components of skin surface structure requires the face to be scanned at very highresolutions, outside the range of most structured light 3D scanners.We present a novel and simple enhancement to the acquisition process, requiring only four photo-graphic  ash-lights and three texture cameras attached to the structured light scanner setup.The three texture cameras capture one texture map (luminance map) of the face as illuminatedby each of the four  ash-lights. Based on those four luminance textures, three normal maps ofthe head are approximated, one for each color channel. Those normal maps are then used toreconstruct a 3D model of the head at a much higher mesh resolution, in order to validate thenormals. Finally, the validated normals are used as a normal map at rendering time. Alternatively,the reconstructed high resolution model can also be used for rendering

A 3D Face Model for Pose and Illumination Invariant Face Recognition

Generative 3D face models are a powerful tool in computer vision. They provide pose and illumination invariance by modeling the space of 3D faces and the imaging process. The power of these models comes at the cost of an expensive and tedious construction process, which has led the community to focus on more easily constructed but less powerful models. With this paper we publish a generative 3D shape and texture model, the Basel face model (BFM), and demonstrate its application to several face recognition task. We improve on previous models by offering higher shape and texture accuracy due to a better scanning device and less correspondence artifacts due to an improved registration algorithm. The same 3D face model can be fit to 2D or 3D images acquired under different situations and with different sensors using an analysis by synthesis method. The resulting model parameters separate pose, lighting, imaging and identity parameters, which facilitates invariant face recognition across sensors and data sets by comparing only the identity parameters. We hope that the availability of this registered face model will spur research in generative models. Together with the model we publish a set of detailed recognition and reconstruction results on standard databases to allow complete algorithm comparisons

A 3D Face Model for Pose and Illumination Invariant Face Recognition

Generative 3D face models are a powerful tool in computer vision. They provide pose and illumination invariance by modeling the space of 3D faces and the imaging process. The power of these models comes at the cost of an expensive and tedious construction process, which has led the community to focus on more easily constructed but less powerful models. With this paper we publish a generative 3D shape and texture model, the Basel face model (BFM), and demonstrate its application to several face recognition task. We improve on previous models by offering higher shape and texture accuracy due to a better scanning device and less correspondence artifacts due to an improved registration algorithm. The same 3D face model can be fit to 2D or 3D images acquired under different situations and with different sensors using an analysis by synthesis method. The resulting model parameters separate pose, lighting, imaging and identity parameters, which facilitates invariant face recognition across sensors and data sets by comparing only the identity parameters. We hope that the availability of this registered face model will spur research in generative models. Together with the model we publish a set of detailed recognition and reconstruction results on standard databases to allow complete algorithm comparisons

Face Reconstruction from Skull Shapes and Physical Attributes

Reconstructing a person’s face from its skeletal remains is a task thathas over many decades fascinated artist and scientist alike. In this paper we treatfacial reconstruction as a machine learning problem. We use separate statisticalshape models to represent the skull and face morphology. We learn the relationshipbetween the parameters of the models by fitting them to a set of MR imagesof the head and using ridge regression on the resulting model parameters. Sincethe facial shape is not uniquely defined by the skull shape, we allow to specifytarget attributes, such as age or weight. Our experiments show that the reconstructionresults are generally close to the original face, and that by specifyingthe right attributes the perceptual and measured difference between the originaland the predicted face is reduced