NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads

We focus on reconstructing high-fidelity radiance fields of human heads, capturing their animations over time, and synthesizing re-renderings from novel viewpoints at arbitrary time steps. To this end, we propose a new multi-view capture setup composed of 16 calibrated machine vision cameras that record time-synchronized images at 7.1 MP resolution and 73 frames per second. With our setup, we collect a new dataset of over 4700 high-resolution, high-framerate sequences of more than 220 human heads, from which we introduce a new human head reconstruction benchmark. The recorded sequences cover a wide range of facial dynamics, including head motions, natural expressions, emotions, and spoken language. In order to reconstruct high-fidelity human heads, we propose Dynamic Neural Radiance Fields using Hash Ensembles (NeRSemble). We represent scene dynamics by combining a deformation field and an ensemble of 3D multi-resolution hash encodings. The deformation field allows for precise modeling of simple scene movements, while the ensemble of hash encodings helps to represent complex dynamics. As a result, we obtain radiance field representations of human heads that capture motion over time and facilitate re-rendering of arbitrary novel viewpoints. In a series of experiments, we explore the design choices of our method and demonstrate that our approach outperforms state-of-the-art dynamic radiance field approaches by a significant margin.Comment: Siggraph 2023, Project Page: https://tobias-kirschstein.github.io/nersemble/ , Video: https://youtu.be/a-OAWqBzld

Learning Neural Parametric Head Models

We propose a novel 3D morphable model for complete human heads based on hybrid neural fields. At the core of our model lies a neural parametric representation that disentangles identity and expressions in disjoint latent spaces. To this end, we capture a person's identity in a canonical space as a signed distance field (SDF), and model facial expressions with a neural deformation field. In addition, our representation achieves high-fidelity local detail by introducing an ensemble of local fields centered around facial anchor points. To facilitate generalization, we train our model on a newly-captured dataset of over 3700 head scans from 203 different identities using a custom high-end 3D scanning setup. Our dataset significantly exceeds comparable existing datasets, both with respect to quality and completeness of geometry, averaging around 3.5M mesh faces per scan 1 1 We will publicly release our dataset along with a public benchmark for both neural head avatar construction as well as an evaluation on a hidden test-set for inference-time fitting.. Finally, we demonstrate that our approach outperforms state-of-the-art methods in terms of fitting error and reconstruction quality

End-to-end Learning for Dimensional Emotion Recognition from Physiological Signals

International audienceDimensional emotion recognition from physiological signals is a highly challenging task. Common methods rely on hand-crafted features that do not yet provide the performance necessary for real-life application. In this work, we exploit a series of convolutional and recurrent neural networks to predict affect from physiological signals, such as electrocardiogram and electrodermal activity, directly from the raw time representation. The motivation behind this so-called end-to-end approach is that, ultimately, the network learns an intermediate representation of the physiological signals that better suits the task at hand. Experimental evaluations show that, this very first study on end-to-end learning of emotion based on physiology , yields significantly better performance in comparison to existing work on the challenging RECOLA database, which includes fully spontaneous affective behaviors displayed during naturalistic interactions. Furthermore, we gain better understanding of the models' inner representations, by demonstrating that some cells' activations in the convolutional network are correlated to a large extent with hand-crafted features

End-to-end Learning for Dimensional Emotion Recognition from Physiological Signals

International audienceDimensional emotion recognition from physiological signals is a highly challenging task. Common methods rely on hand-crafted features that do not yet provide the performance necessary for real-life application. In this work, we exploit a series of convolutional and recurrent neural networks to predict affect from physiological signals, such as electrocardiogram and electrodermal activity, directly from the raw time representation. The motivation behind this so-called end-to-end approach is that, ultimately, the network learns an intermediate representation of the physiological signals that better suits the task at hand. Experimental evaluations show that, this very first study on end-to-end learning of emotion based on physiology , yields significantly better performance in comparison to existing work on the challenging RECOLA database, which includes fully spontaneous affective behaviors displayed during naturalistic interactions. Furthermore, we gain better understanding of the models' inner representations, by demonstrating that some cells' activations in the convolutional network are correlated to a large extent with hand-crafted features

Different approaches to long-term treatment of aHUS due to MCP mutations: a multicenter analysis

Background!#!Atypical hemolytic uremic syndrome (aHUS) is a rare, life-threatening microangiopathy, frequently causing kidney failure. Inhibition of the terminal complement complex with eculizumab is the only licensed treatment but mostly requires long-term administration and risks severe side effects. The underlying genetic cause of aHUS is thought to influence the severity of initial and recurring episodes, with milder courses in patients with mutations in membrane cofactor protein (MCP).!##!Methods!#!Twenty pediatric cases of aHUS due to isolated heterozygous MCP mutations were reported from 12 German pediatric nephrology centers to describe initial presentation, timing of relapses, treatment, and kidney outcome.!##!Results!#!The median age of onset was 4.6 years, with a female to male ratio of 1:3. Without eculizumab maintenance therapy, 50% (9/18) of the patients experienced a first relapse after a median period of 3.8 years. Kaplan-Meier analysis showed a relapse-free survival of 93% at 1 year. Four patients received eculizumab long-term treatment, while 3 patients received short courses. We could not show a benefit from complement blockade therapy on long term kidney function, independent of short-term or long-term treatment. To prevent 1 relapse with eculizumab, the theoretical number-needed-to-treat (NNT) was 15 for the first year and 3 for the first 5 years after initial presentation.!##!Conclusion!#!Our study shows that heterozygous MCP mutations cause aHUS with a risk of first relapse of about 10% per year, resulting in large NNTs for prevention of relapses with eculizumab. More studies are needed to define an optimal treatment schedule for patients with MCP mutations to minimize the risks of the disease and treatment