FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality

We introduce FaceVR, a novel method for gaze-aware facial reenactment in the Virtual Reality (VR) context. The key component of FaceVR is a robust algorithm to perform real-time facial motion capture of an actor who is wearing a head-mounted display (HMD), as well as a new data-driven approach for eye tracking from monocular videos. In addition to these face reconstruction components, FaceVR incorporates photo-realistic re-rendering in real time, thus allowing artificial modifications of face and eye appearances. For instance, we can alter facial expressions, change gaze directions, or remove the VR goggles in realistic re-renderings. In a live setup with a source and a target actor, we apply these newly-introduced algorithmic components. We assume that the source actor is wearing a VR device, and we capture his facial expressions and eye movement in real-time. For the target video, we mimic a similar tracking process; however, we use the source input to drive the animations of the target video, thus enabling gaze-aware facial reenactment. To render the modified target video on a stereo display, we augment our capture and reconstruction process with stereo data. In the end, FaceVR produces compelling results for a variety of applications, such as gaze-aware facial reenactment, reenactment in virtual reality, removal of VR goggles, and re-targeting of somebody's gaze direction in a video conferencing call

{Face2Face}: {R}eal-time Face Capture and Reenactment of {RGB} Videos

We present Face2Face, a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. To this end, we first address the under-constrained problem of facial identity recovery from monocular video by non-rigid model-based bundling. At run time, we track facial expressions of both source and target video using a dense photometric consistency measure. Reenactment is then achieved by fast and efficient deformation transfer between source and target. The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit. Finally, we convincingly re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination. We demonstrate our method in a live setup, where Youtube videos are reenacted in real time

Achieving Efficient and Realistic Full-Radar Simulations and Automatic Data Annotation by Exploiting Ray Meta Data from a Radar Ray Tracing Simulator

In this work, a novel radar simulation concept for efficiently simulating realistic radar data for range, Doppler, and arbitrary antenna positions is introduced. With the concept, the simulated radar signal can also be automatically annotated by splitting it into multiple parts. Annotations that are almost perfect - including the annotation of exotic effects, such as multi-path - can be produced with this approach. Signal parts originating from different parts of an object can be labelled with it as well. To this end, the computation process used in a Monte Carlo shooting and bouncing rays (SBR) simulator is adapted. By considering the hits of each simulated ray, various meta data can be stored, such as hit position, mesh pointer, object IDs, and many more. This collected meta data can then be utilized to predict path-length changes caused by object motion to obtain Doppler information or to apply specific ray filter rules to obtain radar signals that only fulfil specific conditions, such as multiple bounces, or signals that contain specific object IDs. Using this approach, perfect, and otherwise almost impossible, annotation schemes can be realized

A flexible, extensible object oriented real-time near photorealistic visualization system:the system framework design

In this paper we describe a novel, extensible visualization system currently under development at Aston University. We introduce modern programming methods, such as the use of data driven programming, design patterns, and the careful definition of interfaces to allow easy extension using plug-ins, to 3D landscape visualization software. We combine this with modern developments in computer graphics, such as vertex and fragment shaders, to create an extremely flexible, extensible real-time near photorealistic visualization system. In this paper we show the design of the system and the main sub-components. We stress the role of modern programming practices and illustrate the benefits these bring to 3D visualization. © 2006 Springer-Verlag Berlin Heidelberg

A virally encoded de-sumoylase activity is required for cytomegalovirus reactivation from latency

A subset of viral genes is required for the long term latent infection of haematopoietic cells by human cytomegalovirus (HCMV). Here we show that a latency-associated gene product (LUNA) promotes the disruption of cellular PML bodies during latency. Mutation and inhibitor studies reveal that LUNA encodes a novel de-sumoylase activity responsible for this disruption. Specifically, LUNA activity expressed a conserved Asp-Cys-Gly motif common to all de-sumoylases. Importantly, mutation of the putative catalytic cysteine was sufficient to reverse LUNA mediated PML dispersal and markedly reduced the efficiency of viral reactivation. The depletion of PML from cells was sufficient to rescue the reactivation of the LUNA-deficient viruses arguing that targeting PML was an important biological role of LUNA. Finally, we demonstrate that reactivation of naturally latent HCMV is blocked by de-sumoylase inhibitors. Thus latent HCMV primes the cellular environment for efficient reactivation via the activity of a virally encoded de-sumoylas

Killer cell proteases can target viral immediate-early proteins to control human cytomegalovirus infection in a noncytotoxic manner.

Human cytomegalovirus (HCMV) is the most frequent viral cause of congenital defects and can trigger devastating disease in immune-suppressed patients. Cytotoxic lymphocytes (CD8+ T cells and NK cells) control HCMV infection by releasing interferon-γ and five granzymes (GrA, GrB, GrH, GrK, GrM), which are believed to kill infected host cells through cleavage of intracellular death substrates. However, it has recently been demonstrated that the in vivo killing capacity of cytotoxic T cells is limited and multiple T cell hits are required to kill a single virus-infected cell. This raises the question whether cytotoxic lymphocytes can use granzymes to control HCMV infection in a noncytotoxic manner. Here, we demonstrate that (primary) cytotoxic lymphocytes can block HCMV dissemination independent of host cell death, and interferon-α/β/γ. Prior to killing, cytotoxic lymphocytes induce the degradation of viral immediate-early (IE) proteins IE1 and IE2 in HCMV-infected cells. Intriguingly, both IE1 and/or IE2 are directly proteolyzed by all human granzymes, with GrB and GrM being most efficient. GrB and GrM cleave IE1 after Asp398 and Leu414, respectively, likely resulting in IE1 aberrant cellular localization, IE1 instability, and functional impairment of IE1 to interfere with the JAK-STAT signaling pathway. Furthermore, GrB and GrM cleave IE2 after Asp184 and Leu173, respectively, resulting in IE2 aberrant cellular localization and functional abolishment of IE2 to transactivate the HCMV UL112 early promoter. Taken together, our data indicate that cytotoxic lymphocytes can also employ noncytotoxic ways to control HCMV infection, which may be explained by granzyme-mediated targeting of indispensable viral proteins during lytic infection

Highly conserved interaction profiles between clinically relevant mutants of the cytomegalovirus CDK-like kinase pUL97 and human cyclins: functional significance of cyclin H

The complex host interaction network of human cytomegalovirus (HCMV) involves the regulatory protein kinase pUL97, which represents a viral cyclin-dependent kinase (CDK) ortholog. pUL97 interacts with the three human cyclin types T1, H, and B1, whereby the binding region of cyclin T1 and the pUL97 oligomerization region were both assigned to amino acids 231-280. We further addressed the question of whether HCMVs harboring mutations in ORF-UL97, i.e., short deletions or resistance-conferring point mutations, are affected in the interaction with human cyclins and viral replication. To this end, clinically relevant UL97 drug-resistance-conferring mutants were analyzed by whole-genome sequencing and used for genetic marker transfer experiments. The recombinant HCMVs indicated conservation of pUL97-cyclin interaction, since all viral UL97 point mutants continued to interact with the analyzed cyclin types and exerted wild-type-like replication fitness. In comparison, recombinant HCMVs UL97 Δ231-280 and also the smaller deletion Δ236-275, but not Δ241-270, lost interaction with cyclins T1 and H, showed impaired replication efficiency, and also exhibited reduced kinase activity. Moreover, a cellular knock-out of cyclins B1 or T1 did not alter HCMV replication phenotypes or pUL97 kinase activity, possibly indicating alternative, compensatory pUL97-cyclin interactions. In contrast, however, cyclin H knock-out, similar to virus deletion mutants in the pUL97-cyclin H binding region, exhibited strong defective phenotypes of HCMV replication, as supported by reduced pUL97 kinase activity in a cyclin H-dependent coexpression setting. Thus, cyclin H proved to be a very relevant determinant of pUL97 kinase activity and viral replication efficiency. As a conclusion, the results provide evidence for the functional importance of pUL97-cyclin interaction. High selective pressure on the formation of pUL97-cyclin complexes was identified by the use of clinically relevant mutants

Cyclin-Dependent Kinase Activity Controls the Onset of the HCMV Lytic Cycle

The onset of human cytomegalovirus (HCMV) lytic infection is strictly synchronized with the host cell cycle. Infected G0/G1 cells support viral immediate early (IE) gene expression and proceed to the G1/S boundary where they finally arrest. In contrast, S/G2 cells can be infected but effectively block IE gene expression and this inhibition is not relieved until host cells have divided and reentered G1. During latent infection IE gene expression is also inhibited, and for reactivation to occur this block to IE gene expression must be overcome. It is only poorly understood which viral and/or cellular activities maintain the block to cell cycle or latency-associated viral IE gene repression and whether the two mechanisms may be linked. Here, we show that the block to IE gene expression during S and G2 phase can be overcome by both genotoxic stress and chemical inhibitors of cellular DNA replication, pointing to the involvement of checkpoint-dependent signaling pathways in controlling IE gene repression. Checkpoint-dependent rescue of IE expression strictly requires p53 and in the absence of checkpoint activation is mimicked by proteasomal inhibition in a p53 dependent manner. Requirement for the cyclin dependent kinase (CDK) inhibitor p21 downstream of p53 suggests a pivotal role for CDKs in controlling IE gene repression in S/G2 and treatment of S/G2 cells with the CDK inhibitor roscovitine alleviates IE repression independently of p53. Importantly, CDK inhibiton also overcomes the block to IE expression during quiescent infection of NTera2 (NT2) cells. Thus, a timely block to CDK activity not only secures phase specificity of the cell cycle dependent HCMV IE gene expression program, but in addition plays a hitherto unrecognized role in preventing the establishment of a latent-like state