Synthesizing Normalized Faces from Facial Identity Features

We present a method for synthesizing a frontal, neutral-expression image of a person's face given an input face photograph. This is achieved by learning to generate facial landmarks and textures from features extracted from a facial-recognition network. Unlike previous approaches, our encoding feature vector is largely invariant to lighting, pose, and facial expression. Exploiting this invariance, we train our decoder network using only frontal, neutral-expression photographs. Since these photographs are well aligned, we can decompose them into a sparse set of landmark points and aligned texture maps. The decoder then predicts landmarks and textures independently and combines them using a differentiable image warping operation. The resulting images can be used for a number of applications, such as analyzing facial attributes, exposure and white balance adjustment, or creating a 3-D avatar

Cultural Heritage Restoration of a Hemispherical Vault by 3D Modelling and Projection of Video Images with Unknown Parameters and from Unknown Locations

[EN] Reverse engineering applied to architectural restoration for the reconstruction of structural surfaces depends on metric precision. Sometimes there are elements on these surfaces whose value is even higher than the building itself. This is the case for many churches whose ceilings have pictorial works of art. Reconstruction requires the existence of some identifiable remainder and/or a surface geometry that enables mathematical development. In our case, the vault has an irregular hemispherical geometry (without possible mathematical development), and there are no significant remains of the painting (which was destroyed by a fire). Through the 3D modelling of the irregular vault and two historic frames with a camera of unknown geometry, an inverse methodology is designed to project the original painting without metric deformations. For this, a new methodology to locate the camera positions is developed. After, a 3D virtual mathematical model of the complete image on the vault is calculated, and from it, partial 3D virtual images are automatically calculated depending on the variable unknown positions of the video cannons (distributed along the upper corridor of the apse) that will project them (visually forming a perfect complete 3D image)Herráez Boquera, J.; Denia Rios, JL.; Priego De Los Santos, E.; Navarro Esteve, PJ.; Martín Sánchez, MT.; Rodríguez Pereña, J. (2021). Cultural Heritage Restoration of a Hemispherical Vault by 3D Modelling and Projection of Video Images with Unknown Parameters and from Unknown Locations. Applied Sciences. 11(12):1-12. https://doi.org/10.3390/app11125323112111

Multi-touch Detection and Semantic Response on Non-parametric Rear-projection Surfaces

The ability of human beings to physically touch our surroundings has had a profound impact on our daily lives. Young children learn to explore their world by touch; likewise, many simulation and training applications benefit from natural touch interactivity. As a result, modern interfaces supporting touch input are ubiquitous. Typically, such interfaces are implemented on integrated touch-display surfaces with simple geometry that can be mathematically parameterized, such as planar surfaces and spheres; for more complicated non-parametric surfaces, such parameterizations are not available. In this dissertation, we introduce a method for generalizable optical multi-touch detection and semantic response on uninstrumented non-parametric rear-projection surfaces using an infrared-light-based multi-camera multi-projector platform. In this paradigm, touch input allows users to manipulate complex virtual 3D content that is registered to and displayed on a physical 3D object. Detected touches trigger responses with specific semantic meaning in the context of the virtual content, such as animations or audio responses. The broad problem of touch detection and response can be decomposed into three major components: determining if a touch has occurred, determining where a detected touch has occurred, and determining how to respond to a detected touch. Our fundamental contribution is the design and implementation of a relational lookup table architecture that addresses these challenges through the encoding of coordinate relationships among the cameras, the projectors, the physical surface, and the virtual content. Detecting the presence of touch input primarily involves distinguishing between touches (actual contact events) and hovers (near-contact proximity events). We present and evaluate two algorithms for touch detection and localization utilizing the lookup table architecture. One of the algorithms, a bounded plane sweep, is additionally able to estimate hover-surface distances, which we explore for interactions above surfaces. The proposed method is designed to operate with low latency and to be generalizable. We demonstrate touch-based interactions on several physical parametric and non-parametric surfaces, and we evaluate both system accuracy and the accuracy of typical users in touching desired targets on these surfaces. In a formative human-subject study, we examine how touch interactions are used in the context of healthcare and present an exploratory application of this method in patient simulation. A second study highlights the advantages of touch input on content-matched physical surfaces achieved by the proposed approach, such as decreases in induced cognitive load, increases in system usability, and increases in user touch performance. In this experiment, novice users were nearly as accurate when touching targets on a 3D head-shaped surface as when touching targets on a flat surface, and their self-perception of their accuracy was higher

Computational Light Routing: 3D Printed Optical Fibers for Sensing and Display

Despite recent interest in digital fabrication, there are still few algorithms that provide control over how light propagates inside a solid object. Existing methods either work only on the surface or restrict themselves to light diffusion in volumes. We use multi-material 3D printing to fabricate objects with embedded optical fibers, exploiting total internal reflection to guide light inside an object. We introduce automatic fiber design algorithms together with new manufacturing techniques to route light between two arbitrary surfaces. Our implicit algorithm optimizes light transmission by minimizing fiber curvature and maximizing fiber separation while respecting constraints such as fiber arrival angle. We also discuss the influence of different printable materials and fiber geometry on light propagation in the volume and the light angular distribution when exiting the fiber. Our methods enable new applications such as surface displays of arbitrary shape, touch-based painting of surfaces, and sensing a hemispherical light distribution in a single shot.National Science Foundation (U.S.) (Grant CCF-1012147)National Science Foundation (U.S.) (Grant IIS-1116296)United States. Defense Advanced Research Projects Agency (Grant N66001-12-1-4242)Intel Corporation (Science and Technology Center for Visual Computing)Alfred P. Sloan Foundation (Sloan Research Fellowship

Expressive rendering of mountainous terrain

technical reportPainters and cartographers have developed artistic landscape rendering techniques for centuries. Such renderings can visualize complex three-dimensional landscapes in a pleasing and understandable way. In this work we examine a particular type of artistic depiction, panorama maps, in terms of function and style, and we develop methods to automatically generate panorama map reminiscent renderings from GIS data. In particular, we develop image-based procedural surface textures for mountainous terrain. Our methods use the structural information present in the terrain and are developed with perceptual metrics and artistic considerations in mind

Multi-layer Dictionary Learning for Image Classification

International audienceThis paper presents a multi-layer dictionary learning method for classification tasks. The goal of the proposed multi-layer framework is to use the supervised dictionary learning approach locally on raw images in order to learn local features. This method starts by building a sparse representation at the patch-level and relies on a hierarchy of learned dictionaries to output a global sparse representation for the whole image. It relies on a succession of sparse coding and pooling steps in order to find an efficient representation of the data for classification. This method has been tested on a classification task with good results