A survey on mouth modeling and analysis for Sign Language recognition

© 2015 IEEE.Around 70 million Deaf worldwide use Sign Languages (SLs) as their native languages. At the same time, they have limited reading/writing skills in the spoken language. This puts them at a severe disadvantage in many contexts, including education, work, usage of computers and the Internet. Automatic Sign Language Recognition (ASLR) can support the Deaf in many ways, e.g. by enabling the development of systems for Human-Computer Interaction in SL and translation between sign and spoken language. Research in ASLR usually revolves around automatic understanding of manual signs. Recently, ASLR research community has started to appreciate the importance of non-manuals, since they are related to the lexical meaning of a sign, the syntax and the prosody. Nonmanuals include body and head pose, movement of the eyebrows and the eyes, as well as blinks and squints. Arguably, the mouth is one of the most involved parts of the face in non-manuals. Mouth actions related to ASLR can be either mouthings, i.e. visual syllables with the mouth while signing, or non-verbal mouth gestures. Both are very important in ASLR. In this paper, we present the first survey on mouth non-manuals in ASLR. We start by showing why mouth motion is important in SL and the relevant techniques that exist within ASLR. Since limited research has been conducted regarding automatic analysis of mouth motion in the context of ALSR, we proceed by surveying relevant techniques from the areas of automatic mouth expression and visual speech recognition which can be applied to the task. Finally, we conclude by presenting the challenges and potentials of automatic analysis of mouth motion in the context of ASLR

The Conditional Lucas & Kanade Algorithm

The Lucas & Kanade (LK) algorithm is the method of choice for efficient dense image and object alignment. The approach is efficient as it attempts to model the connection between appearance and geometric displacement through a linear relationship that assumes independence across pixel coordinates. A drawback of the approach, however, is its generative nature. Specifically, its performance is tightly coupled with how well the linear model can synthesize appearance from geometric displacement, even though the alignment task itself is associated with the inverse problem. In this paper, we present a new approach, referred to as the Conditional LK algorithm, which: (i) directly learns linear models that predict geometric displacement as a function of appearance, and (ii) employs a novel strategy for ensuring that the generative pixel independence assumption can still be taken advantage of. We demonstrate that our approach exhibits superior performance to classical generative forms of the LK algorithm. Furthermore, we demonstrate its comparable performance to state-of-the-art methods such as the Supervised Descent Method with substantially less training examples, as well as the unique ability to "swap" geometric warp functions without having to retrain from scratch. Finally, from a theoretical perspective, our approach hints at possible redundancies that exist in current state-of-the-art methods for alignment that could be leveraged in vision systems of the future.Comment: 17 pages, 11 figure

3D face morphable models "In-The-Wild"

3D Morphable Models (3DMMs) are powerful statistical models of 3D facial shape and texture, and among the state-of-the-art methods for reconstructing facial shape from single images. With the advent of new 3D sensors, many 3D facial datasets have been collected containing both neutral as well as expressive faces. However, all datasets are captured under controlled conditions. Thus, even though powerful 3D facial shape models can be learnt from such data, it is difficult to build statistical texture models that are sufficient to reconstruct faces captured in unconstrained conditions (in-the-wild). In this paper, we propose the first, to the best of our knowledge, in-the-wild 3DMM by combining a powerful statistical model of facial shape, which describes both identity and expression, with an in-the-wild texture model. We show that the employment of such an in-the-wild texture model greatly simplifies the fitting procedure, because there is no need to optimise with regards to the illumination parameters. Furthermore, we propose a new fast algorithm for fitting the 3DMM in arbitrary images. Finally, we have captured the first 3D facial database with relatively unconstrained conditions and report quantitative evaluations with state-of-the-art performance. Complementary qualitative reconstruction results are demonstrated on standard in-the-wild facial databases

DenseReg: fully convolutional dense shape regression in-the-wild

In this paper we propose to learn a mapping from image pixels into a dense template grid through a fully convolutional network. We formulate this task as a regression problem and train our network by leveraging upon manually annotated facial landmarks “in-the-wild”. We use such landmarks to establish a dense correspondence field between a three-dimensional object template and the input image, which then serves as the ground-truth for training our regression system. We show that we can combine ideas from semantic segmentation with regression networks, yielding a highly-accurate ‘quantized regression’ architecture. Our system, called DenseReg, allows us to estimate dense image-to-template correspondences in a fully convolutional manner. As such our network can provide useful correspondence information as a stand-alone system, while when used as an initialization for Statistical Deformable Models we obtain landmark localization results that largely outperform the current state-of-the-art on the challenging 300W benchmark. We thoroughly evaluate our method on a host of facial analysis tasks, and demonstrate its use for other correspondence estimation tasks, such as the human body and the human ear. DenseReg code is made available at http://alpguler.com/DenseReg.html along with supplementary materials

3D Reconstruction of 'In-the-Wild' Faces in Images and Videos

This is the author accepted manuscript. The final version is available from IEEE via the DOI in this record 3D Morphable Models (3DMMs) are powerful statistical models of 3D facial shape and texture, and are among the state-of-the-art methods for reconstructing facial shape from single images. With the advent of new 3D sensors, many 3D facial datasets have been collected containing both neutral as well as expressive faces. However, all datasets are captured under controlled conditions. Thus, even though powerful 3D facial shape models can be learnt from such data, it is difficult to build statistical texture models that are sufficient to reconstruct faces captured in unconstrained conditions ('in-the-wild'). In this paper, we propose the first 'in-the-wild' 3DMM by combining a statistical model of facial identity and expression shape with an 'in-the-wild' texture model. We show that such an approach allows for the development of a greatly simplified fitting procedure for images and videos, as there is no need to optimise with regards to the illumination parameters. We have collected three new benchmarks that combine 'in-the-wild' images and video with ground truth 3D facial geometry, the first of their kind, and report extensive quantitative evaluations using them that demonstrate our method is state-of-the-art.Engineering and Physical Sciences Research Council (EPSRC

Estimating correspondences of deformable objects "in-the-wild"

This is the author accepted manuscript. The final version is available from IEEE via the DOI in this recordDuring the past few years we have witnessed the development of many methodologies for building and fitting Statistical Deformable Models (SDMs). The construction of accurate SDMs requires careful annotation of images with regards to a consistent set of landmarks. However, the manual annotation of a large amount of images is a tedious, laborious and expensive procedure. Furthermore, for several deformable objects, e.g. human body, it is difficult to define a consistent set of landmarks, and, thus, it becomes impossible to train humans in order to accurately annotate a collection of images. Nevertheless, for the majority of objects, it is possible to extract the shape by object segmentation or even by shape drawing. In this paper, we show for the first time, to the best of our knowledge, that it is possible to construct SDMs by putting object shapes in dense correspondence. Such SDMs can be built with much less effort for a large battery of objects. Additionally, we show that, by sampling the dense model, a part-based SDM can be learned with its parts being in correspondence. We employ our framework to develop SDMs of human arms and legs, which can be used for the segmentation of the outline of the human body, as well as to provide better and more consistent annotations for body joints.Engineering and Physical Sciences Research Council (EPSRC)TekesEuropean Community Horizon 202

Crystal Structure Studies of Human Dental Apatite as a Function of Age

Studies of the average crystal structure properties of human dental apatite as a function of the tooth-age in the range of 5-87 years are reported. The crystallinity of the dental hydroxyapatite decreases with the tooth-age. The a-lattice constant that is associated with the carbonate content in carbonate apatite decreases with the tooth-age in a systematic way, whereas the c-lattice constant does not change significantly. Thermogravimetric measurements demonstrate an increase of the carbonate content with the tooth-age. FTIR spectroscopy reveals both, B and A-type carbonate substitutions with the B-type greater than the A-type substitution by a factor up to ~5. An increase of the carbonate content as a function of the tooth-age can be deduced from the ratio of the v2 CO3 to the v1 PO4 IR modes.Comment: 17 page

Trained Immunity Confers Prolonged Protection From Listeriosis.

Trained immunity refers to the ability of the innate immune system exposed to a first challenge to provide an enhanced response to a secondary homologous or heterologous challenge. We reported that training induced with β-glucan one week before infection confers protection against a broad-spectrum of lethal bacterial infections. Whether this protection persists over time is unknown. To tackle this question, we analyzed the immune status and the response to Listeria monocytogenes (L. monocytogenes) of mice trained 9 weeks before analysis. The induction of trained immunity increased bone marrow myelopoiesis and blood counts of Ly6C <sup>high</sup> inflammatory monocytes and polymorphonuclear neutrophils (PMNs). Ex vivo, whole blood, PMNs and monocytes from trained mice produced increased levels of cytokines in response to microbial products and limited the growth of L. monocytogenes. In vivo, following challenge with L. monocytogenes, peripheral blood leukocytes were massively depleted in control mice but largely preserved in trained mice. PMNs were reduced also in the spleen from control mice, and increased in the spleen of trained mice. In transwell experiments, PMNs from trained mice showed increased spontaneous migration and CXCL2/MIP2α-induced chemotaxis, suggesting that training promotes the migration of PMNs in peripheral organs targeted by L. monocytogenes. Trained PMNs and monocytes had higher glycolytic activity and mitochondrial respiration than control cells when exposed to L. monocytogenes. Bacterial burden and dissemination in blood, spleen and liver as well as systemic cytokines and inflammation (multiplex bead assay and bioluminescence imaging) were reduced in trained mice. In full agreement with these results, mice trained 9 weeks before infection were powerfully protected from lethal listeriosis. Altogether, these data suggest that training increases the generation and the antimicrobial activity of PMNs and monocytes, which may confer prolonged protection from lethal bacterial infection

Explicit Evidence Systems with Common Knowledge

Justification logics are epistemic logics that explicitly include justifications for the agents' knowledge. We develop a multi-agent justification logic with evidence terms for individual agents as well as for common knowledge. We define a Kripke-style semantics that is similar to Fitting's semantics for the Logic of Proofs LP. We show the soundness, completeness, and finite model property of our multi-agent justification logic with respect to this Kripke-style semantics. We demonstrate that our logic is a conservative extension of Yavorskaya's minimal bimodal explicit evidence logic, which is a two-agent version of LP. We discuss the relationship of our logic to the multi-agent modal logic S4 with common knowledge. Finally, we give a brief analysis of the coordinated attack problem in the newly developed language of our logic

Long-range transfer of electron-phonon coupling in oxide superlattices

The electron-phonon interaction is of central importance for the electrical and thermal properties of solids, and its influence on superconductivity, colossal magnetoresistance, and other many-body phenomena in correlated-electron materials is currently the subject of intense research. However, the non-local nature of the interactions between valence electrons and lattice ions, often compounded by a plethora of vibrational modes, present formidable challenges for attempts to experimentally control and theoretically describe the physical properties of complex materials. Here we report a Raman scattering study of the lattice dynamics in superlattices of the high-temperature superconductor $\bf YBa_2 Cu_3 O_7$ and the colossal-magnetoresistance compound $\bf La_{2/3}Ca_{1/3}MnO_{3}$ that suggests a new approach to this problem. We find that a rotational mode of the MnO$_6$ octahedra in $\bf La_{2/3}Ca_{1/3}MnO_{3}$ experiences pronounced superconductivity-induced lineshape anomalies, which scale linearly with the thickness of the $\bf YBa_2 Cu_3 O_7$ layers over a remarkably long range of several tens of nanometers. The transfer of the electron-phonon coupling between superlattice layers can be understood as a consequence of long-range Coulomb forces in conjunction with an orbital reconstruction at the interface. The superlattice geometry thus provides new opportunities for controlled modification of the electron-phonon interaction in complex materials.Comment: 13 pages, 4 figures. Revised version to be published in Nature Material