Less is More: Micro-expression Recognition from Video using Apex Frame

Despite recent interest and advances in facial micro-expression research, there is still plenty room for improvement in terms of micro-expression recognition. Conventional feature extraction approaches for micro-expression video consider either the whole video sequence or a part of it, for representation. However, with the high-speed video capture of micro-expressions (100-200 fps), are all frames necessary to provide a sufficiently meaningful representation? Is the luxury of data a bane to accurate recognition? A novel proposition is presented in this paper, whereby we utilize only two images per video: the apex frame and the onset frame. The apex frame of a video contains the highest intensity of expression changes among all frames, while the onset is the perfect choice of a reference frame with neutral expression. A new feature extractor, Bi-Weighted Oriented Optical Flow (Bi-WOOF) is proposed to encode essential expressiveness of the apex frame. We evaluated the proposed method on five micro-expression databases: CAS(ME)$^2$, CASME II, SMIC-HS, SMIC-NIR and SMIC-VIS. Our experiments lend credence to our hypothesis, with our proposed technique achieving a state-of-the-art F1-score recognition performance of 61% and 62% in the high frame rate CASME II and SMIC-HS databases respectively.Comment: 14 pages double-column, author affiliations updated, acknowledgment of grant support adde

Micro-attention for micro-expression recognition

Micro-expression, for its high objectivity in emotion detection, has emerged to be a promising modality in affective computing. Recently, deep learning methods have been successfully introduced into the micro-expression recognition area. Whilst the higher recognition accuracy achieved, substantial challenges in micro-expression recognition remain. The existence of micro expression in small-local areas on face and limited size of available databases still constrain the recognition accuracy on such emotional facial behavior. In this work, to tackle such challenges, we propose a novel attention mechanism called micro-attention cooperating with residual network. Micro-attention enables the network to learn to focus on facial areas of interest covering different action units. Moreover, coping with small datasets, the micro-attention is designed without adding noticeable parameters while a simple yet efficient transfer learning approach is together utilized to alleviate the overfitting risk. With extensive experimental evaluations on three benchmarks (CASMEII, SAMM and SMIC) and post-hoc feature visualizations, we demonstrate the effectiveness of the proposed micro-attention and push the boundary of automatic recognition of micro-expression

Inferring Facial and Body Language

Machine analysis of human facial and body language is a challenging topic in computer vision, impacting on important applications such as human-computer interaction and visual surveillance. In this thesis, we present research building towards computational frameworks capable of automatically understanding facial expression and behavioural body language. The thesis work commences with a thorough examination in issues surrounding facial representation based on Local Binary Patterns (LBP). Extensive experiments with different machine learning techniques demonstrate that LBP features are efficient and effective for person-independent facial expression recognition, even in low-resolution settings. We then present and evaluate a conditional mutual information based algorithm to efficiently learn the most discriminative LBP features, and show the best recognition performance is obtained by using SVM classifiers with the selected LBP features. However, the recognition is performed on static images without exploiting temporal behaviors of facial expression. Subsequently we present a method to capture and represent temporal dynamics of facial expression by discovering the underlying low-dimensional manifold. Locality Preserving Projections (LPP) is exploited to learn the expression manifold in the LBP based appearance feature space. By deriving a universal discriminant expression subspace using a supervised LPP, we can effectively align manifolds of different subjects on a generalised expression manifold. Different linear subspace methods are comprehensively evaluated in expression subspace learning. We formulate and evaluate a Bayesian framework for dynamic facial expression recognition employing the derived manifold representation. However, the manifold representation only addresses temporal correlations of the whole face image, does not consider spatial-temporal correlations among different facial regions. We then employ Canonical Correlation Analysis (CCA) to capture correlations among face parts. To overcome the inherent limitations of classical CCA for image data, we introduce and formalise a novel Matrix-based CCA (MCCA), which can better measure correlations in 2D image data. We show this technique can provide superior performance in regression and recognition tasks, whilst requiring significantly fewer canonical factors. All the above work focuses on facial expressions. However, the face is usually perceived not as an isolated object but as an integrated part of the whole body, and the visual channel combining facial and bodily expressions is most informative. Finally we investigate two understudied problems in body language analysis, gait-based gender discrimination and affective body gesture recognition. To effectively combine face and body cues, CCA is adopted to establish the relationship between the two modalities, and derive a semantic joint feature space for the feature-level fusion. Experiments on large data sets demonstrate that our multimodal systems achieve the superior performance in gender discrimination and affective state analysis.Research studentship of Queen Mary, the International Travel Grant of the Royal Academy of Engineering, and the Royal Society International Joint Project

Bone remodeling simulations: challenges, problems and applications

La remodelación ósea es el mecanismo que regula la relación entre la morfología del hueso y sus cargas mecánicas externas. Se basa en el hecho de que el hueso se adapta a las condiciones mecánicas a las que está expuesto. Varios factores mecánicos y bioquímicos pueden regular la respuesta final de la remodelación ósea. De hecho, se considera que la remodelación ósea pretende alcanzar varios objetivos mecánicos: reparar el daño para reducir el riesgo de fractura y optimizar la rigidez y resistencia con el mínimo peso. Durante las últimas décadas, se han propuesto un gran número de leyes matemáticas implementadas numéricamente, pero la mayoría de ellas presentan diferentes problemas como la estabilidad, la convergencia o la dependencia de las condiciones iniciales. Por tanto, el objetivo principal de esta tesis es estudiar los modelos de remodelación ósea, mostrando sus retos, su problemática y su aplicación en el ámbito clínico. En primer lugar, se han revisado dos teorías clásicas de la remodelación ósea (conocidas como modelo de Stanford y modelo de Doblaré y García). En ambos casos, se propone un aspecto novedoso planteando que el estímulo homeostático de referencia no es constante, sino que depende localmente de la historia de carga que cada punto local está soportando. Como consecuencia directa de esta hipótesis, se demuestra que las inestabilidades numéricas que normalmente presentan estos algoritmos, pueden quedar resueltas, mejorando claramente los resultados finales. Esta metodología se aplicó a un modelo de elementos finitos 2D/3D mejorando la convergencia de la solución y asegurando su estabilidad numérica a largo plazo. Por otra parte, en un intento de dilucidar las características de adaptación mecánica del hueso en diferentes escalas, se plantea una relación a nivel órgano y a nivel de tejido que depende de un cambio en el estímulo homeostático de referencia acorde con la densidad aparente, mientras que se considera que la densidad de energía de deformación a nivel de tejido permanece invariante. Esta hipótesis mejora la unicidad de la solución y la hace independiente de las condiciones iniciales, ayudando también a su estabilidad numérica. Además, en esta tesis se aborda el modelado de paciente específico que es un tema que está adquiriendo cada vez más importancia. Una de las principales dificultades en la creación de modelos de paciente específico, es la determinación de las cargas que el hueso está realmente soportando. Los datos relativos a pacientes específicos, como la geometría ósea y la distribución de la densidad ósea, puede ser utilizados para determinar estas cargas. Por lo tanto, se ha estudiado la estimación de la cargas con tres diferentes técnicas matemáticas: regresión lineal, redes neuronales artificiales y máquinas de soporte vector. Estas técnicas se han aplicado a un ejemplo teórico para obtener las cargas a través de la densidad aparente que se predice con los modelos de remodelación ósea. Para concluir, la metodología desarrollada que combina modelos de remodelación ósea con redes neuronales se ha aplicado a la predicción de las cargas de cinco tibias de pacientes. Para ello, se han determinado la geometría y la distribución de la densidad a partir de un TAC y se han introducido los valores de densidad en el modelo previamente desarrollado, obteniendo así, las cargas específicas de las tibias de los pacientes. Con el fin de validar la capacidad de esta novedosa técnica, se han comparado las cargas obtenidas de la técnica propuesta con las cargas obtenidas en un análisis de marcha de dichos pacientes. Los errores obtenidos en las predicciones han sido menores de un 6 %. Por lo tanto, se puede concluir que la metodología aquí propuesta, permite determinar de forma aproximada las cargas que un hueso específico soporta.Bone remodeling is the mechanism that regulates the relationship between bone morphology and its external mechanical loads. It is based on the fact that bone adapts itself to the mechanical conditions to which it is exposed. Several mechanical and biochemical factors may regulate the final bone remodeling response. In fact, bone remodeling is hypothesized to achieve several mechanical objectives: repair damage to reduce the risk of fracture and optimize stiffness and strength with minimum weight. During recent decades, a great number of numerically implemented mathematical laws have been proposed, but most of them present different problems as stability, convergence or dependence of the initial conditions. Thus, the main scope of this Thesis is to study bone remodeling models, showing their challenges, their problematic and their applicability in the clinical setting. Firstly, we revisit two classical bone remodeling theories (Stanford model and Doblaré and García model). In both of them, the reference homeostatic stimulus is hypothesized that is not constant, but it is locally dependent on the loading history that each local point is effectively supporting. As a direct consequence of this assumption, we demonstrate that the numerical instabilities that all these algorithms normally present can be solved, clearly improving the final results. For this reason, we applied this methodology to 2D/3D finite element models. This contribution improves the convergence of the solution, leading to its numerical stability in the long-term. In an attempt to elucidate the features of bone adaptation at the di erent scales, we hypothesize that the relationship between the organ level and tissue level depends on the reference homeostatic stimulus changes according to the density and the tissue effective energy remains unchanged. This assumption improves the uniqueness of the solution, independently of the initial conditions selected and clearly helps in its numerical stability. In addition, patient-specific modeling is becoming increasingly important. One of the most challenging diffculties in creating patient-specific models is the determination of the specific load that the bone is really supporting. Real information related to specific patients, such as bone geometry and bone density distribution, can be used to determine patient loads. Therefore, we studied three different mathematical techniques: linear regression, artificial neural networks (ANN) and support vector machines (SVM). These techniques have been applied to a theoretical femur to obtain the load through the density that came from many bone remodeling simulations. Finally, the application of this novel methodology has been applied for the loading prediction of five real tibias. We are able to determine the subject-specific forces from CT data, from which we obtain bone geometry and density distribuviition of the five tibias. Then, the density values at certain bone regions have been introduced in the methodology developed that combines bone remodeling models and artificial neuronal networks (ANN) for obtaining the predicted subject-specific loads. Finally, in order to validate this novel technique for tibia loading predictions, we compare predicted loads with the loads obtained from the patientspecific musculoskeletal model. The errors between both loads were lower tan 6%. Therefore, the methodology proposed has been validate

Geometric data understanding : deriving case specific features

There exists a tradition using precise geometric modeling, where uncertainties in data can be considered noise. Another tradition relies on statistical nature of vast quantity of data, where geometric regularity is intrinsic to data and statistical models usually grasp this level only indirectly. This work focuses on point cloud data of natural resources and the silhouette recognition from video input as two real world examples of problems having geometric content which is intangible at the raw data presentation. This content could be discovered and modeled to some degree by such machine learning (ML) approaches like deep learning, but either a direct coverage of geometry in samples or addition of special geometry invariant layer is necessary. Geometric content is central when there is a need for direct observations of spatial variables, or one needs to gain a mapping to a geometrically consistent data representation, where e.g. outliers or noise can be easily discerned. In this thesis we consider transformation of original input data to a geometric feature space in two example problems. The first example is curvature of surfaces, which has met renewed interest since the introduction of ubiquitous point cloud data and the maturation of the discrete differential geometry. Curvature spectra can characterize a spatial sample rather well, and provide useful features for ML purposes. The second example involves projective methods used to video stereo-signal analysis in swimming analytics. The aim is to find meaningful local geometric representations for feature generation, which also facilitate additional analysis based on geometric understanding of the model. The features are associated directly to some geometric quantity, and this makes it easier to express the geometric constraints in a natural way, as shown in the thesis. Also, the visualization and further feature generation is much easier. Third, the approach provides sound baseline methods to more traditional ML approaches, e.g. neural network methods. Fourth, most of the ML methods can utilize the geometric features presented in this work as additional features.Geometriassa käytetään perinteisesti tarkkoja malleja, jolloin datassa esiintyvät epätarkkuudet edustavat melua. Toisessa perinteessä nojataan suuren datamäärän tilastolliseen luonteeseen, jolloin geometrinen säännönmukaisuus on datan sisäsyntyinen ominaisuus, joka hahmotetaan tilastollisilla malleilla ainoastaan epäsuorasti. Tämä työ keskittyy kahteen esimerkkiin: luonnonvaroja kuvaaviin pistepilviin ja videohahmontunnistukseen. Nämä ovat todellisia ongelmia, joissa geometrinen sisältö on tavoittamattomissa raakadatan tasolla. Tämä sisältö voitaisiin jossain määrin löytää ja mallintaa koneoppimisen keinoin, esim. syväoppimisen avulla, mutta joko geometria pitää kattaa suoraan näytteistämällä tai tarvitaan neuronien lisäkerros geometrisia invariansseja varten. Geometrinen sisältö on keskeinen, kun tarvitaan suoraa avaruudellisten suureiden havainnointia, tai kun tarvitaan kuvaus geometrisesti yhtenäiseen dataesitykseen, jossa poikkeavat näytteet tai melu voidaan helposti erottaa. Tässä työssä tarkastellaan datan muuntamista geometriseen piirreavaruuteen kahden esimerkkiohjelman suhteen. Ensimmäinen esimerkki on pintakaarevuus, joka on uudelleen virinneen kiinnostuksen kohde kaikkialle saatavissa olevan datan ja diskreetin geometrian kypsymisen takia. Kaarevuusspektrit voivat luonnehtia avaruudellista kohdetta melko hyvin ja tarjota koneoppimisessa hyödyllisiä piirteitä. Toinen esimerkki koskee projektiivisia menetelmiä käytettäessä stereovideosignaalia uinnin analytiikkaan. Tavoite on löytää merkityksellisiä paikallisen geometrian esityksiä, jotka samalla mahdollistavat muun geometrian ymmärrykseen perustuvan analyysin. Piirteet liittyvät suoraan johonkin geometriseen suureeseen, ja tämä helpottaa luonnollisella tavalla geometristen rajoitteiden käsittelyä, kuten väitöstyössä osoitetaan. Myös visualisointi ja lisäpiirteiden luonti muuttuu helpommaksi. Kolmanneksi, lähestymistapa suo selkeän vertailumenetelmän perinteisemmille koneoppimisen lähestymistavoille, esim. hermoverkkomenetelmille. Neljänneksi, useimmat koneoppimismenetelmät voivat hyödyntää tässä työssä esitettyjä geometrisia piirteitä lisäämällä ne muiden piirteiden joukkoon

New Methods to Improve Large-Scale Microscopy Image Analysis with Prior Knowledge and Uncertainty

Multidimensional imaging techniques provide powerful ways to examine various kinds of scientific questions. The routinely produced data sets in the terabyte-range, however, can hardly be analyzed manually and require an extensive use of automated image analysis. The present work introduces a new concept for the estimation and propagation of uncertainty involved in image analysis operators and new segmentation algorithms that are suitable for terabyte-scale analyses of 3D+t microscopy images

Face Image and Video Analysis in Biometrics and Health Applications

Computer Vision (CV) enables computers and systems to derive meaningful information from acquired visual inputs, such as images and videos, and make decisions based on the extracted information. Its goal is to acquire, process, analyze, and understand the information by developing a theoretical and algorithmic model. Biometrics are distinctive and measurable human characteristics used to label or describe individuals by combining computer vision with knowledge of human physiology (e.g., face, iris, fingerprint) and behavior (e.g., gait, gaze, voice). Face is one of the most informative biometric traits. Many studies have investigated the human face from the perspectives of various different disciplines, ranging from computer vision, deep learning, to neuroscience and biometrics. In this work, we analyze the face characteristics from digital images and videos in the areas of morphing attack and defense, and autism diagnosis. For face morphing attacks generation, we proposed a transformer based generative adversarial network to generate more visually realistic morphing attacks by combining different losses, such as face matching distance, facial landmark based loss, perceptual loss and pixel-wise mean square error. In face morphing attack detection study, we designed a fusion-based few-shot learning (FSL) method to learn discriminative features from face images for few-shot morphing attack detection (FS-MAD), and extend the current binary detection into multiclass classification, namely, few-shot morphing attack fingerprinting (FS-MAF). In the autism diagnosis study, we developed a discriminative few shot learning method to analyze hour-long video data and explored the fusion of facial dynamics for facial trait classification of autism spectrum disorder (ASD) in three severity levels. The results show outstanding performance of the proposed fusion-based few-shot framework on the dataset. Besides, we further explored the possibility of performing face micro- expression spotting and feature analysis on autism video data to classify ASD and control groups. The results indicate the effectiveness of subtle facial expression changes on autism diagnosis

Injury and Skeletal Biomechanics

This book covers many aspects of Injury and Skeletal Biomechanics. As the title represents, the aspects of force, motion, kinetics, kinematics, deformation, stress and strain are examined in a range of topics such as human muscles and skeleton, gait, injury and risk assessment under given situations. Topics range from image processing to articular cartilage biomechanical behavior, gait behavior under different scenarios, and training, to musculoskeletal and injury biomechanics modeling and risk assessment to motion preservation. This book, together with "Human Musculoskeletal Biomechanics", is available for free download to students and instructors who may find it suitable to develop new graduate level courses and undergraduate teaching in biomechanics