A review of affective computing: From unimodal analysis to multimodal fusion

Affective computing is an emerging interdisciplinary research field bringing together researchers and practitioners from various fields, ranging from artificial intelligence, natural language processing, to cognitive and social sciences. With the proliferation of videos posted online (e.g., on YouTube, Facebook, Twitter) for product reviews, movie reviews, political views, and more, affective computing research has increasingly evolved from conventional unimodal analysis to more complex forms of multimodal analysis. This is the primary motivation behind our first of its kind, comprehensive literature review of the diverse field of affective computing. Furthermore, existing literature surveys lack a detailed discussion of state of the art in multimodal affect analysis frameworks, which this review aims to address. Multimodality is defined by the presence of more than one modality or channel, e.g., visual, audio, text, gestures, and eye gage. In this paper, we focus mainly on the use of audio, visual and text information for multimodal affect analysis, since around 90% of the relevant literature appears to cover these three modalities. Following an overview of different techniques for unimodal affect analysis, we outline existing methods for fusing information from different modalities. As part of this review, we carry out an extensive study of different categories of state-of-the-art fusion techniques, followed by a critical analysis of potential performance improvements with multimodal analysis compared to unimodal analysis. A comprehensive overview of these two complementary fields aims to form the building blocks for readers, to better understand this challenging and exciting research field

Lingüística cognitiva y su aplicación en la enseñanza de español/L2: hacia un aprendizaje más significativo de la expresión de la emoción

The present dissertation, within the field of Cognitive Linguistics applied to Spanish/L2 teaching, presents a collection of 13 published and under-review papers. Among the motivations that have guided this work is the lack of experimental research within the field of CL and Spanish/L2 instruction that presents empirical evidence of the benefits of bringing these two disciplines together. Based on a prior cognitive and contrastive analysis of frequent constructions (i.e., psych verbs, metaphorical expressions with ponerse and tocar, and ironic utterances ), a series of empirical studies are conducted with Spanish/L2 learners at different proficiency levels, from beginner to advanced. The linguistic constructions under study have several aspects in common. First, they are used to express emotions; second, their acquisition in an instructional setting has been considered a real challenge; and third, their inclusion in the curriculum has been heretofore rather neglected. A wide variety of corpora has been used for the analysis of the target constructions: from textbooks, which are the material to which learners are most directly exposed, to corpora from Sketch Engine, Twitter, and interviews with native speakers, among others. Based on findings and in search for further empirical validation, an innovative CL pedagogy has been designed and further implemented at different levels with a large number of students at a North American university. As a novelty, L2 learner performance has been evaluated via assessment tests that, in coherence with the theoretical approach adopted and in line with its cognitive-based pedagogical application, have been carefully designed. Overall, results from the empirical studies examining the effects of a CL-based methodology for both pedagogical material and assessment test design yield statistically positive effects for the cognitive group in comprehension and production tasks at each proficiency level. These promising findings reveal the productivity of this method, as the learning of the target forms scaffolds and, as a result, learners’ communicative, metaphorical, and ironic competences are enhanced. The inclusion of a broader range of psych-verbs at lower levels, of change -of-state and tactile constructions through metaphor awareness, and of verbal ironic cues in the Spanish/L2 curriculum along with their treatment from a CL perspective are advocated. Such an approach should be put into practice in the day-to-day L2 classroom experience and in empirical research looking at the effects of a CL pedagogy. The positive findings in this research highlight the importance of embracing a CL-inspired method for Spanish/L2 teaching and assessing. They also call for a methodological change in the type of assessment. Such transformation requires the learning of Spanish –a language at great expansion– to build from linguistic assumptions from which it is possible to operate. Furthermore, the empirical studies here reported contribute to the small but growing body of literature that researches L2s other than English