Evaluating Engagement in Digital Narratives from Facial Data

Engagement researchers indicate that the engagement level of people in a narrative has an influence on people's subsequent story-related attitudes and beliefs, which helps psychologists understand people's social behaviours and personal experience. With the arrival of multimedia, the digital narrative combines multimedia features (e.g. varying images, music and voiceover) with traditional storytelling. Research on digital narratives has been widely used in helping students gain problem-solving and presentation skills as well as supporting child psychologists investigating children's social understanding such as family/peer relationships through completing their digital narratives. However, there is little study on the effect of multimedia features in digital narratives on the engagement level of people. This research focuses on measuring the levels of engagement of people in digital narratives and specifically on understanding the media effect of digital narratives on people's engagement levels. Measurement tools are developed and validated through analyses of facial data from different age groups (children and young adults) in watching stories with different media features of digital narratives. Data sources used in this research include a questionnaire with Smileyometer scale and the observation of each participant's facial behaviours

DeepFN: Towards Generalizable Facial Action Unit Recognition with Deep Face Normalization

Facial action unit recognition has many applications from market research to psychotherapy and from image captioning to entertainment. Despite its recent progress, deployment of these models has been impeded due to their limited generalization to unseen people and demographics. This work conducts an in-depth analysis of performance across several dimensions: individuals(40 subjects), genders (male and female), skin types (darker and lighter), and databases (BP4D and DISFA). To help suppress the variance in data, we use the notion of self-supervised denoising autoencoders to design a method for deep face normalization(DeepFN) that transfers facial expressions of different people onto a common facial template which is then used to train and evaluate facial action recognition models. We show that person-independent models yield significantly lower performance (55% average F1 and accuracy across 40 subjects) than person-dependent models (60.3%), leading to a generalization gap of 5.3%. However, normalizing the data with the newly introduced DeepFN significantly increased the performance of person-independent models (59.6%), effectively reducing the gap. Similarly, we observed generalization gaps when considering gender (2.4%), skin type (5.3%), and dataset (9.4%), which were significantly reduced with the use of DeepFN. These findings represent an important step towards the creation of more generalizable facial action unit recognition systems

BioInsights: Extracting personal data from "Still" wearable motion sensors

During recent years a large variety of wearable devices have become commercially available. As these devices are in close contact with the body, they have the potential to capture sensitive and unexpected personal data even when the wearer is not moving. This work demonstrates that wearable motion sensors such as accelerometers and gyroscopes embedded in head-mounted and wrist-worn wearable devices can be used to identify the wearer (among 12 participants) and his/her body posture (among 3 positions) from only 10 seconds of “still” motion data. Instead of focusing on large and apparent motions such as steps or gait, the proposed methods amplify and analyze very subtle body motions associated with the beating of the heart. Our findings have the potential to increase the value of pervasive wearable motion sensors but also raise important privacy concerns that need to be considered.National Science Foundation (U.S.). (CCF-1029585

Changing the Channel - From Face to Face to Digital Space: Framing the Foundations of Video Based Presentation & Meeting Channels

Effective presentation skills never go out of style, however, the channel by which we deliver presentations has been rapidly changing over the past two decades. Technological developments have made it easier to bring audiences together in virtual spaces and as a result, more and more presentations are taking place every day through digital channels. The cornerstones of effective and engaging presentations have remained the same for hundreds of years, but digital presentation and meeting channels bring both new challenges and opportunities that need to be examined in order to ensure we as a field are applying and teaching the best practices for this new channel. While some face-to-face presentation skills and best practices carry over to the digital world, there are new and unique practices that must be considered when attempting to engage digital audiences. The primary aim of this manuscript is to provide presenters and facilitators an overview of the unique opportunities and challenges that digital channels present along with details on the best practices and approaches for engaging digital audiences in an effective manner. An examination of future challenges for training and coaching presenters within these digital channels is also discussed

Abordando la medición automática de la experiencia de la audiencia en línea

Trabajo de Fin de Grado del Doble Grado en Ingeniería Informática y Matemáticas, Facultad de Informática UCM, Departamento de Ingeniería del Software e Inteligencia Artificial, Curso 2020/2021The availability of automatic and personalized feedback is a large advantage when facing an audience. An effective way to give such feedback is to analyze the audience experience, which provides valuable information about the quality of a speech or performance. In this document, we present the design and implementation of a computer vision system to automatically measure audience experience. This includes the definition of a theoretical and practical framework grounded on the theatrical perspective to quantify this concept, the development of an artificial intelligence system which serves as a proof-of-concept of our approach, and the creation of a dataset to train our system. To facilitate the data collection step, we have also created a custom video conferencing tool. Additionally, we present the evaluation of our artificial intelligence system and the final conclusions.La disponibilidad de feedback automático y personalizado supone una gran ventaja a la hora de enfrentarse a un público. Una forma efectiva de dar este tipo de feedback es analizar la experiencia de la audiencia, que proporciona información fundamental sobre la calidad de una ponencia o actuación. En este documento exponemos el diseño e implementación de un sistema automático de medición de la experiencia de la audiencia basado en la visión por computador. Esto incluye la definición de un marco teórico y práctico fundamentado en la perspectiva del mundo del teatro para cuantificar el concepto de experiencia de la audiencia, el desarrollo de un sistema basado en inteligencia artificial que sirve como prototipo de nuestra aproximación y la recopilación un conjunto de datos para entrenar el sistema. Para facilitar este último paso hemos desarrolado una aplicación de videoconferencias personalizada. Además, en este trabajo presentamos la evaluación de nuestro sistema de inteligencia artificial y las conclusiones extraídas.Depto. de Ingeniería de Software e Inteligencia Artificial (ISIA)Fac. de InformáticaTRUEunpu

Towards a Video Consumer Leaning Spectrum: A Medium-Centric Approach

Purpose: As TV and digital video converge, there is a need to compare advertising effectiveness, advertising receptivity, and video consumption drivers in this new context. Considering the emerging viewing practices and underlying theories, this study examines the feasibility of the traditional notion of differentiating between lean-back (LB) and lean-forward (LF) media, and proposes a revised approach of addressing video consumption processes and associated advertising effectiveness implications. Methodology: An extensive, systematic literature review examines a total of 715 sources regarding current lean-back/lean-forward media research and alternative approaches as by (1) basic terminologies, (2) limitations of lean-back/lean-forward situations, (3) advertising effectiveness implications, (4) video-specific approaches. Findings/Contribution: Key differences between lean-back and lean-forward video consumption are presented. A conceptual integration of video ad receptivity/effectiveness drivers is proposed to guide future media and marketing research and practice. Video consumption today is no longer lean-back or lean-forward, but a “leaning spectrum” with two dimensions: leaning direction and leaning degree. Designing video content today requires focusing on consumption drivers and platform synergies for owning the “leaning spectrum”

Type-2 Fuzzy Logic based Systems for Adaptive Learning and Teaching within Intelligent E-Learning Environments

The recent years have witnessed an increased interest in e-learning platforms that incorporate adaptive learning and teaching systems that enable the creation of adaptive learning environments to suit individual student needs. The efficiency of these adaptive educational systems relies on the methodology used to accurately gather and examine information pertaining to the characteristics and needs of students and relies on the way that information is processed to form an adaptive learning context. The vast majority of existing adaptive educational systems do not learn from the users’ behaviours to create white-box models to handle the high level of uncertainty and that could be easily read and analysed by the lay user. The data generated from interactions, such as teacher–learner or learner–system interactions within asynchronous environments, provide great opportunities to realise more adaptive and intelligent e-learning platforms rather than propose prescribed pedagogy that depends on the idea of a few designers and experts. Another limitation of current adaptive educational systems is that most of the existing systems ignore gauging the students' engagements levels and mapping them to suitable delivery needs which match the students' knowledge and preferred learning styles. It is necessary to estimate the degree of students’ engagement with the course contents. Such feedback is highly important and useful for assessing the teaching quality and adjusting the teaching delivery in small and large-scale online learning platforms. Furthermore, most of the current adaptive educational systems are used within asynchronous e-learning contexts as self-paced e-learning products in which learners can study in their own time and at their own speed, totally ignorant of synchronous e-learning settings of teacher-led delivery of the learning material over a communication tool in real time. This thesis presents novel theoretical and practical architectures based on computationally lightweight T2FLSs for lifelong learning and adaptation of learners’ and teachers’ behaviours in small- and large-scale asynchronous and synchronous e-learning platforms. In small-scale asynchronous and synchronous e-learning platforms, the presented architecture augments an engagement estimate system using a noncontact, low-cost, and multiuser support 3D sensor Kinect (v2). This is able to capture reliable features including head pose direction and hybrid features of facial expression to enable convenient and robust estimation of engagement in small-scale online and onsite learning in an unconstrained and natural environment in which users are allowed to act freely and move without restrictions. We will present unique real-world experiments in large and small-scale e-learning platforms carried out by 1,916 users from King Abdul-Aziz and Essex universities in Saudi Arabia and the UK over the course of teaching Excel and PowerPoint in which the type 2 system is learnt and adapted to student and teacher behaviour. The type-2 fuzzy system will be subjected to extended and varied knowledge, engagement, needs, and a high level of uncertainty variation in e-learning environments outperforming the type 1 fuzzy system and non-adaptive version of the system by producing better performance in terms of improved learning, completion rates, and better user engagements

Media of things : supporting the production and consumption of object-based media with the internet of things

Ph. D. Thesis.Visual media consumption habits are in a constant state of flux, predicting which platforms and consumption mediums will succeed and which will fail is a fateful business. Virtual Reality and Augmented Reality could be the 3D TVs that went before them, or they could push forward a new level of content immersion and radically change media production forever. Content producers are constantly trying to adapt to these shifts in habits and respond to new technologies. Smaller independent studios buoyed by their new-found audience penetration through sites like YouTube and Facebook can inherently respond to these emerging technologies faster, not weighed down by the “legacy” many. Broadcasters such as the BBC are keen to evolve their content to respond to the challenges of this new world. Producing content that is both more compelling in terms of immersion, and more responsive to technological advances in terms of input and output mediums. This is where the concept of Object-based Broadcasting was born, content that is responsive to the user consuming their content on a phone over a short period of time whilst also providing an immersive multi-screen experience for a smart home environment. One of the primary barriers to the development of Object-based Media is in a feasible set of mechanisms to generate supporting assets and adequately exploit the input and output mediums of the modern home. The underlying question here is how we build these experiences, we obviously can’t produce content for each of the thousands of combinations of devices and hardware we have available to us. I view this challenge to content makers as one of a distinct lack of descriptive and abstract detail at both ends of the production pipeline. In investigating the contribution that the Internet of Things may have to this space I first look to create well described assets in productions using embedded sensing. Detecting non-visual actions and generating detail not possible from vision alone. I then look to exploit existing datasets from production and consumption environments to gain greater understanding of generated media assets and a means to coordinate input/output in the home. Finally, I investigate the opportunities for rich and expressive interaction with devices and content in the home exploiting favourable characteristics of existing interfaces to construct a compelling control interface to Smart Home devices and Object-based experiences. I resolve that the Internet of Things is vital to the development of Object-based Broadcasting and its wider roll-out.British Broadcasting Corporatio

Evaluating child engagement in digital story stems using facial data

Engagement is a key factor in understanding people’s psychology and behaviours and is an understudied topic in children. The area of focus in this thesis is child engagement in the story-stems used in child Attachment evaluations such as the Manchester Child Attachment Task (MCAST). Due to the high cost and time required for conducting Attachment assessments, automated assessments are being developed. These present story-stems in a cost-effective way on a laptop screen to digitalise the interaction between the child and the story, without disrupting the storytelling. However, providing such tests via computer relies on the child being engaged in the digital story-stem. If they are not engaged, then the tests will not be successful and the collected data will be of poor-quality, which will not allow for successful detection of Attachment status. Therefore, the aim of this research is to investigate a range of aspects of child engagement to understand how to engage children in story-stems, and how to measure their engagement levels. This thesis focuses on measuring the levels of child engagement in digital story-stems and specifically on understanding the effect of multimedia digital story-stems on children’s engagement levels to create a better and more engaging digital story-stem. Data sources used in this thesis include the observation of each child’s facial behaviours and a questionnaire with Smiley-o-meter scale. Measurement tools are developed and validated through analyses of facial data from children when watching digital story-stems with different presentation and voice types. Results showed that facial data analysis, using eye-tracking measures and facial action units (AUs) recognition, can be used to measure children’s engagement levels in the context of viewing digital story-stems. Using eye-tracking measures, engaged children have longer fixation durations in both mean and sum of fixation durations, which reflect that a child was deeply engaged in the story-stems. Facial AU recognition had better performance in a binary classification for discriminating engaged or disengaged children than eye-tracking measurements. The most frequently occurring facial action units taken from the engaged classes show that children’s facial action units indicated signs of fear, which suggest that children felt anxiety and distress while watching the story-stems. These feeling of anxiety and distress show that children have a strong emotional engagement and can locate themselves in the story-stems, showing that they were strong engaged. A further contribution in this thesis was to investigate the best way of creating an engaging story-stem. Results showed that an animated video narrated by a female expressive voice was most engaging. Compared to the live-action MCAST video, data showed that children were more engaged in the animated videos. Voice gender and voice expressiveness were two factors of the quality of storytelling voice that were evaluated and both affected children’s engagement levels. The distribution of child engagement across different voice types was compared to find the best storytelling voice type for story-stem design. A female expressive voice had a better performance for displaying the ‘distress’ in the story-stem than other voice types and engaged children more in the story-stems. The quality of the storytelling voice used to narrate story-stems and animated videos both significantly affected children’s levels of engagement. Such digital story-stems make children more engaged in the digital MCAST test