Non-Intrusive Affective Assessment in the Circumplex Model from Pupil Diameter and Facial Expression Monitoring

Automatic methods for affective assessment seek to enable computer systems to recognize the affective state of their users. This dissertation proposes a system that uses non-intrusive measurements of the user’s pupil diameter and facial expression to characterize his /her affective state in the Circumplex Model of Affect. This affective characterization is achieved by estimating the affective arousal and valence of the user’s affective state. In the proposed system the pupil diameter signal is obtained from a desktop eye gaze tracker, while the face expression components, called Facial Animation Parameters (FAPs) are obtained from a Microsoft Kinect module, which also captures the face surface as a cloud of points. Both types of data are recorded 10 times per second. This dissertation implemented pre-processing methods and fixture extraction approaches that yield a reduced number of features representative of discrete 10-second recordings, to estimate the level of affective arousal and the type of affective valence experienced by the user in those intervals. The dissertation uses a machine learning approach, specifically Support Vector Machines (SVMs), to act as a model that will yield estimations of valence and arousal from the features derived from the data recorded. Pupil diameter and facial expression recordings were collected from 50 subjects who volunteered to participate in an FIU IRB-approved experiment to capture their reactions to the presentation of 70 pictures from the International Affective Picture System (IAPS) database, which have been used in large calibration studies and therefore have associated arousal and valence mean values. Additionally, each of the 50 volunteers in the data collection experiment provided their own subjective assessment of the levels of arousal and valence elicited in him / her by each picture. This process resulted in a set of face and pupil data records, along with the expected reaction levels of arousal and valence, i.e., the “labels”, for the data used to train and test the SVM classifiers. The trained SVM classifiers achieved 75% accuracy for valence estimation and 92% accuracy in arousal estimation, confirming the initial viability of non-intrusive affective assessment systems based on pupil diameter and face expression monitoring

Textual hijacking: strategies of resistance and reclaiming the objectified woman in Balzac, Baudelaire, and Degas

From the courtesan Esther in Honoré de Balzac’s Splendeurs et misères des courtisanes (1838-1847) to the femme sterile in Charles Baudelaire’s Les Fleurs du Mal (1857) to Edgar Degas’s nudes, women’s objectified bodies dominated artistic attention in nineteenth-century France. Appearance defined their roles, and tropes often replaced women in narratives centered on male desire. However, the women in these works resist erasure and challenge feminine passivity and marginalization. This dissertation explores their ambiguous female identities and their strategies of resistance. The tension in Balzac’s, Baudelaire’s, and Degas’s works between objectifying women and their textual importance emerges through the relationships among subject, object, and the abject self (as defined by Judith Butler) and among the narrator, the work, and sometimes the reader or viewer. The male gaze limits women’s identities within the subject-object-abject framework. In turn, these women exercise soft power to alter their status and identities. Joseph Nye defines soft power as attracting others and co-opting their power to achieve one’s goals. Through gender theory, I redefine these women, not only as objects of desire, but also as narrative subjects. In Balzac’s novel, Esther negotiates social dynamics to define her identity. She progresses from passive object to untenable abject self to literary subject. By using her body, creating documents, and crafting ritualized social encounters, Esther claims ownership of herself. In Les Fleurs du mal, Baudelaire often portrays women as a pretext for poetics. Yet, “La Chevelure,” “La Beauté,” “L’Homme et la mer,” and “Le Serpent qui danse,” display signs of feminine power. Baudelaire stages interactions between the poet-narrator and the sexualized woman and counteracts the subject-object binary through the gaze. Both the poet-narrator and representations of the feminine are necessary to advance the text. Degas’s nudes hinge upon voyeurism, objectification, and self-representation. Degas’s women are ambiguous, as shown in selected brothel monotypes, bather pastels, lithographs, and sculptures. Through Caroline Armstrong’s and Kathryn Brown’s readings of the monotypes, I demonstrate how these works challenge the male gaze and grant the female nude at least partial status as narrative subject. Tracing these works across media elucidates a female interiority that resists objectification

Sensory interactions in balance and eye movement control

Eye and head movements were recorded during angular/linear motion of the head or neck. Four areas of sensory-motor interaction in human balance control were studied. In the cervico-vestibular section, eye movements elicited by neck torsion were shown to be weak in normal subjects but considerably enhanced in labyrinthine defective patients, in whom they may partly compensate for the lack of the vestibulo-ocular reflex. In the vestibulo-cervical section, experiments showed a diminished ability of patients with absent vestibular function to stabilize the head in space during trunk motion. Other experiments found vestibular abnormalities in patients with cervical dystonia (Spasmodic Torticollis) which could not be explained by the abnormal head posture per se; it was concluded that the vestibular system contributes to human head posture and that the hitherto unexplained neural processes provoking Spasmodic Torticollis interfere with vestibular signals. Under the otolith-canal interaction section, experiments showed that slow phase eye movements of high velocity can be elicited in response to combined angular-linear acceleration, obtained by placing the head eccentrically in an ordinary "Barany" rotating chair. The possibility that the procedure could become a clinically useful test of otolith function was preliminary studied in oto-neurological patients. The section on otolith-visual interaction examines slow phase eye movements in response to lateral linear acceleration of the head. In the presence of visual fixation these responses are strong and compensate for head motion at very short latency, allowing the eyes to maintain fixation on stationary objects. In the dark responses are weak and inappropriate for visual stabilization. The experiments combining angular acceleration or visual stimulation with linear acceleration suggest that/ in order to generate functionally meaningful eye movements, otolith-ocular responses are highly dependent on interaction with other sensory stimuli. This thesis is supported by a series of published papers

Otolith function in human subjects: Perception of motion, reflex eye movements and vision during linear interaural acceleration

The thesis investigates how the otolith organs of the vestibular system, specifically the utricles, assist motion perception and aid visual stabilization, during translational lateral whole-body acceleration. It was found that high gradients of acceleration facilitate the detection of motion and that, for low acceleration gradients, motion perception in normal subjects relies on a 'velocity' threshold detection process. Experiments in patients without vestibular function indicated that, for the stimuli employed, the somatosensory system could be as sensitive to linear motion as the vestibular system. The interaction between the horizontal linear vestibulo-ocular reflex (LVOR) and visual context was characterized in the following experiments. Subjects were accelerated transiently in darkness, or while viewing earth-fixed or head-fixed targets. From onset, the eye velocity response to head translation was enhanced with acceleration level and target proximity, but was only slightly reduced by fixation of head-fixed targets. This suggested that the gain of the LVOR pathway was adjusted before or immediately after motion onset by a parameter depending mainly on viewing distance and less on the knowledge of probable relative target motion. For high relative target velocities, LVORs improved ocular fixation over what would be attained by pursuit alone, although fully compensatory eye movements were not always produced. The LVORs of patients who underwent unilateral vestibular deafferentation suggested that the utricular area generating transaural LVORs is the macular region lateral to the striola. Psychophysical experiments based on a reading task established the functional role of the LVOR for stabilising vision during high-frequency sinusoidal whole-body acceleration. Unlike normal subjects, visual acuity in patients without vestibular function was not better during self-motion than during display oscillation. Finally, the LVOR interaction with canal-ocular reflexes was studied using isolated and combined translational/rotational stimuli. The results showed that, shortly after motion onset, canal stimulation enhances the LVOR evoked by head translation

The effects of running, cycling, and duathlon exercise performance on cardiac function, haemodynamics and regulation

This thesis examined the effects of prolonged exercise, specifically Olympic Distance (OD)duathlon upon ultrasound derived indices of cardiac function, cardiac autonomic regulation measured via heart rate variability (HRV), and high-sensitivity cardiac troponin T (hs-cTnT)release. The primary aims were to (1) ascertain the influence of Olympic distance (OD) duathlon performance on cardiac function; (2) to investigate potential relationships between autonomic regulation, hs-cTnT release, and cardiac function, and (3) to investigate the effect of the individual legs of an OD duathlon on post-exercise cardiac function and to quantify the potential performance reserve of highly-trained endurance athletes when completing standalone legs of the duathlon. Findings from a systematic review and meta-analysis(Chapter 1) on research that performed serial echocardiographic and troponin measurements before and after exercise, intensity predicted changes in post-exercise cardiac troponin release and diastolic function. The findings agreed with previous meta-analyses using a more recent sample of studies; however, the recommendation for future studies to implement advanced cardiac imaging techniques, such as myocardial speckle tracking into their data collection would provide a more sensitive measure of post-exercise cardiac function. Whilst a large degree of heterogeneity in the results exists, this was in part explained by study exercise heart rate, participant age, and the prevalence of cardiac troponin release above the clinical detection threshold. The study performed in Chapter 3 was the first to investigate the effects of OD duathlon exercise on immediate and 24 hours post-exercise cardiac function. Additionally, a second OD duathlon was performed by participants with intra-leg measurements of cardiac function. In a highly trained cohort, there was evidence of transient post-exercise reductions in cardiac function and elevated serum high-sensitivity cardiac troponin T (hs-cTnT) above the clinical reference value, which was largely resolved within 24h of recovery. This study also demonstrated the reliability of lab-based duathlon exercise in a highly trained cohort and identified the pacing features of experienced multi-sport athletes that partially explained the different findings between the running and cycling legs of the duathlon. By investigating each leg of the duathlon individually (10k run, 5k run, 40k cycle), both at duathlon race-pace (DM) and maximal (Max) intensity on separate occasions, the performance reserve of the highly-trained cohort was quantified and further explored. The studies presented in Chapters 4 and 5 revealed that experienced duathletes were able to improve their speed across each leg by between 5-15% in a laboratory setting, compared to the duathlon effort. Additionally, the maximal effort 10k run leg provoked the most persistent changes to cardiac function that were present at 6h of recovery. Changes in cardiac function post DM 10k confirmed the findings of Chapter 3 that the greatest magnitude of cardiac perturbations occur following the initial 10k run leg. Aside from the Max 10k run and 40k cycle trials, all perturbations had resolved within 6h of recovery after each bout of exercise, highlighting the importance of recovery following maximal intensity efforts. The lack of 6h and 24h recovery data in Chapter 4, and Chapters 5 and 6, respectively is a shortcoming of these findings and therefore limits interpretation in the context of providing athletic guidance. Future research in this area should endeavour to include 6h and 24h recovery measures as standard, as multi-sport athletes typically perform multiple daily training sessions. The implications of substantial cardiac fatigue accumulation over many years of endurance training history are still unclear, and athletes may benefit from preventingits occurrence

Emotional persuasion in advertising – analyzing dialectal language, visual images and their interplay in TV commercials

Emotions are gaining ever more traction in marketing research and researchers now broadly recognize the benefits of emotional persuasion. Marketing scholars have become interested in emotions as an aspect of consumer behavior because they are important components of consumers’ responses in pre- and post-purchase buying behavior, in consumer satisfaction, and in shaping attitudes to products, services, and brands. The appeal to emotion is also a central topic of advertising research because the practice targets the consumers’ psychological, social, or symbolic needs to evoke an emotional response. This study investigates emotional persuasion in television commercials and provides insights into consumer persuasion from the respondent’s perspective. Advertising seeking to arouse emotions and interest is intended to make the audience process the message more thoroughly, create a vivid and enticing memory of the brand, and ultimately persuade the consumer to purchase the company’s products or services. The purpose of this study is to investigate emotional persuasion in advertising, more specifically how appeals to emotion are mediated in TV commercials. Television advertising is an important part of modern economies and paid media. Multimodal commercials can simultaneously transmit visual and audio stimuli, which makes them especially persuasive in shaping viewer’ emotions. However, there is a dearth of knowledge of how appeals to emotion are mediated through the interplay of language and moving visual components. This dissertation aims to fill this gap by exploring the emotional persuasion of the joint interplay of language in the Swiss-German dialect and moving images in television commercials. By analyzing such language and images this study provides three interconnected perspectives on emotional persuasion: dialectal language, visual moving images, and their interplay. Accordingly, this cross-disciplinary study touches on the theoretical fields of marketing, linguistics, and psychology. To date, research results have shown positive outcomes of the use of local dialects in the process of persuasion in advertising. However, this study is among the first to investigate how dialectal language can be used in advertising to appeal emotionally to a fragmented target audience. In addition, this thesis is among the first studies to focus on the filmic mediation of appeals to emotion, that is, the joint interplay of language in the Swiss-German dialect and moving images. The data for the empirical study consist of 32 television commercials in the spoken Swiss-German dialect placed by the Swiss cooperative Migros operating in the retail segment and specializing in fast-moving consumer goods. The research is based on a mixed-methods approach and the empirical aspect is conducted in two phases by analyzing commercials quantitatively and qualitatively. In the first phase, content analysis is used as a quantitative method to organize the stream of images and language. In the second phase, the qualitative analysis, the appeals to emotion of the language, images, and their interplay are investigated. The qualitative analysis of the data is divided into two stages: linguistic analysis and semiotic analysis. The linguistic analysis is conducted to study the emotional appeal of the language in the Swiss-German dialect. The semiotic analysis is conducted to uncover the emotional meanings of the images at the connotative level and the emotional meanings of the images in the interplay with the language. The outcome of the study is a framework of emotionally persuasive advertising in emotionally appealing dialectal language, emotionally appealing images, and the interplay of language in dialect and images. The framework can open new perspectives on understanding emotionally appealing advertising. From the managerial point of view, being able to appeal to customers on an emotional level can cut through the noise inherent in advertising, something that is becoming more difficult in today’s media environment filled with messages. Since consumers are exposed to numerous commercials, those that carry an emotional appeal can stand out from the crowd. As a practical implication, the framework is applicable to multimodal advertising in several media channels, including online advertising. The framework can help those designing advertising for fragmented target audiences and help marketers respond to the challenges of localization.-- Tunteisiin vetoava markkinointi on keskeinen aihe niin akateemisessa tutkimuksessa kuin käytännön markkinoinnissakin. Kuluttajien tunteiden on osoitettu olevan keskeisiä tekijöitä tuotteista, palveluista ja brändeistä muodostuvissa asenteissa. Tunteita on pyritty ymmärtämään myös osana ostokäyttäytymistä ja koettua asiakastyytyväisyyttä. Tutkimustulokset osoittavat, että tunteisiin vetoava mainonta on tehokas tapa puhutella katsojia. Vetoamalla kohdeyleisön psykologisiin, sosiaalisiin tai symbolisiin tarpeisiin katsojissa pyritään herättämään tunteita ja saamaan heidät ostamaan mainostettuja tuotteita. Tutkimusten mukaan tunteisiin vetoava mainonta herättää hyvin huomiota ja saa vastaanottajat käsittelemään mainosviestejä syvällisemmin. Näin brändeistä pystytään luomaan eläviä ja mieleenpainuvia muistikuvia. Lisäksi on todettu, että kohdeyleisön puhutteleminen heidän omalla murteellaan vetoaa vahvemmin tunteisiin. Käsillä olevan väitöskirjan tarkoitus on tutkia tunteisiin vetoavaa televisiomainontaa vastaanottajan näkökulmasta. Televisiomainonta on tärkeä osa taloutta ja maksettua mediaa. Televisiomainokset välittävät viestejä sekä näkö- että kuuloaistia hyödyntäen, mikä tehostaa mainosten vetoavuutta ja vaikuttavuutta. Vaikka monesta merkkijärjestelmästä koostuvaa multimodaalista mainontaa on tutkittu aikaisemminkin, aiempi tutkimus ei ole osoittanut, kuinka tunteisiin vetoavien mainosviestien kokonaismerkitys muodostuu sekä kielen että liikkuvien kuvien vuorovaikutuksessa. Käsillä oleva väitöskirja pyrkii täyttämään tämän tutkimusaukon tutkimalla tunteisiin vetoavan mainonnan kolmea toisiinsa kytköksissä olevaa näkökulmaa eli puhuttua kieltä, liikkuvia kuvia ja niiden vuorovaikutusta sveitsinsaksan murteella tuotetuissa televisiomainoksissa. Tämä poikkitieteellinen tutkimus onkin yksi ensimmäisistä, joissa selvitetään sekä puhutun kielen että liikkuvien kuvien yhteistoimintaa mainonnassa. Työn teoreettinen viitekehys kytkeytyy markkinointiin, kielitieteeseen ja psykologiaan. Paikallismurteiden ja mainonnan vaikuttavuuden välisestä yhteydestä on olemassa jonkin verran aiempaa tutkimusnäyttöä. Käsillä oleva väitöskirja on kuitenkin yksi ensimmäisistä tutkimuksista, joissa selvitetään, miten mainosten murteellisella kielellä pyritään puhuttelemaan fragmentoitunutta kohdeyleisöä tunnetasolla. Tutkielman empiirisen osan tutkimusaineisto koostuu 32 televisiomainoksesta, joissa puhutaan sveitsinsaksan murretta. Mainosten julkaisija on sveitsiläinen vähittäiskaupan alaan kuuluva osuuskunta Migros, joka on erikoistunut päivittäis- ja käyttötavaroihin. Tutkimusmetodologisesti työ edustaa monimenetelmätutkimusta, jossa yhdistetään sekä määrällisiä että laadullisia tutkimusmenetelmiä. Ensimmäisessä vaiheessa mainosten puhuttua murteellista kieltä ja liikkuvia kuvia tutkitaan määrällisen sisällönanalyysin avulla. Sisällönanalyysi selvittää kielen ja kuvien määrää mainoksissa. Toisessa vaiheessa tunteisiin vetoavaa murteellista kieltä, liikkuvia kuvia ja näiden vuorovaikutusta analysoidaan laadullisin menetelmin. Tutkimuksessa hyödynnettyjä laadullisia menetelmiä ovat lingvistinen ja semioottinen analyysi. Lingvistisen analyysin avulla selvitetään, miten murteellisella mainoskielellä pyritään vetoamaan katsojien tunteisiin. Semioottisessa analyysissä tutkitaan kuvien tunteisiin vetoavia konnotaatioita sekä tunteisiin vetoavan kielen että liikkuvien kuvien välistä vuorovaikutusta. Väitöskirjatutkimuksen tieteellinen kontribuutio esitetään tunteisiin vetoavan mainonnan mallina, johon tiivistyy murteellisen kielen, liikkuvien kuvien ja näiden vuorovaikutuksen keinot vedota kuluttajiin tunnetasolla. Näin ollen tuotetaan uutta tietoa markkinoinnin ja kielitieteen tutkimukseen. Mallista voi olla hyötyä käytännön markkinointityössä, sillä tunteisiin vetoava mainonta erottautuu paremmin kilpailevien mainosten täyttämästä mediaympäristöstä. Mallia voidaan soveltaa käytettäväksi eri viestintäkanavissa, esimerkiksi verkkomainonnassa. Lisäksi tutkimustulokset voivat auttaa markkinoijia kohdentamaan mainontaa paikallisille kohdeyleisöille ja vastaamaan lokalisoinnin tuomiin haasteisiin

Detecting social signals from the face

This thesis investigates our sensitivity to social signals from the face, both in health and disease, and explores some of the methodologies employed to measure them. The first set of experiments used forced choice and free naIll1ng paradigms to investigate the interpretation of a set of facial expressions by Western and Japanese participants. Performance in the forced choice task exceeded that measured in the free naming task for both cultures, but the Japanese participants were found to be particularly poor at labelling expressions of fear and disgust. The difficulties experienced with translation and interpretation in these tasks led to the development of a psychophysical paradigm which was used to measure the signalling strength of facial expressions without the need for participants to interpret what they saw. Psychophysical tasks were also used to measure sensitivity to eye gaze direction. A 'live' and screen-based task produced comparable thresholds and revealed that our sensitivity to these ocular signals was at least as good as Snellen acuity. Manipulations of the facial surround in the screen-based task revealed that the detection of gaze direction was facilitated by the presence of the facial surround and as such it can be assumed that gaze discriminations are likely to be made in conjunction with other face processing analyses. The tasks developed in these chapters were used to test two patients with bilateral amygdala damage. Patients with this brain injury have been reported to experience difficulties in the interpretation of facial and auditory signals of fear. In this thesis, their performance was found to depend on the task used to measure it. However, neither patient was found to be impaired in their ability to label fearful expressions compared to control participants. Instead, patient SE demonstrated a consistently poor performance in his ability to interpret expressions of disgust. Vll Experiments 2, 3, 4 and 5 of Chapter 3, have also been reported in Perception, 1995, Vol. 24, Supplement, pp. 14. The Face as a long distance transmitter. Jenkins, J., Craven, B. & Bruce, V. Experiments 1,2,3 and 4 of Chapter 3 were also reported in the Technical Report of the Institute of Electronics Information and Communication Engineers. HIP 96-39 (1997-03). Methods for detecting social signals from the face. Jenkins, J., Craven, B., Bruce, V., & Akamatsu, S. Experiments 2 and 5 of Chapter 3, and a selection of the patient studies from Chapter 6 were reported at the Experimental Psychology Society, Bristol meeting, 1996, and at the Applied Vision Association, Annual Meeting, April, 1996. Sensitivity to Expressive Signals from the Human Face: Psychophysical and Neuropsychological Investigations. Jenkins, J., Bruce, V., Calder, A., & Craven, B