Gesture and Speech in Interaction - 4th edition (GESPIN 4)

International audienceThe fourth edition of Gesture and Speech in Interaction (GESPIN) was held in Nantes, France. With more than 40 papers, these proceedings show just what a flourishing field of enquiry gesture studies continues to be. The keynote speeches of the conference addressed three different aspects of multimodal interaction:gesture and grammar, gesture acquisition, and gesture and social interaction. In a talk entitled Qualitiesof event construal in speech and gesture: Aspect and tense, Alan Cienki presented an ongoing researchproject on narratives in French, German and Russian, a project that focuses especially on the verbal andgestural expression of grammatical tense and aspect in narratives in the three languages. Jean-MarcColletta's talk, entitled Gesture and Language Development: towards a unified theoretical framework,described the joint acquisition and development of speech and early conventional and representationalgestures. In Grammar, deixis, and multimodality between code-manifestation and code-integration or whyKendon's Continuum should be transformed into a gestural circle, Ellen Fricke proposed a revisitedgrammar of noun phrases that integrates gestures as part of the semiotic and typological codes of individuallanguages. From a pragmatic and cognitive perspective, Judith Holler explored the use ofgaze and hand gestures as means of organizing turns at talk as well as establishing common ground in apresentation entitled On the pragmatics of multi-modal face-to-face communication: Gesture, speech andgaze in the coordination of mental states and social interaction.Among the talks and posters presented at the conference, the vast majority of topics related, quitenaturally, to gesture and speech in interaction - understood both in terms of mapping of units in differentsemiotic modes and of the use of gesture and speech in social interaction. Several presentations explored the effects of impairments(such as diseases or the natural ageing process) on gesture and speech. The communicative relevance ofgesture and speech and audience-design in natural interactions, as well as in more controlled settings liketelevision debates and reports, was another topic addressed during the conference. Some participantsalso presented research on first and second language learning, while others discussed the relationshipbetween gesture and intonation. While most participants presented research on gesture and speech froman observer's perspective, be it in semiotics or pragmatics, some nevertheless focused on another importantaspect: the cognitive processes involved in language production and perception. Last but not least,participants also presented talks and posters on the computational analysis of gestures, whether involvingexternal devices (e.g. mocap, kinect) or concerning the use of specially-designed computer software forthe post-treatment of gestural data. Importantly, new links were made between semiotics and mocap data

Sticks and Stones May Break My Bones but Words Will Never Hurt Me...Until I See Them: A Qualitative Content Analysis of Trolls in Relation to the Gricean Maxims and (IM)Polite Virtual Speech Acts

The troll is one of the most obtrusive and disruptive bad actors on the internet. Unlike other bad actors, the troll interacts on a more personal and intimate level with other internet users. Social media platforms, online communities, comment boards, and chatroom forums provide them with this opportunity. What distinguishes these social provocateurs from other bad actors are their virtual speech acts and online behaviors. These acts aim to incite anger, shame, or frustration in others through the weaponization of words, phrases, and other rhetoric. Online trolls come in all forms and use various speech tactics to insult and demean their target audiences. The goal of this research is to investigate trolls\u27 virtual speech acts and the impact of troll-like behaviors on online communities. Using Gricean maxims and politeness theory, this study seeks to identify common vernacular, word usage, and other language behaviors that trolls use to divert the conversation, insult others, and possibly affect fellow internet users’ mental health and well-being

Lapses in Responsiveness: Characteristics and Detection from the EEG

Performance lapses in occupations where public safety is paramount can have disastrous consequences, resulting in accidents with multiple fatalities. Drowsy individuals performing an active task, like driving, often cycle rapidly between periods of wake and sleep, as exhibited by cyclical variation in both EEG power spectra and task performance measures. The aim of this project was to identify reliable physiological cues indicative of lapses, related to behavioural microsleep episodes, from the EEG, which could in turn be used to develop a real-time lapse detection (or better still, prediction) system. Additionally, the project also sought to achieve an increased understanding of the characteristics of lapses in responsiveness in normal subjects. A study was conducted to determine EEG and/or EOG cues (if any) that expert raters use to detect lapses that occur during a psychomotor vigilance task (PVT), with the subsequent goal of using these cues to design an automated system. A previously-collected dataset comprising physiological and performance data of 10 air traffic controllers (ATCs) was used. Analysis showed that the experts were unable to detect the vast majority of lapses based on EEG and EOG cues. This suggested that, unlike automated sleep staging, an automated lapse detection system needed to identify features not generally visible in the EEG. Limitations in the ATC dataset led to a study where more comprehensive physiological and performance data were collected from normal subjects. Fifteen non-sleep-deprived male volunteers aged 18-36 years were recruited. All performed a 1-D continuous pursuit visuomotor tracking task for 1 hour during each of two sessions that occurred between 1 and 7 weeks apart. A video camera was used to record head and facial expressions of the subject. EEG was recorded from electrodes at 16 scalp locations according to the 10-20 system at 256 Hz. Vertical and horizontal EOG was also recorded. All experimental sessions were held between 12:30 and 17:00 hours. Subjects were asked to refrain from consuming stimulants or depressants, for 4 h prior to each session. Rate and duration were estimated for lapses identified by a tracking flat spot and/or video sleep. Fourteen of the 15 subjects had one or more lapses, with an overall rate of 39.3 ± 12.9 lapses per hour (mean ± SE) and a lapse duration of 3.4 ± 0.5 s. The study also showed that lapsing and tracking error increased during the first 30 or so min of a 1-h session, then decreased during the remaining time, despite the absence of external temporal cues. EEG spectral power was found to be higher during lapses in the delta, theta, and alpha bands, and lower in the beta, gamma, and higher bands, but correlations between changes in EEG power and lapses were low. Thus, complete lapses in responsiveness are a frequent phenomenon in normal subjects - even when not sleep-deprived - undertaking an extended, monotonous, continuous visuomotor task. This is the first study to investigate and report on the characteristics of complete lapses of responsiveness during a continuous tracking task in non-sleep-deprived subjects. The extent to which non-sleep-deprived subjects experience complete lapses in responsiveness during normal working hours was unexpected. Such findings will be of major concern to individuals and companies in various transport sectors. Models based on EEG power spectral features, such as power in the traditional bands and ratios between bands, were developed to detect the change of brain state during behavioural microsleeps. Several other techniques including spectral coherence and asymmetry, fractal dimension, approximate entropy, and Lempel-Ziv (LZ) complexity were also used to form detection models. Following the removal of eye blink artifacts from the EEG, the signal was transformed into z-scores relative to the baseline of the signal. An epoch length of 2 s and an overlap of 1 s (50%) between successive epochs were used for all signal processing algorithms. Principal component analysis was used to reduce redundancy in the features extracted from the 16 EEG derivations. Linear discriminant analysis was used to form individual classification models capable of detecting lapses using data from each subject. The overall detection model was formed by combining the outputs of the individual models using stacked generalization with constrained least-squares fitting used to determine the optimal meta-learner weights of the stacked system. The performance of the lapse detector was measured both in terms of its ability to detect lapse state (in 1-s epochs) and lapse events. Best performance in lapse state detection was achieved using the detector based on spectral power (SP) features (mean correlation of φ = 0.39 ± 0.06). Lapse event detection performance using SP features was moderate at best (sensitivity = 73.5%, selectivity = 25.5%). LZ complexity feature-based detector showed the highest performance (φ = 0.28 ± 0.06) out of the 3 non-linear feature-based detectors. The SP+LZ feature-based model had no improvement in performance over the detector based on SP alone, suggesting that LZ features contributed no additional information. Alpha power contributed the most to the overall SP-based detection model. Analysis showed that the lapse detection model was detecting phasic, rather than tonic, changes in the level of drowsiness. The performance of these EEG-based lapse detection systems is modest. Further research is needed to develop more sensitive methods to extract cues from the EEG leading to devices capable of detecting and/or predicting lapses

Towards Entity Status

Discourse entities are an important construct in computational linguistics. They introduce an additional level of representation between referring expressions and that which they refer to: the level of mental representation. In this thesis, I first explore some semiotic and communication theoretic aspects of discourse entities. Then, I develop the concept of "entity status". Entity status is a meta-variable that collects two dimensions formations about the role that an entity plays a discourse, and management informations about how the entity is created, accessed, and updated. Finally, the concept is applied to two case studies: the first one focusses on the choice of referring expressions in radio news, while the second looks at the conditions under which a discourse entity can be mentioned as a pronoun.Diskursentitäten sind ein wichtiger Konstrukt in der Computerlinguistik. Sie führen eine zusätzliche Repräsentationsebene ein zwischen referierenden Ausdrücken, und dem, auf das diese Ausdrücke referieren: die Ebene der mentalen Repräsentation. In dieser Dissertation erkunde ich zunächst einige semiotische und kommunikationstheoretische Aspekte von Diskursentitäten. Danach führe ich den Begriff des "Entitätenstatus" ein. Entitätenstatus ist eine Meta-Variable, die zwei Dimensionen von Information über eine Diskursentität vereinigt: Struktur-Informationen über die Rolle, die eine Entität im Diskurs spielt, und Verwaltungs-Informationen über Erstellung, Zugriff und Update. Dieser Begriff wird schlussendlich auf zwei Fallstudien angewendet: die erste Studie konzentriert sich auf die Wahl referierender Ausdrücke in Radionachrichten, während die zweite Studie die Bedingungen untersucht, in denen eine Diskursentität als Pronomen erwähnt werden kann

Can I Get a Witness?—Living While Black Death is Trending

It is not uncommon for graphic scenes of violence and death to infiltrate our timelinesfrom retweets, reposts, and shares. I often question how much control do we really have over the images that enter our feed? In what ways are we affected and influenced by these images? How do we relate to these images and video clips that are played and replayed before us? In what ways are these images evoking or are related to past scenes of racist violence? In what ways are these racially violent moments captured in photos and videos and shared online speaking to a Black consciousness? This project comparatively researches and examines the relationship between past modes and methods of Black trauma curation in the past, to contemporary modes of dissemination on social media in order to argue that contemporary uses of spaces such as Instagram, Facebook, and Twitter serve as an extension of previous scrapbooking methods. By comparing The Emmet Till Generation and their curation of trauma via scrapbooks which were used to galvanize social movements, and impact organizing efforts of the youth, The Trayvon Generation today uses social media in a similar fashion; to bear witness, to organize, and to curate digital memorials for the dead. Witnessing is further extended and complicated on digital platforms, providing an abundance of visual evidence that has proven to be vital in leading tp prosecutions and arrests of violent state officials, and perpetrators of extrajudicial violence. These live or recorded moments of witnessing are used not only as evidence, but to inform the public. However, we have always known that it’s always happening somewhere, even if we aren’t around to witness it. With that said, what are the effects of having the duty, and the responsibility to bear witness? Paying particular attention to Black youth, this project examines their presence and usage of social media spaces. By analyzing young Black people’s use of social media platforms in relation to Darnella Frazier’s strategic use Facebook, this project examines how Black youth and witnessing is currently driving a cultural shift in entertainment media that highlights witnessing death as a significant milestone for Black youth that marks the transition between childhood and adulthood. It is also impacting entertainment media that is not marketed towards Black people, further highlighting Black witnessing of racialized violence at the intersection of technology as both a contemporary and future issue through its inclusion in contemporary media.Witnessing for Black people is framed as being both necessary and traumatic. This project concludes with an in depth examination of speculative media to reveal the implications of both the present and the future intersections of race relations, state violence and technology. Through analyses of interviews, image circulation and dissemination, magazine articles, social media platforms, visual and speculative media, this dissertation works to address and attempts to answer the aforementioned questions

Cognition-aware systems to support information intake and learning

Knowledge is created at an ever-increasing pace putting us under constant pressure to consume and acquire new information. Information gain and learning, however, require time and mental resources. While the proliferation of ubiquitous computing devices, such as smartphones, enables us to consume information anytime and anywhere, technologies are often disruptive rather than sensitive to the current user context. While people exhibit different levels of concentration and cognitive capacity throughout the day, applications rarely take these performance variations into account and often overburden their users with information or fail to stimulate. This work investigates how technology can be used to help people effectively deal with information intake and learning tasks through cognitive context-awareness. By harvesting sensor and usage data from mobile devices, we obtain people's levels of attentiveness, receptiveness, and cognitive performance. We subsequently use this cognition-awareness in applications to help users process information more effectively. Through a series of lab studies, online surveys, and field experiments we follow six research questions to investigate how to build cognition-aware systems. Awareness of user's variations in levels of attention, receptiveness, and cognitive performance allows systems to trigger appropriate content suggestions, manage user interruptions, and adapt User Interfaces in real-time to match tasks to the user's cognitive capacities. The tools, insights, and concepts described in this book allow researchers and application designers to build systems with an awareness of momentary user states and general circadian rhythms of alertness and cognitive performance