Proceedings

Proceedings of the 3rd Nordic Symposium on Multimodal Communication. Editors: Patrizia Paggio, Elisabeth Ahlsén, Jens Allwood, Kristiina Jokinen, Costanza Navarretta. NEALT Proceedings Series, Vol. 15 (2011), vi+87 pp. © 2011 The editors and contributors. Published by Northern European Association for Language Technology (NEALT) http://omilia.uio.no/nealt . Electronically published at Tartu University Library (Estonia) http://hdl.handle.net/10062/22532

The Utility Of Verbal Display Redundancy In Managing Pilot\u27s Cognitive Load During Controller-pilot Voice Communications

Miscommunication between controllers and pilots, potentially resulting from a high pilot cognitive load, has been a causal or contributing factor in a large number of aviation accidents. In this context, failure to communicate can be attributed, among other factors, to an inadequate human-system interface design, the related high cognitive load imposed on the pilot, and poor performance reflected by a higher error rate. To date, voice radio remains in service without any means for managing pilot cognitive load by design (as opposed to training or procedures). Such an oversight is what prompted this dissertation. The goals of this study were (a) to investigate the utility of a voice-to-text transcription (V-T-T) of ATC clearances in managing pilot\u27s cognitive load during controller-pilot communications within the context of a modern flight deck environment, and (b) to validate whether a model of variable relationships which is generated in the domain of learning and instruction would transfer , and to what extend, to an operational domain. First, within the theoretical framework built for this dissertation, all the pertaining factors were analyzed. Second, by using the process of synthesis, and based on guidelines generated from that theoretical framework, a redundant verbal display of ATC clearances (i.e., a V-T-T) was constructed. Third, the synthesized device was empirically examined. Thirty four pilots participated in the study – seventeen pilots with 100-250 total flight hours and seventeen with \u3e 500 total flight hours. All participants had flown within sixty days prior to attending the study. The experiment was conducted one pilot at a time in 2.5-hour blocks. A 2 Verbal Display Redundancy (no-redundancy and redundancy) X 2 Verbal Input Complexity (low and high) X 2 Level of Expertise (novices and experts) mixed-model design was used for the study with 5 IFR clearances in each Redundancy X Complexity condition. The results showed that the amounts of iii reduction of cognitive load and improvement of performance, when verbal display redundancy was provided, were in the range of about 20%. These results indicated that V-T-T is a device which has a tremendous potential to serve as (a) a pilot memory aid, (b) a way to verify a clearance has been captured correctly without having to make a Say again call, and (c) to ultimately improve the margin of safety by reducing the propensity for human error for the majority of pilot populations including those with English as a second language. Fourth, the results from the validation of theoretical models transfer showed that although cognitive load remained as a significant predictor of performance, both complexity and redundancy also had unique significant effects on performance. Furthermore, these results indicated that the relationship between these variables was not as clear-cut in the operational domain investigated here as the models from the domain of learning and instruction suggested. Until further research is conducted, (a) to investigate how changes in the operational task settings via adding additional coding (e.g., permanent record of clearances which can serve as both a memory aid and a way to verify a clearance is captured correctly) affect performance through mechanisms other than cognitive load; and (b) unless the theoretical models are modified to reflect how changes in the input variables impact the outcome in a variety of ways; a degree of prudence should be exercised when the results from the model transfer validation are applied to operational environments similar to the one investigated in this dissertation research

Gesture and Speech in Interaction - 4th edition (GESPIN 4)

International audienceThe fourth edition of Gesture and Speech in Interaction (GESPIN) was held in Nantes, France. With more than 40 papers, these proceedings show just what a flourishing field of enquiry gesture studies continues to be. The keynote speeches of the conference addressed three different aspects of multimodal interaction:gesture and grammar, gesture acquisition, and gesture and social interaction. In a talk entitled Qualitiesof event construal in speech and gesture: Aspect and tense, Alan Cienki presented an ongoing researchproject on narratives in French, German and Russian, a project that focuses especially on the verbal andgestural expression of grammatical tense and aspect in narratives in the three languages. Jean-MarcColletta's talk, entitled Gesture and Language Development: towards a unified theoretical framework,described the joint acquisition and development of speech and early conventional and representationalgestures. In Grammar, deixis, and multimodality between code-manifestation and code-integration or whyKendon's Continuum should be transformed into a gestural circle, Ellen Fricke proposed a revisitedgrammar of noun phrases that integrates gestures as part of the semiotic and typological codes of individuallanguages. From a pragmatic and cognitive perspective, Judith Holler explored the use ofgaze and hand gestures as means of organizing turns at talk as well as establishing common ground in apresentation entitled On the pragmatics of multi-modal face-to-face communication: Gesture, speech andgaze in the coordination of mental states and social interaction.Among the talks and posters presented at the conference, the vast majority of topics related, quitenaturally, to gesture and speech in interaction - understood both in terms of mapping of units in differentsemiotic modes and of the use of gesture and speech in social interaction. Several presentations explored the effects of impairments(such as diseases or the natural ageing process) on gesture and speech. The communicative relevance ofgesture and speech and audience-design in natural interactions, as well as in more controlled settings liketelevision debates and reports, was another topic addressed during the conference. Some participantsalso presented research on first and second language learning, while others discussed the relationshipbetween gesture and intonation. While most participants presented research on gesture and speech froman observer's perspective, be it in semiotics or pragmatics, some nevertheless focused on another importantaspect: the cognitive processes involved in language production and perception. Last but not least,participants also presented talks and posters on the computational analysis of gestures, whether involvingexternal devices (e.g. mocap, kinect) or concerning the use of specially-designed computer software forthe post-treatment of gestural data. Importantly, new links were made between semiotics and mocap data

Negative vaccine voices in Swedish social media

Vaccinations are one of the most significant interventions to public health, but vaccine hesitancy creates concerns for a portion of the population in many countries, including Sweden. Since discussions on vaccine hesitancy are often taken on social networking sites, data from Swedish social media are used to study and quantify the sentiment among the discussants on the vaccination-or-not topic during phases of the COVID-19 pandemic. Out of all the posts analyzed a majority showed a stronger negative sentiment, prevailing throughout the whole of the examined period, with some spikes or jumps due to the occurrence of certain vaccine-related events distinguishable in the results. Sentiment analysis can be a valuable tool to track public opinions regarding the use, efficacy, safety, and importance of vaccination

Re-examining Phonological and Lexical Correlates of Second Language Comprehensibility:The Role of Rater Experience

Few researchers and teachers would disagree that some linguistic aspects of second language (L2) speech are more crucial than others for successful communication. Underlying this idea is the assumption that communicative success can be broadly defined in terms of speakers’ ability to convey the intended meaning to the interlocutor, which is frequently captured through a listener-based rating of comprehensibility or ease of understanding (e.g. Derwing & Munro, 2009; Levis, 2005). Previous research has shown that communicative success – for example, as defined through comprehensible L2 speech – depends on several linguistic dimensions of L2 output, including its segmental and suprasegmental pronunciation, fluency-based characteristics, lexical and grammatical content, as well as discourse structure (e.g. Field, 2005; Hahn, 2004; Kang et al., 2010; Trofimovich & Isaacs, 2012). Our chief objective in the current study was to explore the L2 comprehensibility construct from a language assessment perspective (e.g. Isaacs & Thomson, 2013), by targeting rater experience as a possible source of variance influencing the degree to which raters use various characteristics of speech in judging L2 comprehensibility. In keeping with this objective, we asked the following question: What is the extent to which linguistic aspects of L2 speech contributing to comprehensibility ratings depend on raters’ experience

A Study of Accomodation of Prosodic and Temporal Features in Spoken Dialogues in View of Speech Technology Applications

Inter-speaker accommodation is a well-known property of human speech and human interaction in general. Broadly it refers to the behavioural patterns of two (or more) interactants and the effect of the (verbal and non-verbal) behaviour of each to that of the other(s). Implementation of thisbehavior in spoken dialogue systems is desirable as an improvement on the naturalness of humanmachine interaction. However, traditional qualitative descriptions of accommodation phenomena do not provide sufficient information for such an implementation. Therefore, a quantitativedescription of inter-speaker accommodation is required. This thesis proposes a methodology of monitoring accommodation during a human or humancomputer dialogue, which utilizes a moving average filter over sequential frames for each speaker. These frames are time-aligned across the speakers, hence the name Time Aligned Moving Average (TAMA). Analysis of spontaneous human dialogue recordings by means of the TAMA methodology reveals ubiquitous accommodation of prosodic features (pitch, intensity and speech rate) across interlocutors, and allows for statistical (time series) modeling of the behaviour, in a way which is meaningful for implementation in spoken dialogue system (SDS) environments.In addition, a novel dialogue representation is proposed that provides an additional point of view to that of TAMA in monitoring accommodation of temporal features (inter-speaker pause length and overlap frequency). This representation is a percentage turn distribution of individual speakercontributions in a dialogue frame which circumvents strict attribution of speaker-turns, by considering both interlocutors as synchronously active. Both TAMA and turn distribution metrics indicate that correlation of average pause length and overlap frequency between speakers can be attributed to accommodation (a debated issue), and point to possible improvements in SDS “turntaking” behaviour. Although the findings of the prosodic and temporal analyses can directly inform SDS implementations, further work is required in order to describe inter-speaker accommodation sufficiently, as well as to develop an adequate testing platform for evaluating the magnitude ofperceived improvement in human-machine interaction. Therefore, this thesis constitutes a first step towards a convincingly useful implementation of accommodation in spoken dialogue systems

Chatting online to learn : repair sequences in text-based L1-L2 interaction in CMC

PhD ThesisBecause of the affordability and widespread availability of modern technologies, researchers of second language learning in Taiwan as well as across the globe have frequently examined Computer-Mediated Communication (CMC) in their studies. However, none has hitherto explored the interactional space of text-based, online chat. This doctoral thesis tries to bridge this research gap by investigating interactional phenomena as they arise in online chatting involving L1 and L2 speakers of English. Special attention is given to how the participants interact with each other to achieve mutual understanding through the sequential structures of language-in-use and the use of online interactional resources. The data for the study is provided by 24 paired participants (i.e. 24 English L1 speakers from geographically remote areas and 24 Taiwanese university students as English L2 speakers) chatting for a 10-week period within a private group on Facebook. Their text-based talk-in-interaction data were retrieved and analysed using the techniques of Conversation Analysis. The salient findings are in relation to the sequential structures of repair sequences. Mutual understandings between L1 and L2 speakers were achieved mostly through repair sequences and the deployment of online interactional devices. There is evidence of incidental learning through CMC taking place not only among L2 speakers but also among L1 speakers who learned interactionally in terms of how to adapt themselves and shape their language-in-use to interact with L2 speakers. This raises new issues with regard to the conventional approach to L2 learning in SLA. In examining the online interactional platform, the data collection and analysis, this study is of importance in providing a better understanding of L1 and L2 speakers’ online talk-in-interaction without participants’ physical co-presence. The study also contributes to the development of, and the literature on, methodology and pedagogy. On the basis of the findings, it is suggested that future studies should continue research on the use of CA for SLA in CMC, with participants of various language proficiencies, and compare the similarities and differences between spoken, online-chat, and written data

A videoconferencing tandem exchange involving adult learners of english and spanish as foreign languages : an interactionist perspective

Pour les fins de la présente étude, des vidéoconférences ayant pour objectif l‘apprentissage en tandem par le jumelage d‘étudiants universitaires de l‘anglais langue étrangère et de l‘espagnol langue étrangère ont été soumises à l‘analyse. Partant du cadre notionnel interactionniste (Ellis et al., 2001a; Loewen, 2005; Long, 1980; Varonis & Gass, 1985), elle vise à examiner quatre phénomènes: 1. les lacunes remarquées par les sujets dans leur propre interlangue, c‘est-à-dire la production d‘épisodes visant la forme/EVF pendant qu‘ils transigent le sens au cours des vidéoconférences avec tâches d‘apprentissage par jumelage de l‘anglais langue étrangère et de l‘espagnol langue étrangère; 2. l‘effet produit ultérieurement sur l‘apprentissage langagier quand l‘apprenant remarque lesdites lacunes épisodiquement; 3. les caractéristiques des EVF permettant le mieux de prédire l‘évolution de l‘apprentissage d‘une langue seconde dans le contexte de l‘utilisation des vidéoconférences; 4. les sources d‘appui auxquelles les sujets ont eu recours pour tirer le meilleur parti des possibilités d‘apprentissage qui se présentent dans une vidéoconférence d‘apprentissage jumelé. Cinq étudiants hispanophones d‘une université mexicaine apprenant l‘anglais langue seconde au niveau intermédiaire ont été jumelés avec cinq étudiants anglophones d‘une université américaine apprenant l‘espagnol au niveau intermédiaire. Les séances d‘apprentissage par jumelage sur vidéoconférence avaient pour point de départ sept tâches sur des thèmes susceptibles d‘intéresser les sujets, à effectuer en conversation libre. Les données recueillies sur dix semaines émanent de trois sources principales: la transcription des vidéoconférences (570 pages), des tests effectués après les séances, soit immédiatement ou avec intervalle d‘attente, et un questionnaire. Grâce à l‘analyse des transcriptions effectuées dans le cadre élaboré par Loewen (2005) pour l‘analyse des EVF, il a été possible de constater que dans les rencontres d‘apprentissage jumelées les étudiants produisaient une quantité importante de tels épisodes, autant du côté espagnol qu‘anglais langue étrangère (915 en tout). Les résultats pour tous les objets linguistiques de type EVF faisant l‘objet de tests effectués après les rencontres ont été réunis afin de permettre la réalisation d‘une analyse Équations d‘estimation généralisées (EEG). L‘analyse des tests effectués immédiatement après les rencontres d‘apprentissage jumelés et de ceux effectués avec un intervalle d‘attente révèle que dans les deux cas, les sujets se rappelaient plus de la moitié des objets linguistiques visés, soient des EVF. Bien que la quantité de tels objets dont les sujets se rappelaient aux tests effectués avec un intervalle d‘attente était moindre, cette différence n‘était pas statistiquement significative. À la différence d‘études réalisées antérieurement (Loewen, 2005; Shekary & Tahririan, 2006) pour lesquelles la prise (uptake) réussie constitue un prédicteur valide de l‘évolution de l‘apprentissage d‘une langue seconde, les analyses de la présente étude portent à croire que l‘unique facteur d‘importance pouvant servir à faire de telles prédictions serait plutôt la correction reportée (deferred timing). L‘analyse des sources d‘appui (Chapelle, 2001) dans lesquelles ont puisé les sujets pendant les rencontres jumelées pour rehausser les possibilités d‘apprentissage pendant les vidéoconférences se sont avérées être de trois types principaux: le clavardage, les images et le tableau blanc. Plus généralement, la présente étude permet d‘affirmer que les rencontres jumelées en vidéoconférence constituent une activité utile pour l‘apprentissage et l‘acquisition d‘une langue seconde. Les incidences de cette étude sur l‘enseignement de même que les pistes de recherches futures qu‘elle suggère sont traitées.The present study analyzed videoconferencing tandem language learning exchanges between university students of English as a foreign language (EFL) and Spanish as a foreign language (SFL). Based on an interactionist perspective (Ellis et al., 2001a; Loewen, 2005; Long, 1980; Varonis & Gass, 1985), it sought to explore: 1. participants noticing of the gap in their interlanguage (i.e., production of focus-on-form episodes/FFEs) during the negotiation of meaning in the context of tandem language learning tasks for EFL and SFL, 2. the effect of incidental noticing on subsequent language learning, 3. the characteristics of FFEs that best predict L2 learning in a videoconferencing context, and 4. the support used by participants to enhance language learning opportunities during the exchanges. Five intermediate level Spanish-speaking EFL students from a Mexican university were paired up with five intermediate level Englishspeaking SFL students from an American university. Seven free conversation type tasks on topics of interest to the participants were the basis for the videoconferencing tandem sessions. Data collected over a ten-week period were gathered from three main sources: videoconferencing session transcripts (570 pages), immediate and delayed posttests, and a background questionnaire. Drawing on Loewen‘s (2005) framework for the analysis of FFEs, the transcripts revealed that students generated a substantive number of FFEs in both the EFL and SFL parts of the tandem exchange (915 in total). Results from the immediate and delayed posttests indicated that participants recalled over half of the targeted FFE linguistic items on immediate and delayed posttests. Although fewer items were recalled on the delayed posttests, this difference was not significant. Results from the FFE linguistic items targeted for the posttests were combined in order to carry out a Generalized Estimating Equation (GEE) analysis. In contrast to previous studies (Loewen, 2005; Shekary & Tahririan, 2006), where successful uptake was a valid predictor for L2 learning, the analysis of the present study revealed that the only significant predictor for L2 learning was deferred timing. Analysis of the support (Chapelle, 2001) used by participants to enhance L2 language learning opportunities during videoconferencing revealed three major types: chat, pictures, and use of the whiteboard. More generally, the present study supports the claim that tandem language learning through videoconferencing is a useful activity for promoting L2 learning/acquisition. Implications for teaching and suggestions for future research are discussed