A Framework for Research in Gamified Mobile Guide Applications using Embodied Conversational Agents (ECAs)

Mobile Guides are mobile applications that provide players with local and location-based services (LBS), such as navigation assistance, where and when they need them most. Advances in mobile technologies in recent years have enabled the gamification of these applications, opening up new opportunities to transfer education and culture through game play. However, adding traditional game elements such as PBLs (points, badges, and leaderboards) alone cannot ensure that the intended learning outcomes will be met, as the player’s cognitive resources are shared between the application and the surrounding environment. This distribution of resources prevents players from easily immersing themselves into the educational scenario. Adding artificial conversational characters (ECAs) that simulate the social norms found in real-life human-to-human guide scenarios has the potential to address this problem and improve the player’s experience and learning of cultural narratives [1]. Although significant progress has been made towards creating game-like mobile guides with ECAs ([2], [3]), there is still a lack of a unified framework that enables researchers and practitioners to investigate the potential effects of such applications to players and how to approach the concepts of player experience, cognitive accessibility and usability in this context. This paper presents a theoretically-well supported research framework consisted of four key components: differences in players, different features of the gamified task, aspects of how the ECA looks, sound or behaves and different mobile environments. Furthermore, it provides based on this framework a working definition of what player experience, cognitive accessibility and usability are in the context of game-like mobile guide applications. Finally, a synthesis of the results of six empirical studies conducted within this research framework is discussed and a series of design guidelines for the effective gamification of mobile guide applications using ECAs are presented. Results show that an ECA can positively affect the quality of the player’s experience, but it did not elicit better player retention of cultural narratives and navigation of routes

Directional adposition use in English, Swedish and Finnish

Directional adpositions such as to the left of describe where a Figure is in relation to a Ground. English and Swedish directional adpositions refer to the location of a Figure in relation to a Ground, whether both are static or in motion. In contrast, the Finnish directional adpositions edellä (in front of) and jäljessä (behind) solely describe the location of a moving Figure in relation to a moving Ground (Nikanne, 2003). When using directional adpositions, a frame of reference must be assumed for interpreting the meaning of directional adpositions. For example, the meaning of to the left of in English can be based on a relative (speaker or listener based) reference frame or an intrinsic (object based) reference frame (Levinson, 1996). When a Figure and a Ground are both in motion, it is possible for a Figure to be described as being behind or in front of the Ground, even if neither have intrinsic features. As shown by Walker (in preparation), there are good reasons to assume that in the latter case a motion based reference frame is involved. This means that if Finnish speakers would use edellä (in front of) and jäljessä (behind) more frequently in situations where both the Figure and Ground are in motion, a difference in reference frame use between Finnish on one hand and English and Swedish on the other could be expected. We asked native English, Swedish and Finnish speakers’ to select adpositions from a language specific list to describe the location of a Figure relative to a Ground when both were shown to be moving on a computer screen. We were interested in any differences between Finnish, English and Swedish speakers. All languages showed a predominant use of directional spatial adpositions referring to the lexical concepts TO THE LEFT OF, TO THE RIGHT OF, ABOVE and BELOW. There were no differences between the languages in directional adpositions use or reference frame use, including reference frame use based on motion. We conclude that despite differences in the grammars of the languages involved, and potential differences in reference frame system use, the three languages investigated encode Figure location in relation to Ground location in a similar way when both are in motion. Levinson, S. C. (1996). Frames of reference and Molyneux’s question: Crosslingiuistic evidence. In P. Bloom, M.A. Peterson, L. Nadel & M.F. Garrett (Eds.) Language and Space (pp.109-170). Massachusetts: MIT Press. Nikanne, U. (2003). How Finnish postpositions see the axis system. In E. van der Zee & J. Slack (Eds.), Representing direction in language and space. Oxford, UK: Oxford University Press. Walker, C. (in preparation). Motion encoding in language, the use of spatial locatives in a motion context. Unpublished doctoral dissertation, University of Lincoln, Lincoln. United Kingdo

Human-Computer Interaction

In this book the reader will find a collection of 31 papers presenting different facets of Human Computer Interaction, the result of research projects and experiments as well as new approaches to design user interfaces. The book is organized according to the following main topics in a sequential order: new interaction paradigms, multimodality, usability studies on several interaction mechanisms, human factors, universal design and development methodologies and tools

Predicting Perceived Problems in 24-hour Dietary Recall, and Evaluating its Potential for Behaviour Change: A Quantitative Think-Aloud Approach in an Australian Population

This thesis incorporates the think-aloud methodology across two studies that investigate the experience of 24-hour dietary recall and its potential for improving future nutritional behaviours. The methodological integration incorporated in the thesis aids in revealing different aspects of thinking when recalling and assessing our dietary behaviours, as well as their importance in the accuracy of self-reported psychological and nutritional research

Emotion Recognition by Video: A review

Video emotion recognition is an important branch of affective computing, and its solutions can be applied in different fields such as human-computer interaction (HCI) and intelligent medical treatment. Although the number of papers published in the field of emotion recognition is increasing, there are few comprehensive literature reviews covering related research on video emotion recognition. Therefore, this paper selects articles published from 2015 to 2023 to systematize the existing trends in video emotion recognition in related studies. In this paper, we first talk about two typical emotion models, then we talk about databases that are frequently utilized for video emotion recognition, including unimodal databases and multimodal databases. Next, we look at and classify the specific structure and performance of modern unimodal and multimodal video emotion recognition methods, talk about the benefits and drawbacks of each, and then we compare them in detail in the tables. Further, we sum up the primary difficulties right now looked by video emotion recognition undertakings and point out probably the most encouraging future headings, such as establishing an open benchmark database and better multimodal fusion strategys. The essential objective of this paper is to assist scholarly and modern scientists with keeping up to date with the most recent advances and new improvements in this speedy, high-influence field of video emotion recognition

Sex differences in emotional concordance.

Emotions involve response synchronization across experiential, physiological, and behavioral systems, referred to as concordance or coherence. Women are thought to be more emotionally aware and expressive than men and may therefore display stronger response concordance; however, research on this topic is scant. Using a random-order film-average design, we assessed concordance among experiential (arousal, valence), autonomic (electrodermal activity, heart rate, preejection period, respiratory sinus arrhythmia), respiratory (respiratory rate), and behavioral (corrugator and zygomatic electromyography) responses to 15 two-minute films varying in valence and arousal. We then calculated for each participant and pair of measures a within-subject correlation index using averages from the 15 films. Pronounced individual concordance of up to 0.9 was observed. Arousal-physiology and valence-behavior concordances were particularly pronounced. Women displayed higher concordance than men for almost all pairs of measures. Findings indicate stronger psychophysiological response coupling in women than men and provide novel insights into affective differences between the sexes

Eye quietness and quiet eye in expert and novice golf performance: an electrooculographic analysis

Quiet eye (QE) is the final ocular fixation on the target of an action (e.g., the ball in golf putting). Camerabased eye-tracking studies have consistently found longer QE durations in experts than novices; however, mechanisms underlying QE are not known. To offer a new perspective we examined the feasibility of measuring the QE using electrooculography (EOG) and developed an index to assess ocular activity across time: eye quietness (EQ). Ten expert and ten novice golfers putted 60 balls to a 2.4 m distant hole. Horizontal EOG (2ms resolution) was recorded from two electrodes placed on the outer sides of the eyes. QE duration was measured using a EOG voltage threshold and comprised the sum of the pre-movement and post-movement initiation components. EQ was computed as the standard deviation of the EOG in 0.5 s bins from –4 to +2 s, relative to backswing initiation: lower values indicate less movement of the eyes, hence greater quietness. Finally, we measured club-ball address and swing durations. T-tests showed that total QE did not differ between groups (p = .31); however, experts had marginally shorter pre-movement QE (p = .08) and longer post-movement QE (p < .001) than novices. A group × time ANOVA revealed that experts had less EQ before backswing initiation and greater EQ after backswing initiation (p = .002). QE durations were inversely correlated with EQ from –1.5 to 1 s (rs = –.48 - –.90, ps = .03 - .001). Experts had longer swing durations than novices (p = .01) and, importantly, swing durations correlated positively with post-movement QE (r = .52, p = .02) and negatively with EQ from 0.5 to 1s (r = –.63, p = .003). This study demonstrates the feasibility of measuring ocular activity using EOG and validates EQ as an index of ocular activity. Its findings challenge the dominant perspective on QE and provide new evidence that expert-novice differences in ocular activity may reflect differences in the kinematics of how experts and novices execute skills

Dissociation and interpersonal autonomic physiology in psychotherapy research: an integrative view encompassing psychodynamic and neuroscience theoretical frameworks

Interpersonal autonomic physiology is an interdisciplinary research field, assessing the relational interdependence of two (or more) interacting individual both at the behavioral and psychophysiological levels. Despite its quite long tradition, only eight studies since 1955 have focused on the interaction of psychotherapy dyads, and none of them have focused on the shared processual level, assessing dynamic phenomena such as dissociation. We longitudinally observed two brief psychodynamic psychotherapies, entirely audio and video-recorded (16 sessions, weekly frequency, 45 min.). Autonomic nervous system measures were continuously collected during each session. Personality, empathy, dissociative features and clinical progress measures were collected prior and post therapy, and after each clinical session. Two-independent judges, trained psychotherapist, codified the interactions\u2019 micro-processes. Time-series based analyses were performed to assess interpersonal synchronization and de-synchronization in patient\u2019s and therapist\u2019s physiological activity. Psychophysiological synchrony revealed a clear association with empathic attunement, while desynchronization phases (range of length 30-150 sec.) showed a linkage with dissociative processes, usually associated to the patient\u2019s narrative core relational trauma. Our findings are discussed under the perspective of psychodynamic models of Stern (\u201cpresent moment\u201d), Sander, Beebe and Lachmann (dyad system model of interaction), Lanius (Trauma model), and the neuroscientific frameworks proposed by Thayer (neurovisceral integration model), and Porges (polyvagal theory). The collected data allows to attempt an integration of these theoretical approaches under the light of Complex Dynamic Systems. The rich theoretical work and the encouraging clinical results might represents a new fascinating frontier of research in psychotherapy

The Effect of Rule-Based Scaffoldings on Second Grade Students\u27 Digital Storytelling

Digital storytelling is a powerful method for revitalizing literacy instruction. Past research suggested that digital storytelling activities improve students’ writing skills through construction of various types of stories. However, little research has investigated in what ways educators can promote students’ interests and actual abilities to express narrative discourse in a digital format. Recent research indicated that the use of story grammars help students develop sophisticated stories. From this perspective, Labov’s story grammar emphasized two functions of good story structure: reference—the listeners (or readers) are told what happened, and evaluation—the speakers (or writers) reveal their attitude toward the events of the narrative. Meanwhile, current practitioner based research suggests that Lambert’s seven elements approach of digital storytelling emerged as a practical guideline for creating effective digital stories in elementary classrooms. Therefore, the purpose of this study was to investigate the potential of three instructional approaches: Labov’s story grammar only, Lambert’s seven elements only, and both instructional approaches, as scaffolding(s) for students’ digital storytelling. Specifically, a quantitative research design with three experimental groups and one control group, pre-test and post-test, was employed. Participants included 104 second-graders (largely from high socioeconomic status families), with 26 in each of four classrooms. Therefore, the three instructional scaffold approaches and one non-scaffold supported approach were randomly assigned to each of four classrooms respectively to support students’ story writing, storytelling, story design and construction using Movie Maker software. Students’ understanding of narrative writing was assessed before and after the implementation of the intervention. The results indicated that the instructional scaffolding positively enhanced students’ performance in story writing, storytelling, as well as verbal and visual expression. In particular, the story grammar scaffolding motivated students to produce coherent, more sophisticated stories. The seven elements scaffolding sparked students’ creative verbal and visual expressions and stimulated them to elaborate using a variety of adjectives in their digital stories. When both scaffolding approaches were implemented, students significantly outperformed the other groups on the quality of story content, story coherency and narrative knowledge. The implications of these findings and recommendations for future research are discussed

Horizontal asymmetries derived from script direction : consequences for attention and action

Tese apresentada para cumprimento dos requisitos necessários à obtenção do grau de Doutor em Psicologia na área de especialização de Psicologia Social apresentada no ISPA - Instituto Universitário, no ano de 2021.A direção de leitura e escrita estabelecem uma trajetória preferencial de exploração do espaço que é reforçada por diversas regularidades culturais consistentes com essa direccionalidade. A correlação espaço-movimento cria um esquema para a ação que enviesa a representação da agência humana, estendendo-se à representação de outros conceitos abstratos que não possuem bases sensoriomotoras. A dimensão horizontal é recrutada para melhor compreender estes conceitos, sendo ancorados de acordo com a direção de escrita e leitura da nossa língua. A assimetria espacial que esta direccionalidade induz constitui um contributo crucial para a área do embodiment, tendo sido demonstrado que afeta processos sociais e cognitivos. Contudo, os processos específicos que estas assimetrias ativam permanecem pouco explorados. Em sete estudos, esta dissertação investiga de que forma as assimetrias espaciais afetam inferências sociais e a performance visuo-motora para com estímulos ancorados na dimensão horizontal. O primeiro estudo indica que inferências sociais relacionadas com agência são preferencialmente atribuídas a faces de perfil orientadas para a direita (versus esquerda). Em duas experiências, o segundo estudo mostra que faces orientadas para a direita servem como pistas para a orientação de atenção. Faces orientadas para a direita, que traduzem a direção utilizada para representar a agência humana, facilitam a atenção para e deteção de alvos no campo visual direito, comparativamente a faces orientadas para a esquerda no campo visual esquerdo. No terceiro estudo, as faces foram substituídas por palavras temporais auditivas e visuais, que se sabe serem ancoradas horizontalmente. A assimetria espacial foi testada em duas experiências em comunidades com direções de leitura e escrita opostas (Português e Árabe). Observou-se uma ancoragem contrária do conceito abstrato ‘tempo’ entre as duas amostras (Português: passado-esquerda/futuro-direita; Árabe: passado-direita/futuro-esquerda). Adicionalmente, uma performance assimétrica reversa entre as duas comunidades linguísticas confirma que o mapeamento do tempo é enviesado pelos hábitos ortográficos e pela representação cultural da agência humana. Isto é, palavras temporais que coincidem com a direção induzida por ambos os sistemas de escrita (i.e., palavras relacionadas com futuro), dão origem a vantagem à direita na amostra Portuguesa, e vantagem à esquerda na amostra Árabe. O quarto estudo estendeu estes resultados à categoria da política, tipicamente representada através de coordenadas de esquerda e de direita. Respostas manuais e atencionais foram mais rápidas para alvos localizados à direita após terem sido apresentadas termos políticos de direita (versus alvos à esquerda após termos políticos de esquerda), que correspondiam à direção em que habitualmente se representa movimento. O quinto e último estudo demonstrou que a apresentação de palavras temporais simultaneamente com um tom auditivo não-espacial impede os efeitos de emergirem. Estas pistas bimodais revelaram as condições limitativas dos efeitos da assimetria espacial. Em conclusão, esta dissertação demonstra que existe uma propriedade genérica de movimento que deriva da direção ortográfica e que é transversal à representação de estímulos distintos, em várias tarefas e modalidades sensoriais. Estes resultados oferecem uma perspetiva mais abrangente sobre o impacto prevalente que uma característica da língua aparentemente irrelevante tem em processos cognitivos fundamentais de perceção, atenção, e julgamento.The directional activities of reading and writing have been shown to ground a preferential trajectory when scanning space. This horizontal directional formation is further reinforced by other cultural regularities that overlap with it. This space-movement correlation creates a left-right (or vice-versa) schema for action that biases the representation of human agency and extends to the representation of other abstract concepts lacking experiential sensorimotor bases. Consequently, the horizontal dimension is recruited to reason about abstract concepts that are mapped congruently with one’s dominant reading and writing or script direction. The spatial asymmetry that this combined directionality induces is a core finding in the embodiment area and has been shown to affect important social and cognitive processes. However, the specific processes activated by these asymmetries remain unclear. A series of seven experiments are outlined to investigate how spatial asymmetries affect social inferences and visuomotor performance to stimuli anchored in the horizontal dimension. The first study indicated that a range of agency-related social inferences are preferentially assigned to face profiles oriented rightward (versus leftward). Across two experiments, the second study showed that right oriented faces serve as attention-orienting primes. Rightward faces, which are in line with the direction used to represent human agency, facilitate attention to and detection of targets on the right hemifield, relative to leftward faces and targets on the left hemifield. In the third study, face primes were replaced by visual and auditory time words known to ground horizontally in space. Spatial asymmetries were tested in two experiments with communities holding opposite writing scripts (Portuguese and Arabic). We observed the mapping of time to be reversed between the two samples (Portuguese: past-left/future-right; Arabic: past-right/future-left). Further, a mirrored asymmetric performance between the two linguistic communities confirmed that the mapping of time is biased by orthographic habits and the cultural representation of human agency. That is, time words that coincide with the direction induced by both writing systems (i.e., future-related) gave rise to right-side advantage in the Portuguese sample and left-side advantage in the Arabic sample. The fourth study extended these results to the category of politics, commonly represented through coordinates of left and right. Manual and gaze responses were faster to targets embedded on the right following conservatism-related words (versus the left following socialism-related words) that embody the habitualized rightward movement direction. The fifth and final study demonstrated that presenting time words synchronously with an auditory nonspatial tone impaired cueing effects. These bimodal cues revealed the boundary conditions of the spatial agency bias. Overall, this dissertation underscores that a generic property of movement that is derived from orthographic direction underlies the representation of very distinct stimuli across tasks and sensory modalities. These findings offer a broader perspective on the pervasive impact a seemingly irrelevant feature of language has on fundamental cognitive processes of perception, attention, and judgment