Investigating Social Presence and Communication with Embodied Avatars in Room-Scale Virtual Reality

Submission includes video.Room-scale virtual reality (VR) holds great potential as a medium for communication and collaboration in remote and same-time, same-place settings. Related work has established that movement realism can create a strong sense of social presence, even in the absence of photorealism. Here, we explore the noteworthy attributes of communicative interaction using embodied minimal avatars in room-scale VR in the same-time, same-place setting. Our system is the first in the research community to enable this kind of interaction, as far as we are aware. We carried out an experiment in which pairs of users performed two activities in contrasting variants: VR vs. face-to-face (F2F), and 2D vs. 3D. Objective and subjective measures were used to compare these, including motion analysis, electrodermal activity, questionnaires, retrospective think-aloud protocol, and interviews. On the whole, participants communicated effectively in VR to complete their tasks, and reported a strong sense of social presence. The system's high fidelity capture and display of movement seems to have been a key factor in supporting this. Our results confirm some expected shortcomings of VR compared to F2F, but also some non-obvious advantages. The limited anthropomorphic properties of the avatars presented some difficulties, but the impact of these varied widely between the activities. In the 2D vs. 3D comparison, the basic affordance of freehand drawing in 3D was new to most participants, resulting in novel observations and open questions. We also present methodological observations across all conditions concerning the measures that did and did not reveal differences between conditions, including unanticipated properties of the think-aloud protocol applied to VR

A Virtual Conversational Agent for Teens with Autism: Experimental Results and Design Lessons

We present the design of an online social skills development interface for teenagers with autism spectrum disorder (ASD). The interface is intended to enable private conversation practice anywhere, anytime using a web-browser. Users converse informally with a virtual agent, receiving feedback on nonverbal cues in real-time, and summary feedback. The prototype was developed in consultation with an expert UX designer, two psychologists, and a pediatrician. Using the data from 47 individuals, feedback and dialogue generation were automated using a hidden Markov model and a schema-driven dialogue manager capable of handling multi-topic conversations. We conducted a study with nine high-functioning ASD teenagers. Through a thematic analysis of post-experiment interviews, identified several key design considerations, notably: 1) Users should be fully briefed at the outset about the purpose and limitations of the system, to avoid unrealistic expectations. 2) An interface should incorporate positive acknowledgment of behavior change. 3) Realistic appearance of a virtual agent and responsiveness are important in engaging users. 4) Conversation personalization, for instance in prompting laconic users for more input and reciprocal questions, would help the teenagers engage for longer terms and increase the system's utility

by

th

Machine Learning and Virtual Reality on Body Movements¿ Behaviors to Classify Children with Autism Spectrum Disorder

[EN] Autism spectrum disorder (ASD) is mostly diagnosed according to behavioral symptoms in sensory, social, and motor domains. Improper motor functioning, during diagnosis, involves the qualitative evaluation of stereotyped and repetitive behaviors, while quantitative methods that classify body movements' frequencies of children with ASD are less addressed. Recent advances in neuroscience, technology, and data analysis techniques are improving the quantitative and ecological validity methods to measure specific functioning in ASD children. On one side, cutting-edge technologies, such as cameras, sensors, and virtual reality can accurately detect and classify behavioral biomarkers, as body movements in real-life simulations. On the other, machine-learning techniques are showing the potential for identifying and classifying patients' subgroups. Starting from these premises, three real-simulated imitation tasks have been implemented in a virtual reality system whose aim is to investigate if machine-learning methods on movement features and frequency could be useful in discriminating ASD children from children with typical neurodevelopment. In this experiment, 24 children with ASD and 25 children with typical neurodevelopment participated in a multimodal virtual reality experience, and changes in their body movements were tracked by a depth sensor camera during the presentation of visual, auditive, and olfactive stimuli. The main results showed that ASD children presented larger body movements than TD children, and that head, trunk, and feet represent the maximum classification with an accuracy of 82.98%. Regarding stimuli, visual condition showed the highest accuracy (89.36%), followed by the visual-auditive stimuli (74.47%), and visual-auditive-olfactory stimuli (70.21%). Finally, the head showed the most consistent performance along with the stimuli, from 80.85% in visual to 89.36% in visual-auditive-olfactory condition. The findings showed the feasibility of applying machine learning and virtual reality to identify body movements' biomarkers that could contribute to improving ASD diagnosis.This work was supported by the Spanish Ministry of Economy, Industry, and Competitiveness funded project "Immersive virtual environment for the evaluation and training of children with autism spectrum disorder: T Room" (IDI-20170912) and by the Generalitat Valenciana funded project REBRAND (PROMETEO/2019/105). Furthermore, this work was co-founded by the European Union through the Operational Program of the European Regional development Fund (ERDF) of the Valencian Community 2014-2020 (IDIFEDER/2018/029).Alcañiz Raya, ML.; Marín-Morales, J.; Minissi, ME.; Teruel Garcia, G.; Abad, L.; Chicchi-Giglioli, IA. (2020). Machine Learning and Virtual Reality on Body Movements¿ Behaviors to Classify Children with Autism Spectrum Disorder. Journal of Clinical Medicine. 9(5):1-20. https://doi.org/10.3390/jcm9051260S12095https://www.who.int/news-room/fact-sheets/detail/autism-spectrum-disordersAnagnostou, E., Zwaigenbaum, L., Szatmari, P., Fombonne, E., Fernandez, B. A., Woodbury-Smith, M., … Scherer, S. W. (2014). Autism spectrum disorder: advances in evidence-based practice. Canadian Medical Association Journal, 186(7), 509-519. doi:10.1503/cmaj.121756Lord, C., Risi, S., DiLavore, P. S., Shulman, C., Thurm, A., & Pickles, A. (2006). Autism From 2 to 9 Years of Age. Archives of General Psychiatry, 63(6), 694. doi:10.1001/archpsyc.63.6.694Schmidt, L., Kirchner, J., Strunz, S., Broźus, J., Ritter, K., Roepke, S., & Dziobek, I. (2015). Psychosocial Functioning and Life Satisfaction in Adults With Autism Spectrum Disorder Without Intellectual Impairment. Journal of Clinical Psychology, 71(12), 1259-1268. doi:10.1002/jclp.22225Turner, M. (1999). Annotation: Repetitive Behaviour in Autism: A Review of Psychological Research. Journal of Child Psychology and Psychiatry, 40(6), 839-849. doi:10.1111/1469-7610.00502Lewis, M. H., & Bodfish, J. W. (1998). Repetitive behavior disorders in autism. Mental Retardation and Developmental Disabilities Research Reviews, 4(2), 80-89. doi:10.1002/(sici)1098-2779(1998)4:23.0.co;2-0Mahone, E. M., Bridges, D., Prahme, C., & Singer, H. S. (2004). Repetitive arm and hand movements (complex motor stereotypies) in children. The Journal of Pediatrics, 145(3), 391-395. doi:10.1016/j.jpeds.2004.06.014MacDonald, R., Green, G., Mansfield, R., Geckeler, A., Gardenier, N., Anderson, J., … Sanchez, J. (2007). Stereotypy in young children with autism and typically developing children. Research in Developmental Disabilities, 28(3), 266-277. doi:10.1016/j.ridd.2006.01.004Singer, H. S. (2009). Motor Stereotypies. Seminars in Pediatric Neurology, 16(2), 77-81. doi:10.1016/j.spen.2009.03.008Lidstone, J., Uljarević, M., Sullivan, J., Rodgers, J., McConachie, H., Freeston, M., … Leekam, S. (2014). Relations among restricted and repetitive behaviors, anxiety and sensory features in children with autism spectrum disorders. Research in Autism Spectrum Disorders, 8(2), 82-92. doi:10.1016/j.rasd.2013.10.001GOLDMAN, S., WANG, C., SALGADO, M. W., GREENE, P. E., KIM, M., & RAPIN, I. (2009). Motor stereotypies in children with autism and other developmental disorders. Developmental Medicine & Child Neurology, 51(1), 30-38. doi:10.1111/j.1469-8749.2008.03178.xLord, C., Rutter, M., & Le Couteur, A. (1994). Autism Diagnostic Interview-Revised: A revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. Journal of Autism and Developmental Disorders, 24(5), 659-685. doi:10.1007/bf02172145Volkmar, F. R., State, M., & Klin, A. (2009). Autism and autism spectrum disorders: diagnostic issues for the coming decade. Journal of Child Psychology and Psychiatry, 50(1-2), 108-115. doi:10.1111/j.1469-7610.2008.02010.xReaven, J. A., Hepburn, S. L., & Ross, R. G. (2008). Use of the ADOS and ADI-R in Children with Psychosis: Importance of Clinical Judgment. Clinical Child Psychology and Psychiatry, 13(1), 81-94. doi:10.1177/1359104507086343Torres, E. B., Brincker, M., Isenhower, R. W., Yanovich, P., Stigler, K. A., Nurnberger, J. I., … José, J. V. (2013). Autism: the micro-movement perspective. Frontiers in Integrative Neuroscience, 7. doi:10.3389/fnint.2013.00032Möricke, E., Buitelaar, J. K., & Rommelse, N. N. J. (2015). Do We Need Multiple Informants When Assessing Autistic Traits? The Degree of Report Bias on Offspring, Self, and Spouse Ratings. Journal of Autism and Developmental Disorders, 46(1), 164-175. doi:10.1007/s10803-015-2562-yCHAYTOR, N., SCHMITTEREDGECOMBE, M., & BURR, R. (2006). Improving the ecological validity of executive functioning assessment. Archives of Clinical Neuropsychology, 21(3), 217-227. doi:10.1016/j.acn.2005.12.002Brunswik, E. (1955). Representative design and probabilistic theory in a functional psychology. Psychological Review, 62(3), 193-217. doi:10.1037/h0047470Gillberg, C., & Rasmussen, P. (1994). Brief report: Four case histories and a literature review of williams syndrome and autistic behavior. Journal of Autism and Developmental Disorders, 24(3), 381-393. doi:10.1007/bf02172235Parsons, S. (2016). Authenticity in Virtual Reality for assessment and intervention in autism: A conceptual review. Educational Research Review, 19, 138-157. doi:10.1016/j.edurev.2016.08.001Francis, K. (2005). Autism interventions: a critical update. Developmental Medicine & Child Neurology, 47(7), 493-499. doi:10.1017/s0012162205000952Albinali, F., Goodwin, M. S., & Intille, S. S. (2009). Recognizing stereotypical motor movements in the laboratory and classroom. Proceedings of the 11th international conference on Ubiquitous computing. doi:10.1145/1620545.1620555Pyles, D. A. M., Riordan, M. M., & Bailey, J. S. (1997). The stereotypy analysis: An instrument for examining environmental variables associated with differential rates of stereotypic behavior. Research in Developmental Disabilities, 18(1), 11-38. doi:10.1016/s0891-4222(96)00034-0Nosek, B. A., Hawkins, C. B., & Frazier, R. S. (2011). Implicit social cognition: from measures to mechanisms. Trends in Cognitive Sciences, 15(4), 152-159. doi:10.1016/j.tics.2011.01.005Forscher, P. S., Lai, C. K., Axt, J. R., Ebersole, C. R., Herman, M., Devine, P. G., & Nosek, B. A. (2019). A meta-analysis of procedures to change implicit measures. Journal of Personality and Social Psychology, 117(3), 522-559. doi:10.1037/pspa0000160LeDoux, J. E., & Pine, D. S. (2016). Using Neuroscience to Help Understand Fear and Anxiety: A Two-System Framework. American Journal of Psychiatry, 173(11), 1083-1093. doi:10.1176/appi.ajp.2016.16030353Fenning, R. M., Baker, J. K., Baucom, B. R., Erath, S. A., Howland, M. A., & Moffitt, J. (2017). Electrodermal Variability and Symptom Severity in Children with Autism Spectrum Disorder. Journal of Autism and Developmental Disorders, 47(4), 1062-1072. doi:10.1007/s10803-016-3021-0Nikula, R. (1991). Psychological Correlates of Nonspecific Skin Conductance Responses. Psychophysiology, 28(1), 86-90. doi:10.1111/j.1469-8986.1991.tb03392.xAlcañiz Raya, M., Chicchi Giglioli, I. A., Marín-Morales, J., Higuera-Trujillo, J. L., Olmos, E., Minissi, M. E., … Abad, L. (2020). Application of Supervised Machine Learning for Behavioral Biomarkers of Autism Spectrum Disorder Based on Electrodermal Activity and Virtual Reality. Frontiers in Human Neuroscience, 14. doi:10.3389/fnhum.2020.00090Cunningham, W. A., Raye, C. L., & Johnson, M. K. (2004). Implicit and Explicit Evaluation: fMRI Correlates of Valence, Emotional Intensity, and Control in the Processing of Attitudes. Journal of Cognitive Neuroscience, 16(10), 1717-1729. doi:10.1162/0898929042947919Kopton, I. M., & Kenning, P. (2014). Near-infrared spectroscopy (NIRS) as a new tool for neuroeconomic research. Frontiers in Human Neuroscience, 8. doi:10.3389/fnhum.2014.00549Nickel, P., & Nachreiner, F. (2003). Sensitivity and Diagnosticity of the 0.1-Hz Component of Heart Rate Variability as an Indicator of Mental Workload. Human Factors: The Journal of the Human Factors and Ergonomics Society, 45(4), 575-590. doi:10.1518/hfes.45.4.575.27094Di Martino, A., Yan, C.-G., Li, Q., Denio, E., Castellanos, F. X., Alaerts, K., … Milham, M. P. (2013). The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Molecular Psychiatry, 19(6), 659-667. doi:10.1038/mp.2013.78Van Hecke, A. V., Lebow, J., Bal, E., Lamb, D., Harden, E., Kramer, A., … Porges, S. W. (2009). Electroencephalogram and Heart Rate Regulation to Familiar and Unfamiliar People in Children With Autism Spectrum Disorders. Child Development, 80(4), 1118-1133. doi:10.1111/j.1467-8624.2009.01320.xCoronato, A., De Pietro, G., & Paragliola, G. (2014). A situation-aware system for the detection of motion disorders of patients with Autism Spectrum Disorders. Expert Systems with Applications, 41(17), 7868-7877. doi:10.1016/j.eswa.2014.05.011Goodwin, M. S., Intille, S. S., Albinali, F., & Velicer, W. F. (2010). Automated Detection of Stereotypical Motor Movements. Journal of Autism and Developmental Disorders, 41(6), 770-782. doi:10.1007/s10803-010-1102-zRodrigues, J. L., Gonçalves, N., Costa, S., & Soares, F. (2013). Stereotyped movement recognition in children with ASD. Sensors and Actuators A: Physical, 202, 162-169. doi:10.1016/j.sna.2013.04.019Crippa, A., Salvatore, C., Perego, P., Forti, S., Nobile, M., Molteni, M., & Castiglioni, I. (2015). Use of Machine Learning to Identify Children with Autism and Their Motor Abnormalities. Journal of Autism and Developmental Disorders, 45(7), 2146-2156. doi:10.1007/s10803-015-2379-8Wedyan, M., Al-Jumaily, A., & Crippa, A. (2019). Using machine learning to perform early diagnosis of Autism Spectrum Disorder based on simple upper limb movements. International Journal of Hybrid Intelligent Systems, 15(4), 195-206. doi:10.3233/his-190278Parsons, S., Mitchell, P., & Leonard, A. (2004). The Use and Understanding of Virtual Environments by Adolescents with Autistic Spectrum Disorders. Journal of Autism and Developmental Disorders, 34(4), 449-466. doi:10.1023/b:jadd.0000037421.98517.8dParsons, T. D., Rizzo, A. A., Rogers, S., & York, P. (2009). Virtual reality in paediatric rehabilitation: A review. Developmental Neurorehabilitation, 12(4), 224-238. doi:10.1080/17518420902991719Bowman, D. A., Gabbard, J. L., & Hix, D. (2002). A Survey of Usability Evaluation in Virtual Environments: Classification and Comparison of Methods. Presence: Teleoperators and Virtual Environments, 11(4), 404-424. doi:10.1162/105474602760204309Pastorelli, E., & Herrmann, H. (2013). A Small-scale, Low-budget Semi-immersive Virtual Environment for Scientific Visualization and Research. Procedia Computer Science, 25, 14-22. doi:10.1016/j.procs.2013.11.003Cobb, S. V. G., Nichols, S., Ramsey, A., & Wilson, J. R. (1999). Virtual Reality-Induced Symptoms and Effects (VRISE). Presence: Teleoperators and Virtual Environments, 8(2), 169-186. doi:10.1162/105474699566152Wallace, S., Parsons, S., Westbury, A., White, K., White, K., & Bailey, A. (2010). Sense of presence and atypical social judgments in immersive virtual environments. Autism, 14(3), 199-213. doi:10.1177/1362361310363283Lorenzo, G., Lledó, A., Arráez-Vera, G., & Lorenzo-Lledó, A. (2018). The application of immersive virtual reality for students with ASD: A review between 1990–2017. Education and Information Technologies, 24(1), 127-151. doi:10.1007/s10639-018-9766-7Bailenson, J. N., Yee, N., Merget, D., & Schroeder, R. (2006). The Effect of Behavioral Realism and Form Realism of Real-Time Avatar Faces on Verbal Disclosure, Nonverbal Disclosure, Emotion Recognition, and Copresence in Dyadic Interaction. Presence: Teleoperators and Virtual Environments, 15(4), 359-372. doi:10.1162/pres.15.4.359Cipresso, P., Giglioli, I. A. C., Raya, M. A., & Riva, G. (2018). The Past, Present, and Future of Virtual and Augmented Reality Research: A Network and Cluster Analysis of the Literature. Frontiers in Psychology, 9. doi:10.3389/fpsyg.2018.02086Cummings, J. J., & Bailenson, J. N. (2015). How Immersive Is Enough? A Meta-Analysis of the Effect of Immersive Technology on User Presence. Media Psychology, 19(2), 272-309. doi:10.1080/15213269.2015.1015740Skalski, P., & Tamborini, R. (2007). The Role of Social Presence in Interactive Agent-Based Persuasion. Media Psychology, 10(3), 385-413. doi:10.1080/15213260701533102Slater, M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3549-3557. doi:10.1098/rstb.2009.0138Baños, R. M., Botella, C., Garcia-Palacios, A., Villa, H., Perpiña, C., & Alcañiz, M. (2000). Presence and Reality Judgment in Virtual Environments: A Unitary Construct? CyberPsychology & Behavior, 3(3), 327-335. doi:10.1089/10949310050078760Bente, G., Rüggenberg, S., Krämer, N. C., & Eschenburg, F. (2008). Avatar-Mediated Networking: Increasing Social Presence and Interpersonal Trust in Net-Based Collaborations. Human Communication Research, 34(2), 287-318. doi:10.1111/j.1468-2958.2008.00322.xHeeter, C. (1992). Being There: The Subjective Experience of Presence. Presence: Teleoperators and Virtual Environments, 1(2), 262-271. doi:10.1162/pres.1992.1.2.262Sanchez-Vives, M. V., & Slater, M. (2005). From presence to consciousness through virtual reality. Nature Reviews Neuroscience, 6(4), 332-339. doi:10.1038/nrn1651Mohr, D. C., Burns, M. N., Schueller, S. M., Clarke, G., & Klinkman, M. (2013). Behavioral Intervention Technologies: Evidence review and recommendations for future research in mental health. General Hospital Psychiatry, 35(4), 332-338. doi:10.1016/j.genhosppsych.2013.03.008Neguț, A., Matu, S.-A., Sava, F. A., & David, D. (2016). Virtual reality measures in neuropsychological assessment: a meta-analytic review. The Clinical Neuropsychologist, 30(2), 165-184. doi:10.1080/13854046.2016.1144793Riva, G. (2005). Virtual Reality in Psychotherapy: Review. CyberPsychology & Behavior, 8(3), 220-230. doi:10.1089/cpb.2005.8.220Valmaggia, L. R., Latif, L., Kempton, M. J., & Rus-Calafell, M. (2016). Virtual reality in the psychological treatment for mental health problems: An systematic review of recent evidence. Psychiatry Research, 236, 189-195. doi:10.1016/j.psychres.2016.01.015Mesa-Gresa, P., Gil-Gómez, H., Lozano-Quilis, J.-A., & Gil-Gómez, J.-A. (2018). Effectiveness of Virtual Reality for Children and Adolescents with Autism Spectrum Disorder: An Evidence-Based Systematic Review. Sensors, 18(8), 2486. doi:10.3390/s18082486Cheng, Y., & Ye, J. (2010). Exploring the social competence of students with autism spectrum conditions in a collaborative virtual learning environment – The pilot study. Computers & Education, 54(4), 1068-1077. doi:10.1016/j.compedu.2009.10.011Jarrold, W., Mundy, P., Gwaltney, M., Bailenson, J., Hatt, N., McIntyre, N., … Swain, L. (2013). Social Attention in a Virtual Public Speaking Task in Higher Functioning Children With Autism. Autism Research, 6(5), 393-410. doi:10.1002/aur.1302Forgeot d’Arc, B., Ramus, F., Lefebvre, A., Brottier, D., Zalla, T., Moukawane, S., … Delorme, R. (2014). Atypical Social Judgment and Sensitivity to Perceptual Cues in Autism Spectrum Disorders. Journal of Autism and Developmental Disorders, 46(5), 1574-1581. doi:10.1007/s10803-014-2208-5Maskey, M., Lowry, J., Rodgers, J., McConachie, H., & Parr, J. R. (2014). Reducing Specific Phobia/Fear in Young People with Autism Spectrum Disorders (ASDs) through a Virtual Reality Environment Intervention. PLoS ONE, 9(7), e100374. doi:10.1371/journal.pone.0100374Baron-Cohen, S., Ashwin, E., Ashwin, C., Tavassoli, T., & Chakrabarti, B. (2009). Talent in autism: hyper-systemizing, hyper-attention to detail and sensory hypersensitivity. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1522), 1377-1383. doi:10.1098/rstb.2008.0337Tomchek, S. D., Huebner, R. A., & Dunn, W. (2014). Patterns of sensory processing in children with an autism spectrum disorder. Research in Autism Spectrum Disorders, 8(9), 1214-1224. doi:10.1016/j.rasd.2014.06.006Boyd, B. A., Baranek, G. T., Sideris, J., Poe, M. D., Watson, L. R., Patten, E., & Miller, H. (2010). Sensory features and repetitive behaviors in children with autism and developmental delays. Autism Research, n/a-n/a. doi:10.1002/aur.124Gabriels, R. L., Agnew, J. A., Miller, L. J., Gralla, J., Pan, Z., Goldson, E., … Hooks, E. (2008). Is there a relationship between restricted, repetitive, stereotyped behaviors and interests and abnormal sensory response in children with autism spectrum disorders? Research in Autism Spectrum Disorders, 2(4), 660-670. doi:10.1016/j.rasd.2008.02.002Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., & Sheikh, Y. (2021). OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1), 172-186. doi:10.1109/tpami.2019.2929257Schölkopf, B., Smola, A. J., Williamson, R. C., & Bartlett, P. L. (2000). New Support Vector Algorithms. Neural Computation, 12(5), 1207-1245. doi:10.1162/089976600300015565Yan, K., & Zhang, D. (2015). Feature selection and analysis on correlated gas sensor data with recursive feature elimination. Sensors and Actuators B: Chemical, 212, 353-363. doi:10.1016/j.snb.2015.02.025Chang, C.-C., & Lin, C.-J. (2011). LIBSVM. ACM Transactions on Intelligent Systems and Technology, 2(3), 1-27. doi:10.1145/1961189.1961199O’Neill, M., & Jones, R. S. P. (1997). Journal of Autism and Developmental Disorders, 27(3), 283-293. doi:10.1023/a:1025850431170Foss-Feig, J. H., Kwakye, L. D., Cascio, C. J., Burnette, C. P., Kadivar, H., Stone, W. L., & Wallace, M. T. (2010). An extended multisensory temporal binding window in autism spectrum disorders. Experimental Brain Research, 203(2), 381-389. doi:10.1007/s00221-010-2240-4Courchesne, E., Lincoln, A. J., Kilman, B. A., & Galambos, R. (1985). Event-related brain potential correlates of the processing of novel visual and auditory information in autism. Journal of Autism and Developmental Disorders, 15(1), 55-76. doi:10.1007/bf01837899Russo, N., Foxe, J. J., Brandwein, A. B., Altschuler, T., Gomes, H., & Molholm, S. (2010). Multisensory processing in children with autism: high-density electrical mapping of auditory-somatosensory integration. Autism Research, 3(5), 253-267. doi:10.1002/aur.152Ament, K., Mejia, A., Buhlman, R., Erklin, S., Caffo, B., Mostofsky, S., & Wodka, E. (2014). Evidence for Specificity of Motor Impairments in Catching and Balance in Children with Autism. Journal of Autism and Developmental Disorders, 45(3), 742-751. doi:10.1007/s10803-014-2229-

Development and evaluation of an interactive virtual audience for a public speaking training application

Einleitung: Eine der häufigsten sozialen Ängste ist die Angst vor öffentlichem Sprechen. Virtual-Reality- (VR-) Trainingsanwendungen sind ein vielversprechendes Instrument, um die Sprechangst zu reduzieren und die individuellen Sprachfähigkeiten zu verbessern. Grundvoraussetzung hierfür ist die Implementierung eines realistischen und interaktiven Sprecher-Publikum-Verhaltens. Ziel: Die Studie zielte darauf ab, ein realistisches und interaktives Publikum für eine VR-Anwendung zu entwickeln und zu bewerten, welches für die Trainingsanwendung von öffentlichem Sprechen angewendet wird. Zunächst wurde eine Beobachtungsstudie zu den Verhaltensmustern von Sprecher und Publikum durchgeführt. Anschließend wurden die identifizierten Muster in eine VR-Anwendung implementiert. Die Wahrnehmung der implementierten Interaktionsmuster wurde in einer weiteren Studie aus Sicht der Nutzer evaluiert. Beobachtungsstudie (1): Aufgrund der nicht ausreichenden Datengrundlage zum realen interaktiven Verhalten zwischen Sprecher und Publikum lautet die erste Forschungsfrage "Welche Sprecher-Publikums-Interaktionsmuster können im realen Umfeld identifiziert werden?". Es wurde eine strukturierte, nicht teilnehmende, offene Beobachtungsstudie durchgeführt. Ein reales Publikum wurde auf Video aufgezeichnet und die Inhalte analysiert. Die Stichprobe ergab N = 6484 beobachtete Interaktionsmuster. Es wurde festgestellt, dass Sprecher mehr Dialoge als das Publikum initiieren und wie die Zuschauer auf Gesichtsausdrücke und Gesten der Sprecher reagieren. Implementierungsstudie (2): Um effiziente Wege zur Implementierung der Ergebnisse der Beobachtungsstudie in die Trainingsanwendung zu finden, wurde die Forschungsfrage wie folgt formuliert: "Wie können Interaktionsmuster zwischen Sprecher und Publikum in eine virtuelle Anwendung implementiert werden?". Das Hardware-Setup bestand aus einer CAVE, Infitec-Brille und einem ART Head-Tracking. Die Software wurde mit 3D-Excite RTT DeltaGen 12.2 realisiert. Zur Beantwortung der zweiten Forschungsfrage wurden mehrere mögliche technische Lösungen systematisch untersucht, bis effiziente Lösungen gefunden wurden. Infolgedessen wurden die selbst erstellte Audioerkennung, die Kinect-Bewegungserkennung, die Affectiva-Gesichtserkennung und die selbst erstellten Fragen implementiert, um das interaktive Verhalten des Publikums in der Trainingsanwendung für öffentliches Sprechen zu realisieren. Evaluationsstudie (3): Um herauszufinden, ob die Implementierung interaktiver Verhaltensmuster den Erwartungen der Benutzer entsprach, wurde die dritte Forschungsfrage folgendermaßen formuliert: “Wie beeinflusst die Interaktivität einer virtuellen Anwendung für öffentliches Reden die Benutzererfahrung?”. Eine experimentelle Benutzer-Querschnittsstudie wurde mit N = 57 Teilnehmerinnen (65% Männer, 35% Frauen; Durchschnittsalter = 25.98, SD = 4.68) durchgeführt, die entweder der interaktiven oder nicht-interaktiven VR-Anwendung zugewiesen wurden. Die Ergebnisse zeigten, dass, es einen signifikanten Unterschied in der Wahrnehmung zwischen den beiden Anwendungen gab. Allgemeine Schlussfolgerungen: Interaktionsmuster zwischen Sprecher und Publikum, die im wirklichen Leben beobachtet werden können, wurden in eine VR-Anwendung integriert, die Menschen dabei hilft, Angst vor dem öffentlichen Sprechen zu überwinden und ihre öffentlichen Sprechfähigkeiten zu trainieren. Die Ergebnisse zeigten eine hohe Relevanz der VR-Anwendungen für die Simulation öffentlichen Sprechens. Obwohl die Fragen des Publikums manuell gesteuert wurden, konnte das neu gestaltete Publikum mit den Versuchspersonen interagieren. Die vorgestellte VR-Anwendung zeigt daher einen hohen potenziellen Nutzen, Menschen beim Trainieren von Sprechfähigkeiten zu unterstützen. Die Fragen des Publikums wurden immer noch manuell von einem Bediener reguliert und die Studie wurde mit Teilnehmern durchgeführt, die nicht unter einem hohen Grad an Angst vor öffentlichem Sprechen leiden. Bei zukünftigen Studien sollten fortschrittlichere Technologien eingesetzt werden, beispielsweise Spracherkennung, 3D-Aufzeichnungen oder 3D-Livestreams einer realen Person und auch Teilnehmer mit einem hohen Grad an Angst vor öffentlichen Ansprachen beziehungsweise Sprechen in der Öffentlichkeit.Introduction: Fear of public speaking is the most common social fear. Virtual reality (VR) training applications are a promising tool to improve public speaking skills. To be successful, applications should feature a high scenario fidelity. One way to improve it is to implement realistic speaker-audience interactive behavior. Objective: The study aimed to develop and evaluate a realistic and interactive audience for a VR public speaking training application. First, an observation study on real speaker-audience interactive behavior patterns was conducted. Second, identified patterns were implemented in the VR application. Finally, an evaluation study identified users’ perceptions of the training application. Observation Study (1): Because of the lack of data on real speaker-audience interactive behavior, the first research question to be answered was “What speaker-audience interaction patterns can be identified in real life?”. A structured, non-participant, overt observation study was conducted. A real audience was video recorded, and content analyzed. The sample resulted in N = 6,484 observed interaction patterns. It was found that speakers, more often than audience members, initiate dialogues and how audience members react to speakers’ facial expressions and gestures. Implementation Study (2): To find efficient ways of implementing the results of the observation study in the training application, the second research question was formulated as: “How can speaker-audience interaction patterns be implemented into the virtual public speaking application?”. The hardware setup comprised a CAVE, Infitec glasses, and ART head tracking. The software was realized with 3D-Excite RTT DeltaGen 12.2. To answer the second research question, several possible technical solutions were explored systematically, until efficient solutions were found. As a result, self-created audio recognition, Kinect motion recognition, Affectiva facial recognition, and manual question generation were implemented to provide interactive audience behavior in the public speaking training application. Evaluation Study (3): To find out if implementing interactive behavior patterns met users’ expectations, the third research question was formulated as “How does interactivity of a virtual public speaking application affect user experience?”. An experimental, cross-sectional user study was conducted with (N = 57) participants (65% men, 35% women; Mage = 25.98, SD = 4.68) who used either an interactive or a non-interactive VR application condition. Results revealed that there was a significant difference in users’ perception of the two conditions. General Conclusions: Speaker-audience interaction patterns that can be observed in real life were incorporated into a VR application that helps people to overcome the fear of public speaking and train their public speaking skills. The findings showed a high relevance of interactivity for VR public speaking applications. Although questions from the audience were still regulated manually, the newly designed audience could interact with the speakers. Thus, the presented VR application is of potential value in helping people to train their public speaking skills. The questions from the audience were still regulated manually by an operator and we conducted the study with participants not suffering from high degrees of public speaking fear. Future work may use more advanced technology, such as speech recognition, 3D-records, or live 3D-streams of an actual person and include participants with high degrees of public speaking fear

As cinemáticas e narrativas de jogos digitais: implicações para o design de jogos

Game cinematics integrating filming techniques, cutscenes, and animations are likely to have a pivotal role in the player experience, emotional attachment, and relatedness to the game narrative content. In recent years, the avoidance of cuts to generate a seamless and connected game experience has challenged older patterns on storytelling in games and find new ways of intertwining gameplay with the narrative. However, a one-size-fits-all strategy may not be applied when considering different storytelling purposes, structures, and language used in diversified game genres. The purpose of this research is to examine the contributions of cinematics to affect game story comprehension, especially in young adults. A multi-stage development research method is applied, encompassing the following activities: (1) identification of requirements to develop game cinematics based on the literature review and interviews with 16 scholars and industry professionals in Game UX, and cinematic development; (2) Development of a Game Cinematics experiment for the game Mutation Madness, using first- and third- perspectives; and a (3) Comparative evaluation of game cinematics in terms of experience and narrative comprehension. The cinematics developed are tested with 46 young adults to assess the effect of the Point Of View (POV) camera on visual attention and story comprehension in Mutation Madness, using an eye- tracking experiment. The results suggest that third- perspective in game cinematics constitute omniscient knowledge of the story, evoking the sense of time and focusing on the story agents, while first- perspective visually guides the player to the game events happening in the time of gameplay. This research contributes to the Communication Sciences and Technologies field by presenting a set of best practices developing game cinematics.As cinemáticas de jogos que integram técnicas de filmagem, cutscenes, e animações, tendem a ter um papel central na experiência do jogador, envolvimento emocional, e relação com a narrativa. Nos últimos anos, os padrões mais antigos inerentes ao processo de contar histórias nos jogos têm sido desafiados pela ausência de cortes e interrupções na experiência, de modo a garantir a interconexão e o entrelaçar da jogabilidade com a narrativa. No entanto, a adoção de uma estratégia de dimensão única pode não atender a diferentes propósitos da narrativa, estruturas e linguagem utilizadas em diferentes géneros de jogos. O objetivo desta investigação é compreender o modo como as cinemáticas podem afetar a compreensão de histórias de jogos, especialmente em jovens adultos. O método de investigação de desenvolvimento é aplicado, subdividido nas seguintes etapas: (1) Identificação de requisitos para desenvolver cinemáticas de jogos com base na revisão da literatura e entrevista a 16 académicos e profissionais da indústria em experiência de jogo, e desenvolvimento de cinemáticas; (2) Desenvolvimento de cinemáticas do jogo Mutation Madness, com recurso à primeira e terceira perspetiva; e (3) Avaliação comparativa das cinemáticas de jogo em termos de experiência e compreensão da narrativa. As cinemáticas desenvolvidas são testadas por 46 jovens adultos, recorrendo à metodologia eye-tracking, para avaliar o efeito da perspetiva da câmara [Point Of View (POV)] na atenção visual e na compreensão da narrativa do jogo Mutation Madness. Os resultados sugerem que a cinemática em que é adotada a terceira perspetiva no jogo proposto contribui para um conhecimento omnisciente da história, evocando o sentido do tempo e concentrando-se nos agentes da história, enquanto a primeira perspetiva orienta visualmente o jogador para os eventos do jogo que acontecem no tempo da jogabilidade. Esta investigação contribui para a área das Ciências da Tecnologia da Comunicação ao apresentar um conjunto de melhores práticas para desenvolver as cinemáticas nos jogos.Mestrado em Comunicação Multimédi

Investigating User Satisfaction: An Adaptation of IS Success Model for Short-term Use

Research investigating the acceptance of information systems mostly focuses on systems designed for long-term use, rather than one-time or short-term use. However, short-term use systems are part of the health information technology portfolio. We propose a theoretical model inspired by the D&M IS Success Model to investigate user satisfaction, as a measure of acceptance, with a web-based decision aid designed for short-term decision-making. We hypothesize that media richness affects perceived usefulness, perceived ease of use, learnability, information quality, perceived social presence, and trust, which eventually affect user satisfaction. We propose a mixed method to test hypotheses using eye-tracking, surveys, and interviews. As a pilot qualitative study (N=8), the survey outcomes indicated that constructs performed well (mean 7-point Likert scores >= 5.15 and mean system usability scale = 75). The eye-tracking and interview results showed that participants prefer multimedia, and pictures and visual designs drew their attention to the decision aid website

Affective reactions towards socially interactive agents and their computational modeling

Over the past 30 years, researchers have studied human reactions towards machines applying the Computers Are Social Actors paradigm, which contrasts reactions towards computers with reactions towards humans. The last 30 years have also seen improvements in technology that have led to tremendous changes in computer interfaces and the development of Socially Interactive Agents. This raises the question of how humans react to Socially Interactive Agents. To answer these questions, knowledge from several disciplines is required, which is why this interdisciplinary dissertation is positioned within psychology and computer science. It aims to investigate affective reactions to Socially Interactive Agents and how these can be modeled computationally. Therefore, after a general introduction and background, this thesis first provides an overview of the Socially Interactive Agent system used in this work. Second, it presents a study comparing a human and a virtual job interviewer, which shows that both interviewers induce shame in participants to the same extent. Thirdly, it reports on a study investigating obedience towards Socially Interactive Agents. The results indicate that participants obey human and virtual instructors in similar ways. Furthermore, both types of instructors evoke feelings of stress and shame to the same extent. Fourth, a stress management training using biofeedback with a Socially Interactive Agent is presented. The study shows that a virtual trainer can teach coping techniques for emotionally challenging social situations. Fifth, it introduces MARSSI, a computational model of user affect. The evaluation of the model shows that it is possible to relate sequences of social signals to affective reactions, taking into account emotion regulation processes. Finally, the Deep method is proposed as a starting point for deeper computational modeling of internal emotions. The method combines social signals, verbalized introspection information, context information, and theory-driven knowledge. An exemplary application to the emotion shame and a schematic dynamic Bayesian network for its modeling are illustrated. Overall, this thesis provides evidence that human reactions towards Socially Interactive Agents are very similar to those towards humans, and that it is possible to model these reactions computationally.In den letzten 30 Jahren haben Forschende menschliche Reaktionen auf Maschinen untersucht und dabei das “Computer sind soziale Akteure”-Paradigma genutzt, in dem Reaktionen auf Computer mit denen auf Menschen verglichen werden. In den letzten 30 Jahren hat sich ebenfalls die Technologie weiterentwickelt, was zu einer enormen Veränderung der Computerschnittstellen und der Entwicklung von sozial interaktiven Agenten geführt hat. Dies wirft Fragen zu menschlichen Reaktionen auf sozial interaktive Agenten auf. Um diese Fragen zu beantworten, ist Wissen aus mehreren Disziplinen erforderlich, weshalb diese interdisziplinäre Dissertation innerhalb der Psychologie und Informatik angesiedelt ist. Sie zielt darauf ab, affektive Reaktionen auf sozial interaktive Agenten zu untersuchen und zu erforschen, wie diese computational modelliert werden können. Nach einer allgemeinen Einführung in das Thema gibt diese Arbeit daher, erstens, einen Überblick über das Agentensystem, das in der Arbeit verwendet wird. Zweitens wird eine Studie vorgestellt, in der eine menschliche und eine virtuelle Jobinterviewerin miteinander verglichen werden, wobei sich zeigt, dass beide Interviewerinnen bei den Versuchsteilnehmenden Schamgefühle in gleichem Maße auslösen. Drittens wird eine Studie berichtet, in der Gehorsam gegenüber sozial interaktiven Agenten untersucht wird. Die Ergebnisse deuten darauf hin, dass Versuchsteilnehmende sowohl menschlichen als auch virtuellen Anleiterinnen ähnlich gehorchen. Darüber hinaus werden durch beide Instruktorinnen gleiche Maße von Stress und Scham hervorgerufen. Viertens wird ein Biofeedback-Stressmanagementtraining mit einer sozial interaktiven Agentin vorgestellt. Die Studie zeigt, dass die virtuelle Trainerin Techniken zur Bewältigung von emotional herausfordernden sozialen Situationen vermitteln kann. Fünftens wird MARSSI, ein computergestütztes Modell des Nutzeraffekts, vorgestellt. Die Evaluation des Modells zeigt, dass es möglich ist, Sequenzen von sozialen Signalen mit affektiven Reaktionen unter Berücksichtigung von Emotionsregulationsprozessen in Beziehung zu setzen. Als letztes wird die Deep-Methode als Ausgangspunkt für eine tiefer gehende computergestützte Modellierung von internen Emotionen vorgestellt. Die Methode kombiniert soziale Signale, verbalisierte Introspektion, Kontextinformationen und theoriegeleitetes Wissen. Eine beispielhafte Anwendung auf die Emotion Scham und ein schematisches dynamisches Bayes’sches Netz zu deren Modellierung werden dargestellt. Insgesamt liefert diese Arbeit Hinweise darauf, dass menschliche Reaktionen auf sozial interaktive Agenten den Reaktionen auf Menschen sehr ähnlich sind und dass es möglich ist diese menschlichen Reaktion computational zu modellieren.Deutsche Forschungsgesellschaf