Multisensory learning in adaptive interactive systems

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD

Morality Play: A Model for Developing Games of Moral Expertise

According to cognitive psychologists, moral decision-making is a dual-process phenomenon involving two types of cognitive processes: explicit reasoning and implicit intuition. Moral development involves training and integrating both types of cognitive processes through a mix of instruction, practice, and reflection. Serious games are an ideal platform for this kind of moral training, as they provide safe spaces for exploring difficult moral problems and practicing the skills necessary to resolve them. In this article, we present Morality Play, a model for the design of serious games for ethical expertise development based on the Integrative Ethical Education framework from moral psychology and the Lens of the Toy model for serious game design

Navigation Instruction Validation Tool and Indoor Wayfinding Training System for People with Disabilities

According to World Health Survey, there are 785 million (15.6%) people in the world that live with a disability. It is a well-known fact that lack of access to public transportation is a barrier for people with disabilities in seeking work or accessing health care. In this research, we seek to increase access to public transportation by introducing a virtual pre-travel training system that enables people with disabilities to get familiar with a public transportation venue prior to arriving at the venue. Using this system, users establish a mental map of the target environment prior to their arrival to the physical space, increasing their confidence and therefore increasing their chances of using public transportation. First, we have to guarantee that all navigation instructions sent to our training system are correct. Since the number of navigation instruction increases dramatically, instruction validation becomes a challenge. We propose a video game based validation tool which includes a game scene that represents in 2D the physical environment and uses a game avatar to verify the navigation instructions automatically in the game scene. The avatar traverses the virtual space following the corresponding navigation instructions. Only in case that it successfully reaches the planned destination, the current navigation instruction can be considered as correct. Then, we introduce a virtual reality based pre-travel wayfinding training system to assist people with disabilities to get familiar with a venue prior to their arrival at the physical space, which provides two modes: 1) Self-Guided mode in which the path between a source and a destination is shown to the user from third person perspective, and 2) Exploration mode in which the user explores and interacts with the environment. In the end, we have implemented visual analytics tools that track and evaluate trainees’ performance and help us optimize the game. These tools identify the difficulties faced by the trainees as well as obtain overall statistics on the trainees’ behavior in the indoor environment, helping us understand how to modify the system and adjust it to different classes of disabilities

Virtual Reality Games for Motor Rehabilitation

This paper presents a fuzzy logic based method to track user satisfaction without the need for devices to monitor users physiological conditions. User satisfaction is the key to any product’s acceptance; computer applications and video games provide a unique opportunity to provide a tailored environment for each user to better suit their needs. We have implemented a non-adaptive fuzzy logic model of emotion, based on the emotional component of the Fuzzy Logic Adaptive Model of Emotion (FLAME) proposed by El-Nasr, to estimate player emotion in UnrealTournament 2004. In this paper we describe the implementation of this system and present the results of one of several play tests. Our research contradicts the current literature that suggests physiological measurements are needed. We show that it is possible to use a software only method to estimate user emotion

Augmenting the Spatial Perception Capabilities of Users Who Are Blind

People who are blind face a series of challenges and limitations resulting from their lack of being able to see, forcing them to either seek the assistance of a sighted individual or work around the challenge by way of a inefficient adaptation (e.g. following the walls in a room in order to reach a door rather than walking in a straight line to the door). These challenges are directly related to blind users' lack of the spatial perception capabilities normally provided by the human vision system. In order to overcome these spatial perception related challenges, modern technologies can be used to convey spatial perception data through sensory substitution interfaces. This work is the culmination of several projects which address varying spatial perception problems for blind users. First we consider the development of non-visual natural user interfaces for interacting with large displays. This work explores the haptic interaction space in order to find useful and efficient haptic encodings for the spatial layout of items on large displays. Multiple interaction techniques are presented which build on prior research (Folmer et al. 2012), and the efficiency and usability of the most efficient of these encodings is evaluated with blind children. Next we evaluate the use of wearable technology in aiding navigation of blind individuals through large open spaces lacking tactile landmarks used during traditional white cane navigation. We explore the design of a computer vision application with an unobtrusive aural interface to minimize veering of the user while crossing a large open space. Together, these projects represent an exploration into the use of modern technology in augmenting the spatial perception capabilities of blind users

Effectiveness of a Wii balance board-based system (eBaViR) for balance rehabilitation: a pilot randomized clinical trial in patients with acquired brain injury

Background: Acquired brain injury (ABI) is the main cause of death and disability among young adults. In most cases, survivors can experience balance instability, resulting in functional impairments that are associated with diminished health-related quality of life. Traditional rehabilitation therapy may be tedious. This can reduce motivation and adherence to the treatment and thus provide a limited benefit to patients with balance disorders. We present eBaViR (easy Balance Virtual Rehabilitation), a system based on the Nintendo¿ Wii Balance Board¿ (WBB), which has been designed by clinical therapists to improve standing balance in patients with ABI through motivational and adaptative exercises. We hypothesize that eBaViR, is feasible, safe and potentially effective in enhancing standing balance. Methods. In this contribution, we present a randomized and controlled single blinded study to assess the influence of a WBB-based virtual rehabilitation system on balance rehabilitation with ABI hemiparetic patients. This study describes the eBaViR system and evaluates its effectiveness considering 20 one-hour-sessions of virtual reality rehabilitation (n = 9) versus standard rehabilitation (n = 8). Effectiveness was evaluated by means of traditional static and dynamic balance scales. Results: The final sample consisted of 11 men and 6 women. Mean ±SD age was 47.3 ± 17.8 and mean SD chronicity was 570.9 ± 313.2 days. Patients using eBaViR had a significant improvement in static balance (p = 0.011 in Berg Balance Scale and p = 0.011 in Anterior Reaches Test) compared to patients who underwent traditional therapy. Regarding dynamic balance, the results showed significant improvement over time in all these measures, but no significant group effect or group-by-time interaction was detected for any of them, which suggests that both groups improved in the same way. There were no serious adverse events during treatment in either group. Conclusions: The results suggest that eBaViR represents a safe and effective alternative to traditional treatment to improve static balance in the ABI population. These results have encouraged us to reinforce the virtual treatment with new exercises, so an evolution of the system is currently being developed. © 2011 Gil-Gómez et al; licensee BioMed Central Ltd.This study was funded in part by Ministerio de Educacion y Ciencia Spain, Projects Consolider-C (SEJ2006-14301/PSIC), "CIBER of Physiopathology of Obesity and Nutrition, an initiative of ISCIII" and the Excellence Research Program PROMETEO (Generalitat Valenciana. Conselleria de Educacion, 2008-157).Gil-Gómez, J.; Llorens Rodríguez, R.; Alcañiz Raya, ML.; Colomer Font, C. (2011). Effectiveness of a Wii balance board-based system (eBaViR) for balance rehabilitation: a pilot randomized clinical trial in patients with acquired brain injury. Journal of NeuroEngineering and Rehabilitation. 8(30):1-9. https://doi.org/10.1186/1743-0003-8-30S19830Nichols-Larsen, D. S., Clark, P. C., Zeringue, A., Greenspan, A., & Blanton, S. (2005). Factors Influencing Stroke Survivors’ Quality of Life During Subacute Recovery. Stroke, 36(7), 1480-1484. doi:10.1161/01.str.0000170706.13595.4fTeasell, R., Meyer, M. J., McClure, A., Pan, C., Murie-Fernandez, M., Foley, N., & Salter, K. (2009). Stroke Rehabilitation: An International Perspective. Topics in Stroke Rehabilitation, 16(1), 44-56. doi:10.1310/tsr1601-44Sveistrup, H. (2004). Journal of NeuroEngineering and Rehabilitation, 1(1), 10. doi:10.1186/1743-0003-1-10Holden, M. K. (2005). Virtual Environments for Motor Rehabilitation: Review. CyberPsychology & Behavior, 8(3), 187-211. doi:10.1089/cpb.2005.8.187Crosbie, J. H., Lennon, S., Basford, J. R., & McDonough, S. M. (2007). Virtual reality in stroke rehabilitation: Still more virtual than real. Disability and Rehabilitation, 29(14), 1139-1146. doi:10.1080/09638280600960909Haas, B. M., & Burden, A. M. (2000). Validity of weight distribution and sway measurements of the Balance Performance Monitor. Physiotherapy Research International, 5(1), 19-32. doi:10.1002/pri.181Srivastava, A., Taly, A. B., Gupta, A., Kumar, S., & Murali, T. (2009). Post-stroke balance training: Role of force platform with visual feedback technique. Journal of the Neurological Sciences, 287(1-2), 89-93. doi:10.1016/j.jns.2009.08.051Deutsch, J. E., Borbely, M., Filler, J., Huhn, K., & Guarrera-Bowlby, P. (2008). Use of a Low-Cost, Commercially Available Gaming Console (Wii) for Rehabilitation of an Adolescent With Cerebral Palsy. Physical Therapy, 88(10), 1196-1207. doi:10.2522/ptj.20080062Yong Joo, L., Soon Yin, T., Xu, D., Thia, E., Pei Fen, C., Kuah, C., & Kong, K. (2010). A feasibility study using interactive commercial off-the-shelf computer gaming in upper limb rehabilitation in patients after stroke. Journal of Rehabilitation Medicine, 42(5), 437-441. doi:10.2340/16501977-0528Clark, R. A., Bryant, A. L., Pua, Y., McCrory, P., Bennell, K., & Hunt, M. (2010). Validity and reliability of the Nintendo Wii Balance Board for assessment of standing balance. Gait & Posture, 31(3), 307-310. doi:10.1016/j.gaitpost.2009.11.012Young, W., Ferguson, S., Brault, S., & Craig, C. (2011). Assessing and training standing balance in older adults: A novel approach using the ‘Nintendo Wii’ Balance Board. Gait & Posture, 33(2), 303-305. doi:10.1016/j.gaitpost.2010.10.089Shih, C.-H., Shih, C.-T., & Chiang, M.-S. (2010). A new standing posture detector to enable people with multiple disabilities to control environmental stimulation by changing their standing posture through a commercial Wii Balance Board. Research in Developmental Disabilities, 31(1), 281-286. doi:10.1016/j.ridd.2009.09.013Shih, C.-H., Shih, C.-T., & Chu, C.-L. (2010). Assisting people with multiple disabilities actively correct abnormal standing posture with a Nintendo Wii Balance Board through controlling environmental stimulation. Research in Developmental Disabilities, 31(4), 936-942. doi:10.1016/j.ridd.2010.03.004Folstein, M. F., Folstein, S. E., & McHugh, P. R. (1975). «Mini-mental state». Journal of Psychiatric Research, 12(3), 189-198. doi:10.1016/0022-3956(75)90026-6Geurts, A. C. H., de Haart, M., van Nes, I. J. W., & Duysens, J. (2005). A review of standing balance recovery from stroke. Gait & Posture, 22(3), 267-281. doi:10.1016/j.gaitpost.2004.10.002Marsden, J. F. (2005). The vestibular control of balance after stroke. Journal of Neurology, Neurosurgery & Psychiatry, 76(5), 670-679. doi:10.1136/jnnp.2004.046565Perron, M., Malouin, F., & Moffet, H. (2003). Assessing advanced locomotor recovery after total hip arthroplasty with the timed stair test. Clinical Rehabilitation, 17(7), 780-786. doi:10.1191/0269215503cr696oaMcDowell, B. C., Kerr, C., Parkes, J., & Cosgrove, A. (2005). Validity of a 1 minute walk test for children with cerebral palsy. Developmental Medicine & Child Neurology, 47(11), 744. doi:10.1017/s0012162205001568O’Shea, S. D., Taylor, N. F., & Paratz, J. D. (2007). Measuring Muscle Strength for People With Chronic Obstructive Pulmonary Disease: Retest Reliability of Hand-Held Dynamometry. Archives of Physical Medicine and Rehabilitation, 88(1), 32-36. doi:10.1016/j.apmr.2006.10.002Tyson, S. F., Hanley, M., Chillala, J., Selley, A. B., & Tallis, R. C. (2007). The Relationship Between Balance, Disability, and Recovery After Stroke: Predictive Validity of the Brunel Balance Assessment. Neurorehabilitation and Neural Repair, 21(4), 341-346. doi:10.1177/1545968306296966Brooks, D., Davis, A. M., & Naglie, G. (2006). Validity of 3 Physical Performance Measures in Inpatient Geriatric Rehabilitation. Archives of Physical Medicine and Rehabilitation, 87(1), 105-110. doi:10.1016/j.apmr.2005.08.109Jørgensen, H. S., Nakayama, H., Raaschou, H. O., Vive-Larsen, J., Støier, M., & Olsen, T. S. (1995). Outcome and time course of recovery in stroke. Part II: Time course of recovery. The copenhagen stroke study. Archives of Physical Medicine and Rehabilitation, 76(5), 406-412. doi:10.1016/s0003-9993(95)80568-0Ferrarello, F., Baccini, M., Rinaldi, L. A., Cavallini, M. C., Mossello, E., Masotti, G., … Di Bari, M. (2010). Efficacy of physiotherapy interventions late after stroke: a meta-analysis. Journal of Neurology, Neurosurgery & Psychiatry, 82(2), 136-143. doi:10.1136/jnnp.2009.19642

Advances in Human-Robot Interaction

Rapid advances in the field of robotics have made it possible to use robots not just in industrial automation but also in entertainment, rehabilitation, and home service. Since robots will likely affect many aspects of human existence, fundamental questions of human-robot interaction must be formulated and, if at all possible, resolved. Some of these questions are addressed in this collection of papers by leading HRI researchers

Proceedings of the 3rd IUI Workshop on Interacting with Smart Objects

These are the Proceedings of the 3rd IUI Workshop on Interacting with Smart Objects. Objects that we use in our everyday life are expanding their restricted interaction capabilities and provide functionalities that go far beyond their original functionality. They feature computing capabilities and are thus able to capture information, process and store it and interact with their environments, turning them into smart objects