The impact of multimodal collaborative virtual environments on learning: A gamified online debate

Online learning platforms are integrated systems designed to provide students and teachers with information, tools and resources to facilitate and enhance the delivery and management of learning. In recent years platform designers have introduced gamification and multimodal interaction as ways to make online courses more engaging and immersive. Current Web-based platforms provide a limited degree of immersion in learning experiences, thereby diminishing potential learning impact. To improve immersion, it is necessary to stimulate some or all the human senses by engaging users in an environment that perceptually surrounds them and allows intuitive and rich interaction with other users and its content. Learning in these collaborative virtual environments (CVEs) can be aided by increasing motivation and engagement through the gamification of the educational task. This rich interaction that combines multimodal stimulation and gamification of the learning experience has the potential to draw students into the learning experience and improve learning outcomes. This paper presents the results of an experimental study designed to evaluate the impact of multimodal real-time interaction on user experience and learning of gamified educational tasks completed in a CVE. Secondary school teachers and students participated in the study. The multimodal CVE is an accurate reconstruction of the European Parliament in Brussels, developed using the REVERIE (Real and Virtual Engagement In Realistic Immersive Environment) framework. In the study, we compared the impact of the VR parliament to a non-multimodal control (an educational platform called Edu-Simulation) for the same educational tasks. Our results show that the multimodal CVE improves student learning performance and aspects of subjective experience when compared to the non-multimodal control. More specifically it resulted in a more positive effect on the ability of the students to generate ideas compared to a non-multimodal control. It also facilitated a sense of presence (strong emotional and a degree of spatial) for students in the VE. The paper concludes with a discussion of future work that focusses on combining the best features of both systems in a hybrid system to increase its educational impact and evaluate the prototype in real-world educational scenarios

Staging Transformations for Multimodal Web Interaction Management

Multimodal interfaces are becoming increasingly ubiquitous with the advent of mobile devices, accessibility considerations, and novel software technologies that combine diverse interaction media. In addition to improving access and delivery capabilities, such interfaces enable flexible and personalized dialogs with websites, much like a conversation between humans. In this paper, we present a software framework for multimodal web interaction management that supports mixed-initiative dialogs between users and websites. A mixed-initiative dialog is one where the user and the website take turns changing the flow of interaction. The framework supports the functional specification and realization of such dialogs using staging transformations -- a theory for representing and reasoning about dialogs based on partial input. It supports multiple interaction interfaces, and offers sessioning, caching, and co-ordination functions through the use of an interaction manager. Two case studies are presented to illustrate the promise of this approach.Comment: Describes framework and software architecture for multimodal web interaction managemen

A Multi-channel Application Framework for Customer Care Service Using Best-First Search Technique

It has become imperative to find a solution to the dissatisfaction in response by mobile service providers when interacting with their customer care centres. Problems faced with Human to Human Interaction (H2H) between customer care centres and their customers include delayed response time, inconsistent solutions to questions or enquires and lack of dedicated access channels for interaction with customer care centres in some cases. This paper presents a framework and development techniques for a multi-channel application providing Human to System (H2S) interaction for customer care centre of a mobile telecommunication provider. The proposed solution is called Interactive Customer Service Agent (ICSA). Based on single-authoring, it will provide three media of interaction with the customer care centre of a mobile telecommunication operator: voice, phone and web browsing. A mathematical search technique called Best-First Search to generate accurate results in a search environmen

Measuring stress and cognitive load effects on the perceived quality of a multimodal dialogue system

In this paper we present the results of a pilot study investigating the impact of stress and cognitive load on the perceived interaction quality of a multimodal dialogue system for crisis management. Four test subjects interacted with the system in four differently configured trials aiming to induce low/high levels of stress and cognitive load. To measure the level of stress and cognitive load physiological sensors and subjective ratings were collected. After each trial the subjects filled in an evaluation questionnaire regarding the system interaction quality. In the end we conducted an in-depth interview with each subject. The trials were recorded with a webcam to facilitate the behaviour analysis. Results showed that both factors have an influence on the way subjects perceived the interaction quality, whereas the cognitive load seems to have a higher impact. Further quantitative experiments are needed in order to validate the results and quantify the weight of each factor. \u

An information assistant system for the prevention of tunnel vision in crisis management

In the crisis management environment, tunnel vision is a set of bias in decision makers’ cognitive process which often leads to incorrect understanding of the real crisis situation, biased perception of information, and improper decisions. The tunnel vision phenomenon is a consequence of both the challenges in the task and the natural limitation in a human being’s cognitive process. An information assistant system is proposed with the purpose of preventing tunnel vision. The system serves as a platform for monitoring the on-going crisis event. All information goes through the system before arrives at the user. The system enhances the data quality, reduces the data quantity and presents the crisis information in a manner that prevents or repairs the user’s cognitive overload. While working with such a system, the users (crisis managers) are expected to be more likely to stay aware of the actual situation, stay open minded to possibilities, and make proper decisions

Multimodal and ubiquitous computing systems: supporting independent-living older users

We document the rationale and design of a multimodal interface to a pervasive/ubiquitous computing system that supports independent living by older people in their own homes. The Millennium Home system involves fitting a resident’s home with sensors – these sensors can be used to trigger sequences of interaction with the resident to warn them about dangerous events, or to check if they need external help. We draw lessons from the design process and conclude the paper with implications for the design of multimodal interfaces to ubiquitous systems developed for the elderly and in healthcare, as well as for more general ubiquitous computing applications

A generic architecture and dialogue model for multimodal interaction

This paper presents a generic architecture and a dialogue model for multimodal interaction. Architecture and model are transparent and have been used for different task domains. In this paper the emphasis is on their use for the navigation task in a virtual environment. The dialogue model is based on the information state approach and the recognition of dialogue acts. We explain how pairs of backward and forward looking tags and the preference rules of the dialogue act determiner together determine the structure of the dialogues that can be handled by the system. The system action selection mechanism and the problem of reference resolution are discussed in detail