Dialogue based interfaces for universal access.

Conversation provides an excellent means of communication for almost all people. Consequently, a conversational interface is an excellent mechanism for allowing people to interact with systems. Conversational systems are an active research area, but a wide range of systems can be developed with current technology. More sophisticated interfaces can take considerable effort, but simple interfaces can be developed quite rapidly. This paper gives an introduction to the current state of the art of conversational systems and interfaces. It describes a methodology for developing conversational interfaces and gives an example of an interface for a state benefits web site. The paper discusses how this interface could improve access for a wide range of people, and how further development of this interface would allow a larger range of people to use the system and give them more functionality

Dialogue based interfaces for universal access.

Conversation provides an excellent means of communication for almost all people. Consequently, a conversational interface is an excellent mechanism for allowing people to interact with systems. Conversational systems are an active research area, but a wide range of systems can be developed with current technology. More sophisticated interfaces can take considerable effort, but simple interfaces can be developed quite rapidly. This paper gives an introduction to the current state of the art of conversational systems and interfaces. It describes a methodology for developing conversational interfaces and gives an example of an interface for a state benefits web site. The paper discusses how this interface could improve access for a wide range of people, and how further development of this interface would allow a larger range of people to use the system and give them more functionality

A Framework for Research in Gamified Mobile Guide Applications using Embodied Conversational Agents (ECAs)

Mobile Guides are mobile applications that provide players with local and location-based services (LBS), such as navigation assistance, where and when they need them most. Advances in mobile technologies in recent years have enabled the gamification of these applications, opening up new opportunities to transfer education and culture through game play. However, adding traditional game elements such as PBLs (points, badges, and leaderboards) alone cannot ensure that the intended learning outcomes will be met, as the player’s cognitive resources are shared between the application and the surrounding environment. This distribution of resources prevents players from easily immersing themselves into the educational scenario. Adding artificial conversational characters (ECAs) that simulate the social norms found in real-life human-to-human guide scenarios has the potential to address this problem and improve the player’s experience and learning of cultural narratives [1]. Although significant progress has been made towards creating game-like mobile guides with ECAs ([2], [3]), there is still a lack of a unified framework that enables researchers and practitioners to investigate the potential effects of such applications to players and how to approach the concepts of player experience, cognitive accessibility and usability in this context. This paper presents a theoretically-well supported research framework consisted of four key components: differences in players, different features of the gamified task, aspects of how the ECA looks, sound or behaves and different mobile environments. Furthermore, it provides based on this framework a working definition of what player experience, cognitive accessibility and usability are in the context of game-like mobile guide applications. Finally, a synthesis of the results of six empirical studies conducted within this research framework is discussed and a series of design guidelines for the effective gamification of mobile guide applications using ECAs are presented. Results show that an ECA can positively affect the quality of the player’s experience, but it did not elicit better player retention of cultural narratives and navigation of routes

Spoken conversational search: audio-only interactive information retrieval

Speech-based web search where no keyboard or screens are available to present search engine results is becoming ubiquitous, mainly through the use of mobile devices and intelligent assistants such as Apple's HomePod, Google Home, or Amazon Alexa. Currently, these intelligent assistants do not maintain a lengthy information exchange. They do not track context or present information suitable for an audio-only channel, and do not interact with the user in a multi-turn conversation. Understanding how users would interact with such an audio-only interaction system in multi-turn information seeking dialogues, and what users expect from these new systems, are unexplored in search settings. In particular, the knowledge on how to present search results over an audio-only channel and which interactions take place in this new search paradigm is crucial to incorporate while producing usable systems. Thus, constructing insight into the conversational structure of information seeking processes provides researchers and developers opportunities to build better systems while creating a research agenda and directions for future advancements in Spoken Conversational Search (SCS). Such insight has been identified as crucial in the growing SCS area. At the moment, limited understanding has been acquired for SCS, for example how the components interact, how information should be presented, or how task complexity impacts the interactivity or discourse behaviours. We aim to address these knowledge gaps. This thesis outlines the breadth of SCS and forms a manifesto advancing this highly interactive search paradigm with new research directions including prescriptive notions for implementing identified challenges. We investigate SCS through quantitative and qualitative designs: (i) log and crowdsourcing experiments investigating different interaction and results presentation styles, and (ii) the creation and analysis of the first SCS dataset and annotation schema through designing and conducting an observational study of information seeking dialogues. We propose new research directions and design recommendations based on the triangulation of three different datasets and methods: the log analysis to identify practical challenges and limitations of existing systems while informing our future observational study; the crowdsourcing experiment to validate a new experimental setup for future search engine results presentation investigations; and the observational study to establish the SCS dataset (SCSdata), form the first Spoken Conversational Search Annotation Schema (SCoSAS), and study interaction behaviours for different task complexities. Our principle contributions are based on our observational study for which we developed a novel methodology utilising a qualitative design. We show that existing information seeking models may be insufficient for the new SCS search paradigm because they inadequately capture meta-discourse functions and the system's role as an active agent. Thus, the results indicate that SCS systems have to support the user through discourse functions and be actively involved in the users' search process. This suggests that interactivity between the user and system is necessary to overcome the increased complexity which has been imposed upon the user and system by the constraints of the audio-only communication channel. We then present the first schematic model for SCS which is derived from the SCoSAS through the qualitative analysis of the SCSdata. In addition, we demonstrate the applicability of our dataset by investigating the effect of task complexity on interaction and discourse behaviour. Lastly, we present SCS design recommendations and outline new research directions for SCS. The implications of our work are practical, conceptual, and methodological. The practical implications include the development of the SCSdata, the SCoSAS, and SCS design recommendations. The conceptual implications include the development of a schematic SCS model which identifies the need for increased interactivity and pro-activity to overcome the audio-imposed complexity in SCS. The methodological implications include the development of the crowdsourcing framework, and techniques for developing and analysing SCS datasets. In summary, we believe that our findings can guide researchers and developers to help improve existing interactive systems which are less constrained, such as mobile search, as well as more constrained systems such as SCS systems

The impact of multimodal collaborative virtual environments on learning: A gamified online debate

Online learning platforms are integrated systems designed to provide students and teachers with information, tools and resources to facilitate and enhance the delivery and management of learning. In recent years platform designers have introduced gamification and multimodal interaction as ways to make online courses more engaging and immersive. Current Web-based platforms provide a limited degree of immersion in learning experiences, thereby diminishing potential learning impact. To improve immersion, it is necessary to stimulate some or all the human senses by engaging users in an environment that perceptually surrounds them and allows intuitive and rich interaction with other users and its content. Learning in these collaborative virtual environments (CVEs) can be aided by increasing motivation and engagement through the gamification of the educational task. This rich interaction that combines multimodal stimulation and gamification of the learning experience has the potential to draw students into the learning experience and improve learning outcomes. This paper presents the results of an experimental study designed to evaluate the impact of multimodal real-time interaction on user experience and learning of gamified educational tasks completed in a CVE. Secondary school teachers and students participated in the study. The multimodal CVE is an accurate reconstruction of the European Parliament in Brussels, developed using the REVERIE (Real and Virtual Engagement In Realistic Immersive Environment) framework. In the study, we compared the impact of the VR parliament to a non-multimodal control (an educational platform called Edu-Simulation) for the same educational tasks. Our results show that the multimodal CVE improves student learning performance and aspects of subjective experience when compared to the non-multimodal control. More specifically it resulted in a more positive effect on the ability of the students to generate ideas compared to a non-multimodal control. It also facilitated a sense of presence (strong emotional and a degree of spatial) for students in the VE. The paper concludes with a discussion of future work that focusses on combining the best features of both systems in a hybrid system to increase its educational impact and evaluate the prototype in real-world educational scenarios

Evaluating humanoid embodied conversational agents in mobile guide applications

Evolution in the area of mobile computing has been phenomenal in the last few years. The exploding increase in hardware power has enabled multimodal mobile interfaces to be developed. These interfaces differ from the traditional graphical user interface (GUI), in that they enable a more “natural” communication with mobile devices, through the use of multiple communication channels (e.g., multi-touch, speech recognition, etc.). As a result, a new generation of applications has emerged that provide human-like assistance in the user interface (e.g., the Siri conversational assistant (Siri Inc., visited 2010)). These conversational agents are currently designed to automate a number of tedious mobile tasks (e.g., to call a taxi), but the possible applications are endless. A domain of particular interest is that of Cultural Heritage, where conversational agents can act as personalized tour guides in, for example, archaeological attractions. The visitors to historical places have a diverse range of information needs. For example, casual visitors have different information needs from those with a deeper interest in an attraction (e.g., - holiday learners versus students). A personalized conversational agent can access a cultural heritage database, and effectively translate data into a natural language form that is adapted to the visitor’s personal needs and interests. The present research aims to investigate the information needs of a specific type of visitors, those for whom retention of cultural content is important (e.g., students of history, cultural experts, history hobbyists, educators, etc.). Embodying a conversational agent enables the agent to use additional modalities to communicate this content (e.g., through facial expressions, deictic gestures, etc.) to the user. Simulating the social norms that guide the real-world human-to-human interaction (e.g., adapting the story based on the reactions of the users), should at least theoretically optimize the cognitive accessibility of the content. Although a number of projects have attempted to build embodied conversational agents (ECAs) for cultural heritage, little is known about their impact on the users’ perceived cognitive accessibility of the cultural heritage content, and the usability of the interfaces they support. In particular, there is a general disagreement on the advantages of multimodal ECAs in terms of users’ task performance and satisfaction over nonanthropomorphised interfaces. Further, little is known about what features influence what aspects of the cognitive accessibility of the content and/or usability of the interface. To address these questions I studied the user experiences with ECA interfaces in six user studies across three countries (Greece, UK and USA). To support these studies, I introduced: a) a conceptual framework based on well-established theoretical models of human cognition, and previous frameworks from the literature. The framework offers a holistic view of the design space of ECA systems b) a research technique for evaluating the cognitive accessibility of ECA-based information presentation systems that combine data from eye tracking and facial expression recognition. In addition, I designed a toolkit, from which I partially developed its natural language processing component, to facilitate rapid development of mobile guide applications using ECAs. Results from these studies provide evidence that an ECA, capable of displaying some of the communication strategies (e.g., non-verbal behaviours to accompany linguistic information etc.) found in the real-world human guidance scenario, is not affecting and effective in enhancing the user’s ability to retain cultural content. The findings from the first two studies, suggest than an ECA has no negative/positive impact on users experiencing content that is similar (but not the same) across different locations (see experiment one, in Chapter 7), and content of variable difficulty (see experiment two, in Chapter 7). However, my results also suggest that improving the degree of content personalization and the quality of the modalities used by the ECA can result in both effective and affecting human-ECA interactions. Effectiveness is the degree to which an ECA facilitates a user in accomplishing the navigation and information tasks. Similarly, affecting is the degree to which the ECA changes the quality of the user’s experience while accomplishing the navigation and information tasks. By adhering to the above rules, I gradually improved my designs and built ECAs that are affecting. In particular, I found that an ECA can affect the quality of the user’s navigation experience (see experiment three in Chapter 7), as well as how a user experiences narrations of cultural value (see experiment five, in Chapter 8). In terms of navigation, I found sound evidence that the strongest impact of the ECAs nonverbal behaviours is on the ability of users to correctly disambiguate the navigation of an ECA instructions provided by a tour guide system. However, my ECAs failed to become effective, and to elicit enhanced navigation or retention performances. Given the positive impact of ECAs on the disambiguation of navigation instructions, the lack of ECA-effectiveness in navigation could be attributed to the simulated mobile conditions. In a real outdoor environment, where users would have to actually walk around the castle, an ECA could have elicited better navigation performance, than a system without it. With regards to retention performance, my results suggest that a designer should not solely consider the impact of an ECA, but also the style and effectiveness of the question-answering (Q&A) with the ECA, and the type of user interacting with the ECA (see experiments four and six, in Chapter 8). I found that that there is a correlation between how many questions participants asked per location for a tour, and the information they retained after the completion of the tour. When participants were requested to ask the systems a specific number of questions per location, they could retain more information than when they were allowed to freely ask questions. However, the constrained style of interaction decreased their overall satisfaction with the systems. Therefore, when enhanced retention performance is needed, a designer should consider strategies that should direct users to ask a specific number of questions per location for a tour. On the other hand, when maintaining the positive levels of user experiences is the desired outcome of an interaction, users should be allowed to freely ask questions. Then, the effectiveness of the Q&A session is of importance to the success/failure of the user’s interaction with the ECA. In a natural-language question-answering system, the system often fails to understand the user’s question and, by default, it asks the user to rephrase again. A problem arises when the system fails to understand a question repeatedly. I found that a repetitive request to rephrase the same question annoys participants and affects their retention performance. Therefore, in order to ensure effective human-ECA Q&A, the repeat messages should be built in a way to allow users to figure out how to ask the system questions to avoid improper responses. Then, I found strong evidence that an ECA may be effective for some type of users, while for some others it may be not. I found that an ECA with an attention-grabbing mechanism (see experiment six, in Chapter 8), had an inverse effect on the retention performance of participants with different gender. In particular, it enhanced the retention performance of the male participants, while it degraded the retention performance of the female participants. Finally, a series of tentative design recommendations for the design of both affecting and effective ECAs in mobile guide applications in derived from the work undertaken. These are aimed at ECA researchers and mobile guide designers

Using Virtual Worlds to Identify Multidimensional Student Engagement in High School Foreign Language Learning Classrooms

Virtual world environments have evolved from object-oriented, text-based online games to complex three-dimensional immersive social spaces where the lines between reality and computer-generated begin to blur. Educators use virtual worlds to create engaging three-dimensional learning spaces for students, but the impact of virtual worlds in comparison to the traditional face-to-face counterpart has been uncertain in terms of multidimensional student engagement. Research has a need to determine the impact of virtual worlds on student engagement in comparison to the traditional face-to-face environment. The study examined the effects of virtual world and face-to-face learning environments on high school foreign language students\u27 emotional, cognitive, and behavioral engagement, as well as combined engagement. A two-way MANOVA was used to determine the effect of traditional face-to-face and virtual world learning environments on combined student engagement. A 2 x 2 analysis of covariance was used to determine the effect of traditional face-to-face and virtual world learning environments on emotional student engagement. A 2 x 2 analysis of covariance was also used to determine the effect of traditional face-to-face and virtual world learning environments on cognitive student engagement. A t-test was used to determine the effect of traditional face-to-face and virtual world learning environments on behavioral engagement. The study did not find evidence of overall, cognitive, emotional, or behavioral engagement difference between the two learning environments. The findings indicate the virtual world environment is similar to the traditional face-to-face environment in terms of student engagement. School administrators and teachers can benefit from this research when determining effective means of creating highly engaging learning environments for students. Virtual worlds can be a medium for engaging learning opportunities for students in face-to-face and virtual schools. Additional research in this area is recommended to determine the impact of virtual worlds with different student populations and subject areas