Evaluating humanoid embodied conversational agents in mobile guide applications

Evolution in the area of mobile computing has been phenomenal in the last few years. The exploding increase in hardware power has enabled multimodal mobile interfaces to be developed. These interfaces differ from the traditional graphical user interface (GUI), in that they enable a more “natural” communication with mobile devices, through the use of multiple communication channels (e.g., multi-touch, speech recognition, etc.). As a result, a new generation of applications has emerged that provide human-like assistance in the user interface (e.g., the Siri conversational assistant (Siri Inc., visited 2010)). These conversational agents are currently designed to automate a number of tedious mobile tasks (e.g., to call a taxi), but the possible applications are endless. A domain of particular interest is that of Cultural Heritage, where conversational agents can act as personalized tour guides in, for example, archaeological attractions. The visitors to historical places have a diverse range of information needs. For example, casual visitors have different information needs from those with a deeper interest in an attraction (e.g., - holiday learners versus students). A personalized conversational agent can access a cultural heritage database, and effectively translate data into a natural language form that is adapted to the visitor’s personal needs and interests. The present research aims to investigate the information needs of a specific type of visitors, those for whom retention of cultural content is important (e.g., students of history, cultural experts, history hobbyists, educators, etc.). Embodying a conversational agent enables the agent to use additional modalities to communicate this content (e.g., through facial expressions, deictic gestures, etc.) to the user. Simulating the social norms that guide the real-world human-to-human interaction (e.g., adapting the story based on the reactions of the users), should at least theoretically optimize the cognitive accessibility of the content. Although a number of projects have attempted to build embodied conversational agents (ECAs) for cultural heritage, little is known about their impact on the users’ perceived cognitive accessibility of the cultural heritage content, and the usability of the interfaces they support. In particular, there is a general disagreement on the advantages of multimodal ECAs in terms of users’ task performance and satisfaction over nonanthropomorphised interfaces. Further, little is known about what features influence what aspects of the cognitive accessibility of the content and/or usability of the interface. To address these questions I studied the user experiences with ECA interfaces in six user studies across three countries (Greece, UK and USA). To support these studies, I introduced: a) a conceptual framework based on well-established theoretical models of human cognition, and previous frameworks from the literature. The framework offers a holistic view of the design space of ECA systems b) a research technique for evaluating the cognitive accessibility of ECA-based information presentation systems that combine data from eye tracking and facial expression recognition. In addition, I designed a toolkit, from which I partially developed its natural language processing component, to facilitate rapid development of mobile guide applications using ECAs. Results from these studies provide evidence that an ECA, capable of displaying some of the communication strategies (e.g., non-verbal behaviours to accompany linguistic information etc.) found in the real-world human guidance scenario, is not affecting and effective in enhancing the user’s ability to retain cultural content. The findings from the first two studies, suggest than an ECA has no negative/positive impact on users experiencing content that is similar (but not the same) across different locations (see experiment one, in Chapter 7), and content of variable difficulty (see experiment two, in Chapter 7). However, my results also suggest that improving the degree of content personalization and the quality of the modalities used by the ECA can result in both effective and affecting human-ECA interactions. Effectiveness is the degree to which an ECA facilitates a user in accomplishing the navigation and information tasks. Similarly, affecting is the degree to which the ECA changes the quality of the user’s experience while accomplishing the navigation and information tasks. By adhering to the above rules, I gradually improved my designs and built ECAs that are affecting. In particular, I found that an ECA can affect the quality of the user’s navigation experience (see experiment three in Chapter 7), as well as how a user experiences narrations of cultural value (see experiment five, in Chapter 8). In terms of navigation, I found sound evidence that the strongest impact of the ECAs nonverbal behaviours is on the ability of users to correctly disambiguate the navigation of an ECA instructions provided by a tour guide system. However, my ECAs failed to become effective, and to elicit enhanced navigation or retention performances. Given the positive impact of ECAs on the disambiguation of navigation instructions, the lack of ECA-effectiveness in navigation could be attributed to the simulated mobile conditions. In a real outdoor environment, where users would have to actually walk around the castle, an ECA could have elicited better navigation performance, than a system without it. With regards to retention performance, my results suggest that a designer should not solely consider the impact of an ECA, but also the style and effectiveness of the question-answering (Q&A) with the ECA, and the type of user interacting with the ECA (see experiments four and six, in Chapter 8). I found that that there is a correlation between how many questions participants asked per location for a tour, and the information they retained after the completion of the tour. When participants were requested to ask the systems a specific number of questions per location, they could retain more information than when they were allowed to freely ask questions. However, the constrained style of interaction decreased their overall satisfaction with the systems. Therefore, when enhanced retention performance is needed, a designer should consider strategies that should direct users to ask a specific number of questions per location for a tour. On the other hand, when maintaining the positive levels of user experiences is the desired outcome of an interaction, users should be allowed to freely ask questions. Then, the effectiveness of the Q&A session is of importance to the success/failure of the user’s interaction with the ECA. In a natural-language question-answering system, the system often fails to understand the user’s question and, by default, it asks the user to rephrase again. A problem arises when the system fails to understand a question repeatedly. I found that a repetitive request to rephrase the same question annoys participants and affects their retention performance. Therefore, in order to ensure effective human-ECA Q&A, the repeat messages should be built in a way to allow users to figure out how to ask the system questions to avoid improper responses. Then, I found strong evidence that an ECA may be effective for some type of users, while for some others it may be not. I found that an ECA with an attention-grabbing mechanism (see experiment six, in Chapter 8), had an inverse effect on the retention performance of participants with different gender. In particular, it enhanced the retention performance of the male participants, while it degraded the retention performance of the female participants. Finally, a series of tentative design recommendations for the design of both affecting and effective ECAs in mobile guide applications in derived from the work undertaken. These are aimed at ECA researchers and mobile guide designers

Useful shortcuts: Using design heuristics for consent and permission in smart home devices

Prior research in smart home privacy highlights significant issues with how users understand, permit, and consent to data use. Some of the underlying issues point to unclear data protection regulations, lack of design principles, and dark patterns. In this paper, we explore heuristics (also called “mental shortcuts” or “rules of thumb”) as a means to address security and privacy design challenges in smart homes. First, we systematically analyze an existing body of data on smart homes to derive a set of heuristics for the design of consent and permission. Second, we apply these heuristics in four participatory co-design workshops (n = 14) and report on their use. Third, we analyze the use of the heuristics through thematic analysis highlighting heuristic application, purpose, and effectiveness in successful and unsuccessful design outcomes. We conclude with a discussion of the wider challenges, opportunities, and future work for improving design practices for consent in smart homes

간편 교육용 모바일 앱 구현을 위한 모델 개발: 평가행렬법에 대한 사례 연구

학위논문 (석사)-- 서울대학교 대학원 사범대학 교육학과(교육공학전공), 2017. 8. Cheolil Lim.Mobile devices, especially smartphones, have become one of the most indispensable parts of our lives. The popularity of smartphones and mobile applications has been increasing and mobile technology has been enhanced day by day. Accordingly, educators too have been trying to utilize these technologies for educational purposes. Through the various educational studies on mobile devices, these technologies already have proved how much they are powerful in higher education settings. The unique capabilities of mobile technologies, including connectivity, cameras, sensors, and GPS provide a variety of learning experiences and offer new opportunities for learners both inside and outside the classroom. This study has been progressed as followed. First, the initial Model for Developing Simple Educational Mobile Applications (MODSEMA) has been constructed based on the corresponding literature review. Second, According to the feedbacks derived from the interviews conducted with three experts on MODSEMA, it has been revised. After that, the modified MODSEMA has been validated through a case study in which the app named Evaluation Matrix, used for promoting creativity, has been developed by following it. During and after the implementation of the case study, MODSEMA has been revised whenever is needed, and then, the ultimate MODSEMA has been proposed for educators, researchers and developers. This study has a significance due to the followings. First, the proposed model, MODSEMA, encompasses all development process, provides a detailed guidance on how to apply it and demonstrates its implementation method via a case study. Besides, it serves information on major mobile platforms, development environments, tools and user interface (UI) frameworks, which are continuously changing and developing technologies providing new opportunities and alternatives for developers. In addition, MODSEMA guides on how to evaluate the mobile application depending on its development stage. Second, this study clearly shows that thanks to evolving technology enabling to develop hybrid apps working on multiple mobile platforms, it is possible to develop an educational mobile application which has a high usability including effectiveness, efficiency and satisfaction and which copes with the heterogeneity problem of classrooms. The advantages of hybrid apps are not only that they are available in multiple mobile services including IOS, Android and Windows Phone, but also that they provide high performance and high-quality UI as much as native apps. Lastly, this study also verified that learners want to utilize mobile applications more for educational purposes, a conclusion derived from the analyses of usability questionnaires conducted as a part of the case study.CHAPTER I. INTRODUCTION 1 1. Background of the Study 1 2. Research Questions 4 3. Definition of Terms 5 CHAPTER II. LITERATURE REVIEW 6 1. Mobile Devices for Learning 6 2. Mobile Application Development 8 2.1. Major Mobile Operating Systems (Platforms) 8 2.2. Existing Approaches to Developing Mobile Applications for Educational Purposes 10 2.3. Challenges in Mobile Application Development 14 3. Evaluation Matrix for CPS 16 3.1. Creative Problem Solving (CPS) 17 3.2. Tools used in CPS 18 3.3. Evaluation Matrix 20 CHAPTER III. RESEARCH METHODOLOGY 24 1. Research Method 24 2. Research Procedure 25 CHAPTER IV: RESULTS 29 1. The Initial Model for Developing a Simple Educational Mobile Application (MODSEMA) 29 2. Interview with Experts on the initial MODSEMA 32 3. The Revised MODSEMA 37 3.1. ANALYSIS 37 3.2. DESIGN 65 3.3. DEVELOPMENT 78 3.4. IMPLEMENTATION & EVALUATION 80 4. A Case Study: Evaluation Matrix App 85 4.1. ANALYSIS 85 4.2. DESIGN 92 4.3. DEVELOPMENT 108 4.4. IMPLEMENTATION & EVALUATION 126 5. The Final Model for Developing a Simple Educational Mobile Application (MODSEMA) 139 5.1. Changing the Place of Identify the user interface design requirements for the Mobile App Step 139 5.2. Inserting Determine the database type and its service Step into Analysis Phase 140 5.3. Adding Pilot Test Step into Development Phase 142 5.4. Providing an Iterative Process inside of Development Phase 142 CHAPTER V: DISCUSSION & CONCLUSION 143 1. DISCUSSION 143 1.1. Implications on MODSEMA 144 1.2. Implications on Evaluation Matrix app 147 1.3. Suggestions for App Developers 149 2. CONCLUSION 153 2.1. Summary and Conclusion 153 2.2. Limitations 159 2.3. Future Works 161 REFERENCES 162 APPENDIXES 180Maste

Requirements engineering for explainable systems

Information systems are ubiquitous in modern life and are powered by evermore complex algorithms that are often difficult to understand. Moreover, since systems are part of almost every aspect of human life, the quality in interaction and communication between humans and machines has become increasingly important. Hence the importance of explainability as an essential element of human-machine communication; it has also become an important quality requirement for modern information systems. However, dealing with quality requirements has never been a trivial task. To develop quality systems, software professionals have to understand how to transform abstract quality goals into real-world information system solutions. Requirements engineering provides a structured approach that aids software professionals in better comprehending, evaluating, and operationalizing quality requirements. Explainability has recently regained prominence and been acknowledged and established as a quality requirement; however, there is currently no requirements engineering recommendations specifically focused on explainable systems. To fill this gap, this thesis investigated explainability as a quality requirement and how it relates to the information systems context, with an emphasis on requirements engineering. To this end, this thesis proposes two theories that delineate the role of explainability and establish guidelines for the requirements engineering process of explainable systems. These theories are modeled and shaped through five artifacts. These theories and artifacts should help software professionals 1) to communicate and achieve a shared understanding of the concept of explainability; 2) to comprehend how explainability affects system quality and what role it plays; 3) in translating abstract quality goals into design and evaluation strategies; and 4) to shape the software development process for the development of explainable systems. The theories and artifacts were built and evaluated through literature studies, workshops, interviews, and a case study. The findings show that the knowledge made available helps practitioners understand the idea of explainability better, facilitating the creation of explainable systems. These results suggest that the proposed theories and artifacts are plausible, practical, and serve as a strong starting point for further extensions and improvements in the search for high-quality explainable systems

Usability analysis of contending electronic health record systems

In this paper, we report measured usability of two leading EHR systems during procurement. A total of 18 users participated in paired-usability testing of three scenarios: ordering and managing medications by an outpatient physician, medicine administration by an inpatient nurse and scheduling of appointments by nursing staff. Data for audio, screen capture, satisfaction rating, task success and errors made was collected during testing. We found a clear difference between the systems for percentage of successfully completed tasks, two different satisfaction measures and perceived learnability when looking at the results over all scenarios. We conclude that usability should be evaluated during procurement and the difference in usability between systems could be revealed even with fewer measures than were used in our study. © 2019 American Psychological Association Inc. All rights reserved.Peer reviewe

Instructional Message Design: Theory, Research, and Practice (Volume 2)

Message design is all around us, from the presentations we see in meetings and classes, to the instructions that come with our latest tech gadgets, to multi-million-dollar training simulations. In short, instructional message design is the real-world application of instructional and learning theories to design the tools and technologies used to communicate and effectively convey information. This field of study pulls from many applied sciences including cognitive psychology, industrial design, graphic design, instructional design, information technology, and human performance technology to name just a few. In this book we visit several foundational theories that guide our research, look at different real-world applications, and begin to discuss directions for future best practice. For instance, cognitive load and multimedia learning theories provide best practice, virtual reality and simulations are only a few of the multitude of applications. Special needs learners and designing for online, e-learning, and web conferencing are only some of many applied areas where effective message design can improve outcomes. Studying effective instructional message design tools and techniques has and will continue to be a critical aspect of the overall instructional design process. Hopefully, this book will serve as an introduction to these topics and inspire your curiosity to explore further!https://digitalcommons.odu.edu/distancelearning_books/1003/thumbnail.jp

Instructional Message Design: Theory, Research, and Practice (Volume 2)

Message design is all around us, from the presentations we see in meetings and classes, to the instructions that come with our latest tech gadgets, to multi-million-dollar training simulations. In short, instructional message design is the real-world application of instructional and learning theories to design the tools and technologies used to communicate and effectively convey information. This field of study pulls from many applied sciences including cognitive psychology, industrial design, graphic design, instructional design, information technology, and human performance technology to name just a few. In this book we will visit several foundational theories that guide our research, look at different real-world applications, and begin to discuss directions for future best practice. For instance, cognitive load and multimedia learning theories provide best practice, virtual reality and simulations are only a few of the multitude of applications. Special needs learners and designing for online, e-learning, and web conferencing are only some of many applied areas where effective message design can improve outcomes. Studying effective instructional message design tools and techniques has and will continue to be a critical aspect of the overall instructional design process. Hopefully, this book will serve as an introduction to these topics and inspire your curiosity to explore further

Accessibility of Health Data Representations for Older Adults: Challenges and Opportunities for Design

Health data of consumer off-the-shelf wearable devices is often conveyed to users through visual data representations and analyses. However, this is not always accessible to people with disabilities or older people due to low vision, cognitive impairments or literacy issues. Due to trade-offs between aesthetics predominance or information overload, real-time user feedback may not be conveyed easily from sensor devices through visual cues like graphs and texts. These difficulties may hinder critical data understanding. Additional auditory and tactile feedback can also provide immediate and accessible cues from these wearable devices, but it is necessary to understand existing data representation limitations initially. To avoid higher cognitive and visual overload, auditory and haptic cues can be designed to complement, replace or reinforce visual cues. In this paper, we outline the challenges in existing data representation and the necessary evidence to enhance the accessibility of health information from personal sensing devices used to monitor health parameters such as blood pressure, sleep, activity, heart rate and more. By creating innovative and inclusive user feedback, users will likely want to engage and interact with new devices and their own data

Human-Computer Interaction

In this book the reader will find a collection of 31 papers presenting different facets of Human Computer Interaction, the result of research projects and experiments as well as new approaches to design user interfaces. The book is organized according to the following main topics in a sequential order: new interaction paradigms, multimodality, usability studies on several interaction mechanisms, human factors, universal design and development methodologies and tools