Evaluating embodied conversational agents in multimodal interfaces

Based on cross-disciplinary approaches to Embodied Conversational Agents, evaluation methods for such human-computer interfaces are structured and presented. An introductory systematisation of evaluation topics from a conversational perspective is followed by an explanation of social-psychological phenomena studied in interaction with Embodied Conversational Agents, and how these can be used for evaluation purposes. Major evaluation concepts and appropriate assessment instruments – established and new ones – are presented, including questionnaires, annotations and log-files. An exemplary evaluation and guidelines provide hands-on information on planning and preparing such endeavours

Effects of color and animation on visual short -term memory in computer-environment learning tasks

The goals of this research study were to gain a better understanding of the effects of color and animation on visual short-term memory (STM) in computer-environment learning tasks. The research methodology was based on an investigative design that attempted to address weaknesses of previous visual STM research (Klauer & Zhao, 2004) to provide information readily applicable to instructional design and technology. A 3 x 2 x 3 totally-within participants, repeated-measures Factorial Analysis of Variance (ANOVA) design was utilized to investigate limitations on visual short-term memory learning processes in computer-learning environments. Participant scores for accuracy of visual learning tasks and participant latency in responses to visual learning tasks were measured. Findings suggest achromatic color information interferes with visual STM learning tasks in specific contexts. Findings also suggest that the use of animations in visual STM learning tasks may interfere with complex visual STM learning. Numerous questions are raised for further research utilizing experimental designs that focus on visual STM learning tasks in contrast to designs that focus solely on identifying visual STM parameters.;Keywords. Computer aided instruction; Computer based instruction; Double dissociations; Dual coding; Multimedia learning theory; Information processing, Instructional design, Phonological loop, Short-term memory, Visual and spatial short-term memory, Visual stimuli, Visuospatial sketch pad, Working memory

Impact of Service Characteristics on Rational and Emotional Components of Information Systems Service Evaluations

Information systems (IS) research and practice have recognized the need to move the IS field to a more service oriented paradigm. This requires a good understanding of how IS services are evaluated and the factors that influence the perceptions of service performance. Measures of IS service quality have provided an insight into the rational/technical factors that influence the evaluation of IS services. Recently, the need for the investigation of additional factors that influence IS service evaluations has been recognized. One such factor that can influence the evaluation of an IS service is the emotional response that the IS service elicits in a recipient. Emotional responses play a major role in building attitudes, beliefs and behavioral intentions. However, IS service research has focused more on the rational aspects of these phenomena while largely ignoring the emotional aspects when explaining IS service evaluations. This research seeks to provide a better understanding of how individuals evaluate IS services by focusing on the salient characteristics of the IS service that can influence these evaluations. To achieve this, the research focuses on two research objectives: (1) to investigate the how the individual components of IS service evaluations – the emotional and rational evaluation components – impact various behaviors associated with the IS service and (2) to investigate how specific, theory driven service characteristics impact the emotional and relational components of IS service evaluation. A controlled experiment is used to investigate IS service evaluations and the characteristics of IS services that influence them. Results suggest that both emotional and rational components of IS service evaluations have significant impacts on behavioral intentions associated with the IS service. Furthermore, findings indicate that while the specificity of service output impacts both the emotional and rational evaluations of the IS service, the complexity of the service task only influences the emotional component by increasing the level of emotional evaluations associated with the service. Proximity between service provider and service recipient was found to have no significant impact on the emotional evaluation of the service

Technology and Testing

From early answer sheets filled in with number 2 pencils, to tests administered by mainframe computers, to assessments wholly constructed by computers, it is clear that technology is changing the field of educational and psychological measurement. The numerous and rapid advances have immediate impact on test creators, assessment professionals, and those who implement and analyze assessments. This comprehensive new volume brings together leading experts on the issues posed by technological applications in testing, with chapters on game-based assessment, testing with simulations, video assessment, computerized test development, large-scale test delivery, model choice, validity, and error issues. Including an overview of existing literature and ground-breaking research, each chapter considers the technological, practical, and ethical considerations of this rapidly-changing area. Ideal for researchers and professionals in testing and assessment, Technology and Testing provides a critical and in-depth look at one of the most pressing topics in educational testing today

Towards an understanding of competing constructions of risk for impact assessment

Since the inception of the Impact Assessment (IA) tool in the late 4567s, a pivotal role has been assigned to science and expert knowledge, in a rational scientific approach to anticipating the environmental effects of proposed projects. Embedded in this philosophy is a realist interpretation of risk, such that measurable properties are allocated that can be determined based on probabilities. The intention is to reduce uncertainty and improve the accuracy of forecasting. Whilst this approach adds value to IA, it has limitations in respect of the human dimensions of risk which influence the process and outcomes of the assessment. This research responds to this problem with an exploration into competing constructions of risk for the IA discipline. I begin this thesis by highlighting how IA has been affected by the passage of risk, over several decades, from the domain of science to a wider public discourse linked to fear and anxiety about living in a “Runaway World” (Giddens, L77L). Relevant in this context, are espoused sustainability principles for IA relating to inclusivity and equity. Underscored by numerous critics are associated challenges, particularly when it comes to incorporating social values and acknowledging the role of power in IA. I propose that these challenges can be linked to interpretations of risk - realist on one hand, and societal, cultural and cognitive on the other. The approach to uncovering the implications of competing constructions of risk for IA relies on the method of critical discourse analysis (CDA), and a Foucauldian notion of discourse linked to power. I describe three distinct theories focusing on the social, psychological, and cultural dimensions of risk. These include risk society theory, the psychometric paradigm and cultural theory. To demonstrate their relevance, each theory is applied in a discourse analysis of three South African case studies: a specialist study for a fuel storage facility, an environmental impact assessment (EIA) for a nuclear power plant, and a strategic environmental assessment (SEA) for proposed hydrofracking for shale gas in the Karoo. The studies highlight limitations to the realist interpretation of risk, particularly in morally and politically contested circumstances. My argument is for a richer understanding of risk for IA, along a continuum which accommodates pluralism. I conclude that alternative risk theories provide deeper insight into social values and power dynamics, with a view to advancing the IA discipline to meet the challenges posed by increasing levels of uncertainty in an everchanging world

Learning from Sequential User Data: Models and Sample-efficient Algorithms

Recent advances in deep learning have made learning representation from ever-growing datasets possible in the domain of vision, natural language processing (NLP), and robotics, among others. However, deep networks are notoriously data-hungry; for example, training language models with attention mechanisms sometimes requires trillions of parameters and tokens. In contrast, we can often access a limited number of samples in many tasks. It is crucial to learn models from these `limited\u27 datasets. Learning with limited datasets can take several forms. In this thesis, we study how to select data samples sequentially such that downstream task performance is maximized. Moreover, we study how to introduce prior knowledge in the deep networks to maximize prediction performance. We focus on four sequential tasks: computerized adaptive testing in psychometrics, sketching in recommender systems, knowledge tracing in computer-assisted education, and career path modeling in the labor market. In the first two tasks, we devise novel sample-efficient algorithms to query a minimal number of sequential samples to improve future predictions. We propose a Bilevel Optimization-Based framework for computerized adaptive testing to learn a data-driven question selection algorithm that improves existing data selection policies. We also tackle the sketching problem in the recommender system, with the task of recommending the next item using a stored subset of prior data samples. In this setting, we develop a data-driven sequential selection algorithm that tackles evolving downstream task distribution. In the last two tasks, we devise novel neural models to introduce prior knowledge exploiting limited data samples. For knowledge tracing, we propose a novel neural architecture, inspired by cognitive and psychometric models, to improve the prediction of students\u27 future performance and utilize the labeled data samples efficiently. For career path modeling, we propose a novel and interpretable monotonic nonlinear state-space model to analyze online user professional profiles and provide actionable feedback and recommendations to users on how they can reach their career goals. The data-driven differentiable data selection algorithms for the first two tasks open up future directions to query (a non-differentiable operation) a minimal number of samples optimally to maximize prediction performance. The structures, introduced in the neural architecture for the models in the last two tasks using prior knowledge, open up future directions to learn deep models augmented with prior knowledge using limited data samples

A Systematic Analysis of Accessibility Education Within Computing Disciplines

Accessible technologies improve the usability for all users, including 1 billion people in the world who have a disability. Although there is a demand for accessible technologies, there is currently no requirement for universities to integrate this content within the computing curriculum. A systematic comparison of teaching efficacy is important to effectively prepare future computing professionals with the skills to create accessible technologies. This dissertation contains a mixed-methods cross-sectional and longitudinal analysis of undergraduate Software Engineering and Information Technology students’ learning of accessibility. Four teaching conditions were assessed at Rochester Institute of Technology: content lectures, projects, exposure to stakeholders with a disability, and collaboration with a team member who had a disability. Evidence of student learning was obtained through questionnaires, project reports, and interview data. Student learning was quantified by a knowledge of programming techniques, awareness of accessible technologies, and attitudes towards individuals with a disability. The cross-sectional analysis spanned three years (spring 2016-2019), fourteen courses, and seven distinct professors. We found that students in all conditions gained an increased knowledge of implementation methods. Students who were exposed to a stakeholder with a disability obtained significantly higher scores in their prosocial sympathetic attitudes, awareness of accessible technologies, and knowledge of programming techniques following the course. Students in the other conditions obtained significant changes in only a subset of these measures. While students in all conditions obtained significantly higher knowledge scores in the short term, only students who had a project or a team member with a disability sustained significantly higher knowledge scores two years after exposure. In interviews, senior-level students revealed that there were multiple factors outside the classroom that dissuaded them from furthering their learning of accessibility. Students mentioned a lack of person-centered topics in major software development processes (e.g., agile, waterfall) and workplace tasks. Without direct reinforcement, students focused on functional software requirements and expressed that accessibility would only be necessary in select front-end development career paths or domains. While current work in computer accessibility education evaluates learning during, or immediately following, one course, this dissertation provides a systematic comparison of student learning throughout multiple courses and instructors. The findings within this dissertation may be used to inform future curriculum plans and educational initiatives

Quality Assessment Methods for Textual Conversational Interfaces: A Multivocal Literature Review

The evaluation and assessment of conversational interfaces is a complex task since such software products are challenging to validate through traditional testing approaches. We conducted a systematic Multivocal Literature Review (MLR), on five different literature sources, to provide a view on quality attributes, evaluation frameworks, and evaluation datasets proposed to provide aid to the researchers and practitioners of the field. We came up with a final pool of 118 contributions, including grey (35) and white literature (83). We categorized 123 different quality attributes and metrics under ten different categories and four macro-categories: Relational, Conversational, User-Centered and Quantitative attributes. While Relational and Conversational attributes are most commonly explored by the scientific literature, we testified a predominance of User-Centered Attributes in industrial literature. We also identified five different academic frameworks/tools to automatically compute sets of metrics, and 28 datasets (subdivided into seven different categories based on the type of data contained) that can produce conversations for the evaluation of conversational interfaces. Our analysis of literature highlights that a high number of qualitative and quantitative attributes are available in the literature to evaluate the performance of conversational interfaces. Our categorization can serve as a valid entry point for researchers and practitioners to select the proper functional and non-functional aspects to be evaluated for their products

Investigation into the impact of the usability factor on the acceptance of mobile transactions:Empirical study in Saudi Arabia

Both the rapid advances in wireless technologies and the high proliferation rate of mobile communication services and artefacts have had a profound impact on industry, and are beginning to offer interesting and advantageous new services. In particular, the mobile transaction (mtransaction) system has emerged, enabling users to pay for physical and digital goods and services using their mobile devices whenever they want, regardless of their location. Although it is anticipated that m-transactions will enjoy a bright future, there is apparently still reluctance among users to accept mobile transactions, particularly in Saudi Arabia. Furthermore, it is believed that usability is critical to the success and acceptance of mobile technology innovations in general, and m-transactions in particular. While most of the research into IT innovation acceptance is based on TAM (Technology Acceptance Model) and its many variants, the literature review reveals that there is little research on mobile application usability as an important antecedent factor for the success and acceptance of mobile technologies. This is particularly true in developing countries, especially the Middle East, therefore this project aims to fill this gap in the research. This paper is an extension of four previous studies by the authors, which indicated that usability is the most influential factor in Saudi users’ intentions of utilising mobile transactions. It aims to investigate and empirically test the usability level of mobile transactions from the perspective of Saudi users; this will be achieved by conducting usability experiments with real mobile applications within the target group. In other words, this study is a part of a larger research project and it serves as a validation/verification tool to answer the question “Why does the ease of use of mobile transactions have the highest impact on the intention to use mobile transactions in Saudi Arabia?” Moreover, it will produce a set of empirically-based recommendations for the developers and providers of mobile applications who are interested in the Saudi market. The results of the this study showed that the usability of the tested application, which is considered one of the most popular commercial mobile application in Saudi Arabia (i.e. ‘Souq.com’), is unsatisfactory, with an average SUS Score of 50. Not surprisingly, this low usability negatively affected the willingness of the participants to use the application again

The Integrated User Experience Evaluation Model: A Systematic Approach To Integrating User Experience Data Sources

Evaluating the user experience (UX) associated with product interaction is a challenge for current human-systems developers. This is largely due to a lack of theoretical guidance for directing how best to assess the UX and a paucity of tools to support such evaluation. This dissertation provided a framework and tools for guiding and supporting evaluation of the user experience. This doctoral research involved reviewing the literature on UX, using this knowledge to build first build a theoretical model of the UX construct and later develop a theoretical model to for the evaluation of UX in order to aid evaluators – the integrated User eXperience EValuation (iUXEV), and empirically validating select components of the model through three case studies. The developed evaluation model was subjected to a three phase validation process that included the development and application of different components of the model separately. The first case study focused on developing a tool and method for assessing the affective component of UX which resulted in lessons learned for the integration of the tool and method into the iUXEV model. The second case study focused on integrating several tools that target different components of UX and resulted in a better understanding of how the data could be utilized as well as identify the need for an integration method to bring the data together. The third case study focused on the application of the results of an usability evaluation on an organizational setting which resulted in the identification of challenges and needs faced by practitioners. Taken together, this body of research, from the theoretically-driven iUXEV model to the newly developed emotional assessment tool, extends the user experience / usability body of knowledge and state-of-practice for interaction design practitioners who are challenged with holistic user experience evaluations, thereby advancing the state-of-the-art in UX design and evaluation