On the Development of Adaptive and User-Centred Interactive Multimodal Interfaces

Multimodal systems have attained increased attention in recent years, which has made possible important improvements in the technologies for recognition, processing, and generation of multimodal information. However, there are still many issues related to multimodality which are not clear, for example, the principles that make it possible to resemble human-human multimodal communication. This chapter focuses on some of the most important challenges that researchers have recently envisioned for future multimodal interfaces. It also describes current efforts to develop intelligent, adaptive, proactive, portable and affective multimodal interfaces

Layered evaluation of interactive adaptive systems : framework and formative methods

Peer reviewedPostprin

User-centred design of flexible hypermedia for a mobile guide: Reflections on the hyperaudio experience

A user-centred design approach involves end-users from the very beginning. Considering users at the early stages compels designers to think in terms of utility and usability and helps develop the system on what is actually needed. This paper discusses the case of HyperAudio, a context-sensitive adaptive and mobile guide to museums developed in the late 90s. User requirements were collected via a survey to understand visitors’ profiles and visit styles in Natural Science museums. The knowledge acquired supported the specification of system requirements, helping defining user model, data structure and adaptive behaviour of the system. User requirements guided the design decisions on what could be implemented by using simple adaptable triggers and what instead needed more sophisticated adaptive techniques, a fundamental choice when all the computation must be done on a PDA. Graphical and interactive environments for developing and testing complex adaptive systems are discussed as a further step towards an iterative design that considers the user interaction a central point. The paper discusses how such an environment allows designers and developers to experiment with different system’s behaviours and to widely test it under realistic conditions by simulation of the actual context evolving over time. The understanding gained in HyperAudio is then considered in the perspective of the developments that followed that first experience: our findings seem still valid despite the passed time

Towards responsive Sensitive Artificial Listeners

This paper describes work in the recently started project SEMAINE, which aims to build a set of Sensitive Artificial Listeners – conversational agents designed to sustain an interaction with a human user despite limited verbal skills, through robust recognition and generation of non-verbal behaviour in real-time, both when the agent is speaking and listening. We report on data collection and on the design of a system architecture in view of real-time responsiveness

Affective learning: improving engagement and enhancing learning with affect-aware feedback

This paper describes the design and ecologically valid evaluation of a learner model that lies at the heart of an intelligent learning environment called iTalk2Learn. A core objective of the learner model is to adapt formative feedback based on students’ affective states. Types of adaptation include what type of formative feedback should be provided and how it should be presented. Two Bayesian networks trained with data gathered in a series of Wizard-of-Oz studies are used for the adaptation process. This paper reports results from a quasi-experimental evaluation, in authentic classroom settings, which compared a version of iTalk2Learn that adapted feedback based on students’ affective states as they were talking aloud with the system (the affect condition) with one that provided feedback based only on the students’ performance (the non-affect condition). Our results suggest that affect-aware support contributes to reducing boredom and off-task behavior, and may have an effect on learning. We discuss the internal and ecological validity of the study, in light of pedagogical considerations that informed the design of the two conditions. Overall, the results of the study have implications both for the design of educational technology and for classroom approaches to teaching, because they highlight the important role that affect-aware modelling plays in the adaptive delivery of formative feedback to support learning

Specification Techniques for Multi-Modal Dialogues in the U-Wish Project

In this paper we describe the development of a specification\ud technique for specifying interactive web-based services. We\ud wanted to design a language that can be a means of\ud communication between designers and developers of interactive services, that makes it easier to develop web-based services fitted to the users and that shortens the pathway from design to implementation. The language, still under development, is based on process algebra and can be\ud connected to the results of task analysis. We have been\ud working on the automatic generation of executable prototypes\ud out of the specifications. In this way the specification\ud language can establish a connection between users, design\ud and implementation. A first version of this language is\ud available as well as prototype tools for executing the specifications. Ideas will be given as to how to make the connection between specifications and task analysis

Conversational affective social robots for ageing and dementia support

Socially assistive robots (SAR) hold significant potential to assist older adults and people with dementia in human engagement and clinical contexts by supporting mental health and independence at home. While SAR research has recently experienced prolific growth, long-term trust, clinical translation and patient benefit remain immature. Affective human-robot interactions are unresolved and the deployment of robots with conversational abilities is fundamental for robustness and humanrobot engagement. In this paper, we review the state of the art within the past two decades, design trends, and current applications of conversational affective SAR for ageing and dementia support. A horizon scanning of AI voice technology for healthcare, including ubiquitous smart speakers, is further introduced to address current gaps inhibiting home use. We discuss the role of user-centred approaches in the design of voice systems, including the capacity to handle communication breakdowns for effective use by target populations. We summarise the state of development in interactions using speech and natural language processing, which forms a baseline for longitudinal health monitoring and cognitive assessment. Drawing from this foundation, we identify open challenges and propose future directions to advance conversational affective social robots for: 1) user engagement, 2) deployment in real-world settings, and 3) clinical translation

Innovative Approach for Renewing Instructional Design Applied in the Context of e-Learning

This research aims to present an overview of an innovative approach for renewing instructional design by offering new form of research in e-learning design. Instructional innovations need to stand back and review the design of innovative components for e-learning system. We postulate in this chapter that the design of learning devices requires the design of complex multifaceted object which supports adaptive learning and lets learners bring into play their knowledge in order to carry out the prescribed tasks. Our approach is centred on the design of this complex object called pedagogical instrument, whose molecules are the smallest collection of the components retaining the properties of that material (according to the teaching intentions). The goal that we have set ourselves is to create the first specification of the micro-instructional engineering design. This specification aims at micro pedagogical technology which needs various types of competencies; our discussion implies that instructional innovation is better likely to succeed if it takes into account the actors, the constraints and the standards required to describe the tools allowing the integration of ICT

A task-driven design model for collaborative AmI systems

Proceedings of the CAISE*06 Workshop on Ubiquitous Mobile Information and Collaboration Systems UMICS '06. Luxemburg, June 5-9, 2006.The proceedings of this workshop also appeared in printed version In T. Latour and M. Petit (eds), Proceedings of Workshops and Doctoral Consortium, The 18th International Conference on Advanced Information Systems Engineering - Trusted Information Systems (CAiSE'06), June 5-9, 2006, Presses Universitaires de Namur, 2006, ISBN 2-87037-525.Also published online by CEUR Workshop Proceedings (CEUR-WS.org, ISSN 1613-0073)Ambient intelligence (AmI) is a promising paradigm for humancentred interaction based on mobile and context-aware computing, natural interfaces and collaborative work. AMENITIES (a conceptual and methodological framework based on task-based models) has been specially devised for collaborative systems and is the starting point for a new design proposal for application to AmI systems. This paper proposes a task-based model for designing collaborative AmI systems, which attempts to gather the computational representation of the concepts involved (tasks, laws, etc.) and the relationships between them in order to develop a complete functional environment in relation with the features of AmI systems (collaborative, context-aware, dynamic, proactive, etc.). The research has been applied to an e-learning environment and is implemented using a blackboard model.This research is partially supported by a Spanish R&D Project TIN2004-03140, Ubiquitous Collaborative Adaptive Training (U-CAT)

Construction and abstraction: contrasting methods of supporting model building in learning science

No description supplie