When Does Disengagement Correlate with Performance in Spoken Dialog Computer Tutoring?

In this paper we investigate how student disengagement relates to two performance metrics in a spoken dialog computer tutoring corpus, both when disengagement is measured through manual annotation by a trained human judge, and also when disengagement is measured through automatic annotation by the system based on a machine learning model. First, we investigate whether manually labeled overall disengagement and six different disengagement types are predictive of learning and user satisfaction in the corpus. Our results show that although students’ percentage of overall disengaged turns negatively correlates both with the amount they learn and their user satisfaction, the individual types of disengagement correlate differently: some negatively correlate with learning and user satisfaction, while others don’t correlate with eithermetric at all. Moreover, these relationships change somewhat depending on student prerequisite knowledge level. Furthermore, using multiple disengagement types to predict learning improves predictive power. Overall, these manual label-based results suggest that although adapting to disengagement should improve both student learning and user satisfaction in computer tutoring, maximizing performance requires the system to detect and respond differently based on disengagement type. Next, we present an approach to automatically detecting and responding to user disengagement types based on their differing correlations with correctness. Investigation of ourmachine learningmodel of user disengagement shows that its automatic labels negatively correlate with both performance metrics in the same way as the manual labels. The similarity of the correlations across the manual and automatic labels suggests that the automatic labels are a reasonable substitute for the manual labels. Moreover, the significant negative correlations themselves suggest that redesigning ITSPOKE to automatically detect and respond to disengagement has the potential to remediate disengagement and thereby improve performance, even in the presence of noise introduced by the automatic detection process

Adapting the scheduling of illustrations and graphs to learners in conceptual physics tutoring

This research investigates how to schedule multiple graphical representations in a dialogue-based conceptual physics tutor. Research on multiple graphical representations in tutoring suggests either frequently switching representations or fading from concrete to abstract representations. However, other research communities suggest that the best representation or scheduling can be dependent on various student and tutoring context factors. This thesis investigates whether these factors are important when considering a schedule of representations. Three major hypotheses are investigated. H1: that the best representational format for physics concepts is related to properties of the student and the tutoring context. H2: that it is possible to build models that predict the best representational format using student and tutoring context information. H3: that picking the representational format based upon student and tutoring context information will produce better learning gains than not considering student and tutoring context information. Additionally, this work addresses the question of whether multiple representations produce greater learning gains than a single representation (H4). A first experiment was performed to both investigate H1 and to collect data for H2. ANOVAs showed significant interaction effects in learning between low and high pretesters and between high and low spatial reasoning ability subjects, supporting the first hypothesis. Using the data collected and features describing student and tutoring context information, models were learned to predict when to show illustrations or graphs. That these models could be learned, produce meaningful rules, and outperformed a baseline supports H2. A new modeling algorithm was developed to learn these models by augmenting multiple linear regression to consider certain syntactic constraints. A third study was run to test H3 and H4 and to extrinsically evaluate the adaptive policy learned. One third of subjects had an adaptive scheduling of representations, one third a fixed alternating scheduling, and one third saw only one representation. In support of H3, subjects with high incoming knowledge sometimes perform better when receiving adaptive scheduling over an alternating scheduling, but there are also counter examples. For H4, it is not supported in general: showing only illustrations is best overall, but in some cases some subjects benefit from multiple representations

Affective modelling and feedback in programming practice systems

指導教員：角　

Modélisation des émotions de l’apprenant et interventions implicites pour les systèmes tutoriels intelligents

La modélisation de l’expérience de l’utilisateur dans les Interactions Homme-Machine est un enjeu important pour la conception et le développement des systèmes adaptatifs intelligents. Dans ce contexte, une attention particulière est portée sur les réactions émotionnelles de l’utilisateur, car elles ont une influence capitale sur ses aptitudes cognitives, comme la perception et la prise de décision. La modélisation des émotions est particulièrement pertinente pour les Systèmes Tutoriels Émotionnellement Intelligents (STEI). Ces systèmes cherchent à identifier les émotions de l’apprenant lors des sessions d’apprentissage, et à optimiser son expérience d’interaction en recourant à diverses stratégies d’interventions. Cette thèse vise à améliorer les méthodes de modélisation des émotions et les stratégies émotionnelles utilisées actuellement par les STEI pour agir sur les émotions de l’apprenant. Plus précisément, notre premier objectif a été de proposer une nouvelle méthode pour détecter l’état émotionnel de l’apprenant, en utilisant différentes sources d’informations qui permettent de mesurer les émotions de façon précise, tout en tenant compte des variables individuelles qui peuvent avoir un impact sur la manifestation des émotions. Pour ce faire, nous avons développé une approche multimodale combinant plusieurs mesures physiologiques (activité cérébrale, réactions galvaniques et rythme cardiaque) avec des variables individuelles, pour détecter une émotion très fréquemment observée lors des sessions d’apprentissage, à savoir l’incertitude. Dans un premier lieu, nous avons identifié les indicateurs physiologiques clés qui sont associés à cet état, ainsi que les caractéristiques individuelles qui contribuent à sa manifestation. Puis, nous avons développé des modèles prédictifs permettant de détecter automatiquement cet état à partir des différentes variables analysées, à travers l’entrainement d’algorithmes d’apprentissage machine. Notre deuxième objectif a été de proposer une approche unifiée pour reconnaître simultanément une combinaison de plusieurs émotions, et évaluer explicitement l’impact de ces émotions sur l’expérience d’interaction de l’apprenant. Pour cela, nous avons développé une plateforme hiérarchique, probabiliste et dynamique permettant de suivre les changements émotionnels de l'apprenant au fil du temps, et d’inférer automatiquement la tendance générale qui caractérise son expérience d’interaction à savoir : l’immersion, le blocage ou le décrochage. L’immersion correspond à une expérience optimale : un état dans lequel l'apprenant est complètement concentré et impliqué dans l’activité d’apprentissage. L’état de blocage correspond à une tendance d’interaction non optimale où l'apprenant a de la difficulté à se concentrer. Finalement, le décrochage correspond à un état extrêmement défavorable où l’apprenant n’est plus du tout impliqué dans l’activité d’apprentissage. La plateforme proposée intègre trois modalités de variables diagnostiques permettant d’évaluer l’expérience de l’apprenant à savoir : des variables physiologiques, des variables comportementales, et des mesures de performance, en combinaison avec des variables prédictives qui représentent le contexte courant de l’interaction et les caractéristiques personnelles de l'apprenant. Une étude a été réalisée pour valider notre approche à travers un protocole expérimental permettant de provoquer délibérément les trois tendances ciblées durant l’interaction des apprenants avec différents environnements d’apprentissage. Enfin, notre troisième objectif a été de proposer de nouvelles stratégies pour influencer positivement l’état émotionnel de l’apprenant, sans interrompre la dynamique de la session d’apprentissage. Nous avons à cette fin introduit le concept de stratégies émotionnelles implicites : une nouvelle approche pour agir subtilement sur les émotions de l’apprenant, dans le but d’améliorer son expérience d’apprentissage. Ces stratégies utilisent la perception subliminale, et plus précisément une technique connue sous le nom d’amorçage affectif. Cette technique permet de solliciter inconsciemment les émotions de l’apprenant, à travers la projection d’amorces comportant certaines connotations affectives. Nous avons mis en œuvre une stratégie émotionnelle implicite utilisant une forme particulière d’amorçage affectif à savoir : le conditionnement évaluatif, qui est destiné à améliorer de façon inconsciente l’estime de soi. Une étude expérimentale a été réalisée afin d’évaluer l’impact de cette stratégie sur les réactions émotionnelles et les performances des apprenants.Modeling the user’s experience within Human-Computer Interaction is an important challenge for the design and development of intelligent adaptive systems. In this context, a particular attention is given to the user’s emotional reactions, as they decisively influence his cognitive abilities, such as perception and decision-making. Emotion modeling is particularly relevant for Emotionally Intelligent Tutoring Systems (EITS). These systems seek to identify the learner’s emotions during tutoring sessions, and to optimize his interaction experience using a variety of intervention strategies. This thesis aims to improve current methods on emotion modeling, as well as the emotional strategies that are presently used within EITS to influence the learner’s emotions. More precisely, our first objective was to propose a new method to recognize the learner’s emotional state, using different sources of information that allow to measure emotions accurately, whilst taking account of individual characteristics that can have an impact on the manifestation of emotions. To that end, we have developed a multimodal approach combining several physiological measures (brain activity, galvanic responses and heart rate) with individual variables, to detect a specific emotion, which is frequently observed within computer tutoring, namely : uncertainty. First, we have identified the key physiological indicators that are associated to this state, and the individual characteristics that contribute to its manifestation. Then, we have developed predictive models to automatically detect this state from the analyzed variables, trough machine learning algorithm training. Our second objective was to propose a unified approach to simultaneously recognize a combination of several emotions, and to explicitly evaluate the impact of these emotions on the learner’s interaction experience. For this purpose, we have developed a hierarchical, probabilistic and dynamic framework, which allows one to track the learner’s emotional changes over time, and to automatically infer the trend that characterizes his interaction experience namely : flow, stuck or off-task. Flow is an optimal experience : a state in which the learner is completely focused and involved within the learning activity. The state of stuck is a non-optimal trend of the interaction where the learner has difficulty to maintain focused attention. Finally, the off-task behavior is an extremely unfavorable state where the learner is not involved anymore within the learning session. The proposed framework integrates three-modality diagnostic variables that sense the learner’s experience including : physiology, behavior and performance, in conjunction with predictive variables that represent the current context of the interaction and the learner’s personal characteristics. A human-subject study was conducted to validate our approach through an experimental protocol designed to deliberately elicit the three targeted trends during the learners’ interaction with different learning environments. Finally, our third objective was to propose new strategies to positively influence the learner’s emotional state, without interrupting the dynamics of the learning session. To this end, we have introduced the concept of implicit emotional strategies : a novel approach to subtly impact the learner’s emotions, in order to improve his learning experience. These strategies use the subliminal perception, and more precisely a technique known as affective priming. This technique aims to unconsciously solicit the learner’s emotions, through the projection of primes charged with specific affective connotations. We have implemented an implicit emotional strategy using a particular form of affective priming namely : the evaluative conditioning, which is designed to unconsciously enhance self-esteem. An experimental study was conducted in order to evaluate the impact of this strategy on the learners’ emotional reactions and performance

Social Interactions in Immersive Virtual Environments: People, Agents, and Avatars

Immersive virtual environments (IVEs) have received increased popularity with applications in many fields. IVEs aim to approximate real environments, and to make users react similarly to how they would in everyday life. An important use case is the users-virtual characters (VCs) interaction. We interact with other people every day, hence we expect others to appropriately act and behave, verbally and non-verbally (i.e., pitch, proximity, gaze, turn-taking). These expectations also apply to interactions with VCs in IVEs, and this thesis tackles some of these aspects. We present three projects that inform the area of social interactions with a VC in IVEs, focusing on non-verbal behaviours. In our first study on interactions between people, we collaborated with the Social Neuroscience group at the Institute of Cognitive Neuroscience from UCL on a dyad multi-modal interaction. This aims to understand the conversation dynamics, focusing on gaze and turn-taking. The results show that people have a higher frequency of gaze change (from averted to direct and vice versa) when they are being looked at compared to when they are not. When they are not being looked at, they are also directing their gaze to their partners more compared to when they are being looked at. Another contribution of this work is the automated method of annotating speech and gaze data. Next, we consider agents’ higher-level non-verbal behaviours, covering social attitudes. We present a pipeline to collect data and train a machine learning (ML) model that detects social attitudes in a user-VC interaction. Here we collaborated with two game studios: Dream Reality Interaction and Maze Theory. We present a case study for the ML pipeline on social engagement recognition for the Peaky Blinders narrative VR game from Maze Theory studio. We use a reinforcement learning algorithm with imitation learning rewards and a temporal memory element. The results show that the model trained with raw data does not generalise and performs worse (60% accuracy) than the one trained with socially meaningful data (83% accuracy). In IVEs, people embody avatars and their appearance can impact social interactions. In collaboration with Microsoft Research, we report a longitudinal study in mixed-reality on avatar appearance in real-work meetings between co-workers comparing personalised full-body realistic and cartoon avatars. The results imply that when participants use realistic avatars first, they may have higher expectations and they perceive their colleagues’ emotional states with less accuracy. Participants may also become more accustomed to cartoon avatars as time passes and the overall use of avatars may lead to less accurately perceiving negative emotions. The work presented here contributes towards the field of detecting and generating nonverbal cues for VCs in IVEs. These are also important building blocks for creating autonomous agents for IVEs. Additionally, this work contributes to the games and work industry fields through an immersive ML pipeline for detecting social attitudes and through insights into using different avatar styles over time in real-world meetings

Customer co-creation in innovations : a protocol for innovating with end users

The transition into the information revolution or age has made it possible for consumers and users to interfere in the conceptualization, design, production and sales processes of firms. Consumers and users can express their needs in more direct way to producing firms, they have access to the way products and services are made, and last but not least, have access to information on competing products and services that even producers don’t know about. Consumers have become more knowledgeable and are therefore capable of designing and producing their own products and services. The success of innovations or new product and service development is highly dependent on whether they take in consideration the needs and demands of potential users and consumers. In other words, a market orientation is essential for the success of an innovation. Firms would therefore welcome the idea of consumers and users expressing their demands and probably appreciate consumers who want to participate in the new product or service development, because they would have immediate feedback on the potential success of the innovation. Question is, however, how to achieve this and how to successfully co-create with customers in the innovation process. This design research addresses customer co-creation in innovations for product and service industries. It addresses how firms should successfully activate customers or users and what process they should follow, i.e. the kind of customers or users to involve, the tools and techniques to apply, and procedures to be followed. It develops the appropriate interventions for this in a Customer Co-Creation in Innovations (3CI) - Protocol. The nature of this research is prescriptive, based on the Design Science principles, aiming to design a solution for firms that are interested in the co-creation role that customers can play in their organizations regarding innovations. The research results in a protocol which organizations that want to co-create with customers in their innovation process, can use or apply, to effectively co-create with these customers. Effectively in this sense means that the customer input will be of added value to the innovation, resulting in the outcome that the organization succeeds in bringing the innovation into the market or in use. This doesn’t necessarily mean that the innovation will be a commercial success, because this success depends on more and other factors than just customer co-creation. But, in this context customer co-creation gives the organization the necessary confirmation that the innovation fits needs and demands in the market, and thus leads to a higher adaptation than one should expect when not co-creating with customers. There is an abundance of literature that argue the benefits of involving customers in the innovation process, while other address the issue of which customers to involve, so, the research focuses itself on best practices, experiments, and such to develop this protocol. This has been accomplished by studying the diverse modes or appearances of customer involvement in product or service development, such as market research, empathic design, user-centered design, co-design, mass customization, user innovation, open source software development, user generated content, crowdsourcing, and customer co-creation. Although there is a lot of overlap and similarities among these modes of involvement, there are also many differences, indicating that customer co-creation in innovations is contingent on many factors and aspects. To reduce the confusion, a construct of customer co-creation in innovations has been developed, which has been defined as the process where product manufacturers and/or service providers actively engage with their end users or customers in (parts or phases of) innovation projects to jointly perform innovation activities and co-create value, with the aim of increasing effectiveness and efficiency of the innovation process. Effectiveness refers to (1) the result of meeting users’ and customers’ needs and demands in a better way; and (2) increasing customer loyalty. Efficiency refers to (1) the reduction of research and development costs; and (2) the reduction of development time. And to analyze differences and similarities so that the appropriate design propositions can stated, a 3CI framework was developed, covering the following topics: (1) how to determine whether a firm can co-create with its customers in innovations, which are the so called context conditions; (2) how to identify, select, and motivate potential customers to participate in customer-open innovations; (3) how to engage and involve these customers in the innovation process in an effective and efficient way, the process, procedures and methods one can follow, the tools one can use to accomplish this. With this framework the practice of customer co-creation was analyzed by means of five case studies, in which two of them, the author was an actor in designing and executing the process of co-creation. The cases, selected for their diversity, reveal the opportunities and challenges of customer-inclusive innovation. Customer involvement was at least a partial success in all cases. At the same time, it was never a ‘silver bullet’ to permanently transform the way the company worked. 3CI seems to be capable to support both incidental and repeating innovation initiatives of a firm. Another observation is that, whether a B2B or B2C type of firm, a manufacturer or service provider, small or large firms, all seem to be capable of and suited for 3CI. Common in all cases, however, is that the organization’s offerings and markets should be heterogeneous, thereby containing opportunities to either develop line extensions or really novel (radical) offerings. The technology base of the organization, however, does not seem to be a prerequisite. Another theme cutting across the cases is the nature of an ‘innovation community’, where users test, experiment with and modify or enhance existing prototypes and products, paving the roadway to innovation. As for the relationship between innovation type and type of customer, the cases undoubtedly demonstrate that ‘ordinary’ users can provide useful input to develop radical or novel innovations. The cases also demonstrate that nearly all innovation activities can be conducted by co-creating with customers, including needs assessment, ideation, the screening of ideas or concepts, concept testing, design and development, the commercialization of the innovation and even the re-innovation or use stage. So, although one could get the idea of 3CI being of particular interest in the front end of an innovation stage, we see that in all later stages 3CI can be beneficial as well. Typical across all cases is also the contingency of the channel of involvement (online versus face-to-face) with the amount of customers involved, which we have typed as the degree of openness. The more people are involved, the more open (no secrecy) the co-creation is and the more the involvement is obtained through the online channel, either with communities or on an open call. Conversely, the less participants, the more secrecy is needed and the sooner the physical presence or offline participation seems to be imminent in participation. Finally, regarding the use of tools it can be concluded that sophisticated methods for customer co-creation are a complement rather than the sole source of user information. More important seems to be the occurrence of a dialogue between firm and participating customers, implying that the quality of the interaction depends on mutual trust, appreciation, commitment and equality. Tools that support this dialogue, such as the ZMET¿, OBR, or similar techniques, seem to be important to assure effective and efficient contribution from customers. Subsequently, the design process was conducted, first by defining 16 design requirements for the protocol – subdivided in functional and use requirements, and design restrictions and boundary conditions – followed by the development of the design propositions. A grand total of 28 design propositions have been identified, regarding the context of 3CI (10 propositions), the customer requirements (10 propositions) and process (8 propositions). The context propositions reflected the context decisions to be made, i.e. the appropriate strategy, the suitability of the firm’s market, the initiator for the co-creation (firm or customer), and the type of innovation (incremental vs. radical, open vs. closed mode). Wherever appropriate we have also reviewed the appropriate methods, tools and techniques for the best implementation of the interventions. These are the first decisions the firm has to make when undertaking the 3CI Journey. Only when these decisions are made a next step, i.e. determining which customer requirements are appropriate, can be made. It has been argued that any organization can co-create with its customers in innovations, provided that they adopt and maintain a market oriented strategy, along with the necessary tools, space, freedom and transparency for customers to participate. Customer co-creation leads to at least effective incremental innovations, but when the organization applies Customer Knowledge Methods it increases the chance for an effective radical innovation. If secrecy is required, a closed mode approach of co-creation can be followed, entailing that a minimum amount and diversity of external participants are involved, provided that there is a clear scope of innovation objectives and the market it is intended for. Finally, organizations can either rely on customer-initiated ideas or initiate an innovation itself. In the first approach the organization is recommended to create and maintain a customer community, which can be observed and interacted with to elicit the customers’ ideas. The 10 customer design propositions deal with the type of customers to co-create with in innovations and the available interventions to engage with and maintain involvement from the selected participants. We have argued that all (potential) customers are eligible to participate, as long as they have a certain use experience with the product, service or category of innovation. Only in the case of a radical innovation, the company can choose to add some lead users in order to increase the chance of generating really novel ideas or concepts. To find these lead users, the company can make an appeal on the customer community, since lead users are usually known in communities. In order to benefit in the best way from the participating ordinary and lead users, the company should select them on the basis of their willingness to participate. On top of that, participants should be trained or educated in the tools, techniques and methods that are applied during their involvement. To prevent a decrease of intrinsic motivation with participants, companies have to be very prudent with the promise and administering of financial rewards. Rewards can be given, but preferably unexpected and contingent on task complexity and performance demonstrated by the participant. Depending on the channel of involvement, a minimum of 15 to an undetermined maximum of participants is possible, provided that the company reserves sufficient resources to handle the amount of participants. To our initial 20 design propositions we have added an additional 8 design propositions regarding the process of co-creation. We have seen that all innovation stages are suited to co-create with customers. For the appropriate activities in which these customers can contribute we have developed a table depicting activities and contributions per innovation stage. Co-creation can take place in one, more or all stages; to receive the most benefit, customers should be involved as early as possible in the innovation process. To prevent loss of attention, de-motivation and premature abandonment, we have proposed to change participants with ongoing activities; relying on the same customers in all stages can result in ‘myopic’ results. Both online and offline co-creation are possible, depending on openness, amount of participants and available resources. If participation is online, we recommend applying crowdsourcing methods and techniques, preferably within the customer community. To support an effective communication, we finally proposed to use metaphor or analogy based ‘language’ and to treat the participants as if they were team members. Through scrutinizing and analyzing the 28 design propositions in relation to one another and some pre-defined design requirements, we have identified four main routes – metaphorically named the dreamcatcher, contest, touchstone and employment route – that a company can follow when aiming to co-create with customers in the innovation process. The dreamcatcher route appeals on a user community – existing or yet to be created, preferably online, but with a physical possibility – where existing products, services or platforms are used, reviewed and discussed by customers. The company observes and participates in this discussion through a dialogue, possibly also moderating the community. Opportunities are identified by the company and translated into innovation projects by the company, in which customers again can participate. In the contest route the company can pose users with a specific question or request, a challenge, for which they are expected to think of a solution, of which typically one, or a limited amount of solutions are eligible. The intention is to specifically involve the customer in the front end of the innovation, because the company does not know or is not aware yet of customer needs and wants, or the intended product or service requirements. Customer input is then required in the first stage (Conception), but is not necessary excluded in later stages, where customers can test prototypes, assist in the commercialization and the re-innovation. In the touchstone route the company can decide to co-create with customers in any, arbitrary stage or activity of the innovation process, a sort of a one off. In such a case, the company usually has already identified the opportunities, the innovation project and its goals. Customer co-creation is opportune to verify assumptions, fill in details, and provide additional, not thought of product or service requirements. Of course it is possible to co-create with the customer in more than one activity, but this approach is seen as discrete co-creation activities to support just that particular and specific stage, in which the co-creation is required, usually in the implementation stage and thereafter. Finally, in the employment route the company can integrate one or more (limited amount of) customers in the innovation project, e.g. by temporarily employing them. This approach is of particular interest in idea generation, design and development activities, i.e. the Conception and Implementation stage, but later stages aren’t excluded. We can see this approach applied in customized projects, where it is the intention to create something for a specific set of customers or segment. This can be on request by the customer or because the company has discovered an unfulfilled or unattended set of needs with these customers, e.g. through dreamcatching. To decide which route(s) is or are appropriate we have discussed some premises and considerations – objectives for co-creation, stages and contributions for co-creation, type and openness of innovation – that a company has to assess systematically. Each route was elaborated on, providing preparation steps and do’s and don’ts for an effective and efficient contribution from customers. The four routes are also interrelated and do not exclude one another, but nevertheless provide a company with the optimal approach for 3CI. The 3CI-protocol is therefore a robust, handy guideline for companies to co-create with their customers in innovations. Because of the systematic and rigorous analysis and synthesis of theory and practice, the protocol can be applied in most situations. To test and prove the correctness of this last assertion we validated the design by having it reviewed by some potential users, some experts and some scholars, and to base the conclusion of its validity on the opinions of these reviewers. A total of 25 potential reviewers, both national and international, consisting of product/service developers, co-creation intermediaries, consultants and scholars were approached independently from and ‘blind’ to each other to conduct this review. Ten of them consented in participation; three abandoned the review process prematurely for personal reasons, leaving a total of 7 reviewers that have submitted comments. It was agreed on to enhance the review with a Delphi if responses were very divergent. All reviewers found the protocol useful and helpful for guiding the process of customer co-creation. Comments or critique referred mainly to the readability of the protocol, with the remark that users might lose attention because of the academic reasoning. Some of them provided useful additions to the protocol in order to enhance the readability. Also, suggestions were made to promote the protocol to practice, for instance by publishing it via a community and a management book. The comments did not contain divergent viewpoints on the subject, the design and its content, so the Delphi was left out. Based on these comments and suggestions by the reviewers, we have redesigned the protocol into the 3CI Protocol version 1.0, which can be published as a separate document, detached from this thesis, which all potential users can get hold of and apply without having to acquire a copy of the thesis. We propose to use this protocol to further validate it in practice and giving us feedback on its effectiveness. Our main contribution to research in management and organization has been to develop a comprehensive how-to guideline for practitioners, based on and grounded in a diversity of theory. Therefore, we believe that we have contributed with a design that is applicable in all kind of business and organizational contexts where the interaction with end users is aimed at developing new offerings. However, modesty is also in place, when we observe that this has to be proven, yet. Further research can be aimed at obtaining this proof, while other research could focus on the underlying assumptions, which we named generative mechanisms, of the design

Intrinsic and extrinsic evaluation of an automatic user disengagement detector for an uncertainty-adaptive spoken dialogue system

We present a model for detecting user disengagement during spoken dialogue interactions. Intrinsic evaluation of our model (i.e., with respect to a gold standard) yields results on par with prior work. However, since our goal is immediate implementation in a system that already detects and adapts to user uncertainty, we go further than prior work and present an extrinsic evaluation of our model (i.e., with respect to the real-world task). Correlation analyses show crucially that our automatic disengagement labels correlate with system performance in the same way as the gold standard (manual) labels, while regression analyses show that detecting user disengagement adds value over and above detecting only user uncertainty when modeling performance. Our results suggest that automatically detecting and adapting to user disengagement has the potential to significantly improve performance even in the presence of noise, when compared with only adapting to one affective state or ignoring affect entirely