Mixed Initiative Systems for Human-Swarm Interaction: Opportunities and Challenges

Human-swarm interaction (HSI) involves a number of human factors impacting human behaviour throughout the interaction. As the technologies used within HSI advance, it is more tempting to increase the level of swarm autonomy within the interaction to reduce the workload on humans. Yet, the prospective negative effects of high levels of autonomy on human situational awareness can hinder this process. Flexible autonomy aims at trading-off these effects by changing the level of autonomy within the interaction when required; with mixed-initiatives combining human preferences and automation's recommendations to select an appropriate level of autonomy at a certain point of time. However, the effective implementation of mixed-initiative systems raises fundamental questions on how to combine human preferences and automation recommendations, how to realise the selected level of autonomy, and what the future impacts on the cognitive states of a human are. We explore open challenges that hamper the process of developing effective flexible autonomy. We then highlight the potential benefits of using system modelling techniques in HSI by illustrating how they provide HSI designers with an opportunity to evaluate different strategies for assessing the state of the mission and for adapting the level of autonomy within the interaction to maximise mission success metrics.Comment: Author version, accepted at the 2018 IEEE Annual Systems Modelling Conference, Canberra, Australi

Modelling human teaching tactics and strategies for tutoring systems

One of the promises of ITSs and ILEs is that they will teach and assist learning in an intelligent manner. Historically this has tended to mean concentrating on the interface, on the representation of the domain and on the representation of the student’s knowledge. So systems have attempted to provide students with reifications both of what is to be learned and of the learning process, as well as optimally sequencing and adjusting activities, problems and feedback to best help them learn that domain. We now have embodied (and disembodied) teaching agents and computer-based peers, and the field demonstrates a much greater interest in metacognition and in collaborative activities and tools to support that collaboration. Nevertheless the issue of the teaching competence of ITSs and ILEs is still important, as well as the more specific question as to whether systems can and should mimic human teachers. Indeed increasing interest in embodied agents has thrown the spotlight back on how such agents should behave with respect to learners. In the mid 1980s Ohlsson and others offered critiques of ITSs and ILEs in terms of the limited range and adaptability of their teaching actions as compared to the wealth of tactics and strategies employed by human expert teachers. So are we in any better position in modelling teaching than we were in the 80s? Are these criticisms still as valid today as they were then? This paper reviews progress in understanding certain aspects of human expert teaching and in developing tutoring systems that implement those human teaching strategies and tactics. It concentrates particularly on how systems have dealt with student answers and how they have dealt with motivational issues, referring particularly to work carried out at Sussex: for example, on responding effectively to the student’s motivational state, on contingent and Vygotskian inspired teaching strategies and on the plausibility problem. This latter is concerned with whether tactics that are effectively applied by human teachers can be as effective when embodied in machine teachers

Designing Attention-aware Business Intelligence and Analytics Dashboards to Support Task Resumption

External interruptions are a common phenomenon in today’s working environment. Specifically, attentional shifts in working environments lead to task resumption failures that refer to the improper resuming of a primary task after an interruption and negatively influencing the individual performance of employees. Business Intelligence & Analytics (BI&A) systems are well recognized as an essential concept to support decision making of employees. One important and frequently used BI&A system component are dashboards. BI&A dashboards enable collecting, summarizing, and presenting business information from different resources to decision makers. When working with BI&A dashboards, interruptions and resulting task resumption failures have negative consequences on decision-making processes. This research in progress paper addresses this problem and provides design knowledge for attention-aware BI&A dashboards that support users during task resumption. We follow a Design Science Research (DSR) approach and derive theory-grounded design principles for task resumption support on BI&A dashboards. Moreover, to evaluate the suggested principles, an instantiation is realized. In our instantiation, real-time tracking of eye-movement data is used to capture visual attention of the users and provide visual feedback after task resumption. We introduce testable hypotheses and present preliminary results of a pre-test lab experiment

An Empirical Study Comparing Unobtrusive Physiological Sensors for Stress Detection in Computer Work.

Several unobtrusive sensors have been tested in studies to capture physiological reactions to stress in workplace settings. Lab studies tend to focus on assessing sensors during a specific computer task, while in situ studies tend to offer a generalized view of sensors' efficacy for workplace stress monitoring, without discriminating different tasks. Given the variation in workplace computer activities, this study investigates the efficacy of unobtrusive sensors for stress measurement across a variety of tasks. We present a comparison of five physiological measurements obtained in a lab experiment, where participants completed six different computer tasks, while we measured their stress levels using a chest-band (ECG, respiration), a wristband (PPG and EDA), and an emerging thermal imaging method (perinasal perspiration). We found that thermal imaging can detect increased stress for most participants across all tasks, while wrist and chest sensors were less generalizable across tasks and participants. We summarize the costs and benefits of each sensor stream, and show how some computer use scenarios present usability and reliability challenges for stress monitoring with certain physiological sensors. We provide recommendations for researchers and system builders for measuring stress with physiological sensors during workplace computer use

Designing intelligent support for learning from and in everyday contexts

Motivation and engagement in learning benefit from a good match of learning settings and materials to individual learner contexts. This includes intrinsic context factors such as prior knowledge and personal interests but also extrinsic factors such as the current environment. Recent developments in adaptive and intelligent technology enable the personalisation of context-aware learning. For example, computer vision algorithms, machine translation, and Augmented Reality make it possible to support the creation of meaningful connections between learners and their context. However, for successful adoption in everyday life, these technologies also need to consider the learner experience. This thesis investigates the design of personalised context-aware learning experiences through the lens of ubiquitous and self-directed language learning as a multi-faceted learning domain. Specifically, it presents and discusses the design, implementation, and evaluation of technology support for learning in and from learners’ everyday contexts with a strong focus on the learner perspective and user experience. The work is guided by four different roles that technology can take on in context-aware ubiquitous learning: For enhancing learning situations, it can (1) sense and (2) trigger in learners’ everyday contexts. For enhancing learning contents, it can (3) augment activities and (4) generate learning material from learner everyday contexts. With regards to the sensing role, the thesis investigates how learners typically use mobile learning apps in everyday contexts. Activity and context logging, combined with experience sampling, confirm that mobile learning sessions spread across the day and occur in different settings. However, they are typically short and frequently interrupted. This indicates that learners may benefit from better integrating learning into everyday contexts, e.g. by supporting task resumption. Subsequently, we explore how this integration could be supported with intelligent triggers linked to opportune moments for learning. We conceptualise and evaluate different trigger types based on interaction patterns and context detection. Our findings show that simple interactions (e.g. plugging in headphones) are promising for capturing both availability and willingness to engage in a learning activity. We discuss how similar interaction triggers could be adapted to match individual habits. In the area of enhancing learning contents, we first investigate how enjoyable everyday activities could be augmented for learning without disrupting these activities. Specifically, we assess the learner experience with interactive grammar support in e-readers and adapted captions for audio-visual media. Participants in our studies felt that the learning augmentations successfully supported their learning process. The information load of the learning support should match the learners’ current needs to maintain the activity flow. Learners may need encouragement to opt for novel concepts optimised for learning (e.g. time-synchronised captions) rather than sticking to habits (e.g. standard captions). Next, the thesis explores learner needs and preferences in generating their own personalised learning material from their context. We design and evaluate automated content generation methods that generate learning opportunities from objects in the learner’s environment. The connection to the learner’s context is established with state-of-the-art technology, such as object detection and Augmented Reality. Through several user studies, we show that learning performance and engagement with auto-generated personalised learning material is comparable to predefined and manually generated content. Findings further indicate that the success of personalisation depends on the effort required to generate content and whether the generation results match the learner’s expectations. Through the different perspectives examined in this thesis, we provide new insights into challenges and opportunities that we synthesise in a framework for context-aware ubiquitous learning technology. The findings also have more general implications for the interaction design of personalised and context-aware intelligent systems. Notably, for the auto-generation of personalised content, it is essential to consider not only correctness from a technological perspective but also how users may perceive the results.Lernmotivation und Engagement profitieren davon, wenn Lernumgebungen und Lernmaterialien auf den individuellen Kontext der Lernenden abgestimmt sind. Dieser umfasst sowohl intrinsische Faktoren wie Vorkenntnisse und persönliche Interessen, aber auch extrinsische Faktoren wie die aktuelle Umgebung. Aktuelle Weiterentwicklungen im Bereich adaptiver und intelligenter Technologien ermöglichen es, Lernen kontextbewusst zu personalisieren. So können mithilfe von Computer-Vision-Algorithmen, maschineller Übersetzung und Augmented Reality sinnvolle Verknüpfungen zwischen Lernenden und ihrem Kontext geschaffen werden. Allerdings müssen diese Technologien für einen erfolgreichen Einsatz im Alltag auch die Lernerfahrung mit einbeziehen. Diese Arbeit untersucht die Gestaltung personalisierter kontextbewusster Lernerfahrungen aus der Perspektive des ubiquitären und self-directed Learning im Sprachenlernen, einem vielseitigen Lernbereich. Insbesondere wird die Konzeption, Implementierung und Evaluierung von Technologieunterstützung für das Sprachenlernen in und aus dem Alltagskontext der Lernenden vorgestellt und diskutiert, wobei der Schwerpunkt auf der Perspektive der Lernenden und der Nutzererfahrung liegt. Die Arbeit orientiert sich an vier verschiedenen Rollen, die Technologie im kontextbewussten Lernen einnehmen kann. Um Lernsituationen anzureichern, kann Technologie im Alltagskontext von Lernenden (1) erfassen und (2) auslösen. Um Lerninhalte anzureichern, kann Technologie aus dem Alltagskontext (3) Aktivitäten augmentieren und (4) Inhalte generieren. Im Hinblick auf die erfassende Rolle von Technologie wird in dieser Arbeit untersucht, wie die Lernenden mobile Lern-Apps in alltäglichen Kontexten nutzen. Die Aufzeichnung von Aktivitäten und Kontexten in Kombination mit Experience Sampling bestätigt, dass Lerneinheiten im mobilen Lernen über den Tag verteilt sind und in verschiedenen Umgebungen stattfinden. Allerdings sind sie in der Regel kurz und werden häufig unterbrochen. Dies deutet darauf hin, dass die Lernenden von einer besseren Integration des Lernens in ihren Alltagskontext profitieren könnten, z. B. durch Unterstützung des Wiedereinstiegs nach einer Unterbrechung. Anschließend untersuchen wir, wie diese Integration durch intelligente Trigger unterstützt werden könnte, die mit passenden Lernzeitpunkten verknüpft sind. Wir konzipieren und evaluieren verschiedene Arten von Triggern auf Basis von Interaktionsmustern und Kontexterkennung. Unsere Ergebnisse zeigen, dass einfache Interaktionen (z. B. das Einstecken von Kopfhörern) vielversprechend dafür sind, sowohl die Verfügbarkeit als auch die Bereitschaft für eine Lernaktivität zu erfassen. Wir diskutieren, wie ähnliche Interaktionstrigger an individuelle Gewohnheiten angepasst werden können. Im Bereich der Augmentierung von Lerninhalten untersuchen wir zunächst, wie unterhaltsame Alltagsaktivitäten für das Lernen aufbereitet werden können, ohne diese Aktivitäten zu beeinträchtigen. Konkret bewerten wir die Lernerfahrung mit interaktiver Grammatikunterstützung in E-Readern und angepassten Untertiteln für audiovisuelle Medien. Die Teilnehmer:innen unserer Studien fanden, dass die Lernunterstützung ihren Lernprozess erfolgreich förderte. Die Informationslast im Lernsystem sollte auf die aktuellen Bedürfnisse der Lernenden angepasst werden, damit das Flow-Erlebnis nicht beeinträchtigt wird. Die Lernenden brauchen möglicherweise Ermutigung dafür, sich für neuartige, lernoptimierte Konzepte zu entscheiden (z. B. zeitsynchrone Untertitel), anstatt an Gewohnheiten festzuhalten (z. B. Standarduntertitel). Als Nächstes werden in dieser Arbeit die Bedürfnisse und Präferenzen der Lernenden bei der Erstellung ihres eigenen personalisierten Lernmaterials aus ihrem Kontext untersucht. Insbesondere werden Methoden zur automatischen Generierung von Inhalten entwickelt und evaluiert, die Lernmöglichkeiten aus Objekten in der Umgebung des Lernenden generieren. Die Verbindung zum Kontext des Lernenden wird durch aktuelle Technologien wie Objekterkennung und Augmented Reality hergestellt. Wir zeigen anhand mehrerer Nutzerstudien, dass die Lernleistung und das Engagement bei automatisch personalisiertem Lernmaterial mit vordefinierten und manuell erstellten Inhalten vergleichbar sind. Die Ergebnisse zeigen außerdem, dass der Erfolg der Personalisierung vom Aufwand abhängt, der für die Erstellung der Inhalte erforderlich ist, und davon, ob die generierten Materialien den Erwartungen der Lernenden entsprechen. Die verschiedenen Perspektiven, die in dieser Arbeit untersucht werden, bieten neue Einblicke in Herausforderungen und Möglichkeiten, die wir in einem Framework für kontextbewusste ubiquitäre Lerntechnologie zusammenfassen. Die Ergebnisse haben auch allgemeinere Auswirkungen auf die Gestaltung der Interaktion mit personalisierten und kontextbewussten intelligenten Systemen. Beispielsweise ist es bei der automatischen Generierung personalisierter Inhalte wichtig, nicht nur die Korrektheit aus technologischer Sicht zu berücksichtigen, sondern auch, wie die Nutzer die Ergebnisse wahrnehmen

Communicative patterns in organizational (healthcare) teams

As team processes are often consigned to a ‘black box’, this dissertation contributes to unpacking team-level communicative processes as drivers for organizational team functioning. Both conceptually and empirically, we aimed to untangle how communicative processes unfold during collaboration periods in organizational teams. In the first part of this dissertation, we contributed to a more process-oriented understanding of team-based collective intelligence. In addition, we developed a comprehensive framework to study communicative patterns from various aspects (i.e., content, structure, and temporality) and showed how patterned communication may relate to team and organizational-level outcomes. In our own empirical work, we found fine-grained evidence for more back-and-forth communicative patterns underlying the decision-making process in multidisciplinary healthcare team meetings, which seems to be rooted in insufficient orientation of the patients’ background problems. In addition, we observed that team members respond with emotionally laden communication after naturally occurring workflow interruptions, together with more conversational clarification. In sum, both scholars and practitioners benefit from understanding patterned communication because these insights offer sound foundations to reflect on improvements regarding organizational team functioning