Temporal Models for History-Aware Explainability

On one hand, there has been a growing interest towards the application of AI-based learning and evolutionary programming for self-adaptation under uncertainty. On the other hand, self-explanation is one of the self-* properties that has been neglected. This is paradoxical as self-explanation is inevitably needed when using such techniques. In this paper, we argue that a self-adaptive autonomous system (SAS) needs an infrastructure and capabilities to be able to look at its own history to explain and reason why the system has reached its current state. The infrastructure and capabilities need to be built based on the right conceptual models in such a way that the system's history can be stored, queried to be used in the context of the decision-making algorithms. The explanation capabilities are framed in four incremental levels, from forensic self-explanation to automated history-aware (HA) systems. Incremental capabilities imply that capabilities at Level n should be available for capabilities at Level n + 1. We demonstrate our current reassuring results related to Level 1 and Level 2, using temporal graph-based models. Specifically, we explain how Level 1 supports forensic accounting after the system's execution. We also present how to enable on-line historical analyses while the self-adaptive system is running, underpinned by the capabilities provided by Level 2. An architecture which allows recording of temporal data that can be queried to explain behaviour has been presented, and the overheads that would be imposed by live analysis are discussed. Future research opportunities are envisioned

Supporting Situation Awareness and Workspace Awareness in Co-located Collaborative Systems Involving Dynamic Data

Co-located technologies can provide digital functionality to support collaborative work for multiple users in the same physical space. For example, digital tabletop computers — large interactive tables that allow users to directly interact with the content — can provide the most up-to-date map information while users can work together face-to-face. Combinations of interactive devices, large and small, can also be used together in a multi-device environment to support collaborative work of large groups. This environment allows individuals to utilize different networked devices. In some co-located group work, integrating automation into the available technologies can provide benefits such as automatically switching between different data views or updating map information based on underlying changes in deployed field agents’ locations. However, dynamic changes in the system state can create confusion for users and lead to low situation awareness. Furthermore, with the large size of a tabletop system or with multiple devices being used in the workspace, users may not be able to observe collaborators’ actions due to physical separations between users. Consequently, workspace awareness — knowledge of collaborators’ up-to-the-moment actions — can be difficult to maintain. As a result, users may be frustrated, and the collaboration may become inefficient or ineffective. The current tabletop applications involving dynamic data focus on interaction and information sharing techniques for collaboration rather than providing situation awareness support. Moreover, the situation awareness literature focuses primarily on single-user applications, whereas, the literature in workspace awareness primarily focuses on remote collaborative work. The aim of this dissertation was in supporting situation awareness of system-automated dynamic changes and workspace awareness of collaborators’ actions. The first study (Timeline Study) presented in this dissertation used tabletop systems to investigate supporting situation awareness of automated changes and workspace awareness, and the second study (Callout Bubble Study) followed up to further investigate workspace awareness support in the context of multi-device classrooms. Digital tabletop computers are increasingly being used for complex domains involving dynamic data, such as coastal surveillance and emergency response. Maintaining situation awareness of these changes driven by the system is crucial for quick and appropriate response when problems arise. However, distractors in the environment can make users miss the changes and negatively impact their situation awareness, e.g., the large size of the table and conversations with team members. As interactive event timelines have been shown to improve response time and decision accuracy after interruptions, in this dissertation they were adapted to the context of collaborative tabletop applications to address the lack of situation awareness due to dynamic changes. A user study was conducted to understand design factors related to the adaption and their impacts on situation awareness and workspace awareness. The Callout Bubble Study investigated workspace awareness support for multi-device classrooms, where students were co-located with their personal devices and were connected through a large shared virtual canvas. This context was chosen due to the environment’s ability to support work in large groups and the increasing prevalence of individual devices in co-located collaborative workspaces. By studying another co-located context, this research also sought to combine the lessons learned and provide a set of more generalized design recommendations for co-located technologies. Existing work on workspace awareness focuses on remote collaboration; however, the co-located users may not need all the information beneficial for remote work. This study aimed to balance awareness and distraction to improve students’ workspace awareness maintenance while minimizing distraction to their learning. A Callout Bubble was designed to augment students’ interactions in the shared online workspace, and a field study was conducted to understand how it impacted the students’ collaboration behaviour. Overall, the research presented in this dissertation aimed to investigate information visualizations for supporting situation awareness and workspace awareness in co-located collaborative environments. The contributions included the design of an interactive event timeline and an investigation of how the control placement (how many timelines and where they should be located) and feedback location (whether to display feedback to the group or to individuals when users interact with timelines) factors affected situation awareness. The empirical results revealed that individual timelines were more effective in facilitating situation awareness maintenance and the timelines were used mainly for perceiving new changes. Furthermore, this dissertation contributed in the design of a workspace awareness cue, Callout Bubble. The field study revealed that Callout Bubbles were effective in improving students’ coordination and self-monitoring behaviours, which in turn reduced teachers’ workloads. The dissertation provided overall design lessons learned for supporting awareness in co-located collaborative environments

Visual Event Cueing in Linked Spatiotemporal Data

abstract: The media disperses a large amount of information daily pertaining to political events social movements, and societal conflicts. Media pertaining to these topics, no matter the format of publication used, are framed a particular way. Framing is used not for just guiding audiences to desired beliefs, but also to fuel societal change or legitimize/delegitimize social movements. For this reason, tools that can help to clarify when changes in social discourse occur and identify their causes are of great use. This thesis presents a visual analytics framework that allows for the exploration and visualization of changes that occur in social climate with respect to space and time. Focusing on the links between data from the Armed Conflict Location and Event Data Project (ACLED) and a streaming RSS news data set, users can be cued into interesting events enabling them to form and explore hypothesis. This visual analytics framework also focuses on improving intervention detection, allowing users to hypothesize about correlations between events and happiness levels, and supports collaborative analysis.Dissertation/ThesisMasters Thesis Computer Science 201

Adventures in the Not Quite Yet: using performance techniques to raise design awareness about digital networks

Technologists promise a future in which pervasive, distributed networks enable radical change to social and political geographies. Design of these abstract, intangible futures is difficult and carries a special risk of excluding people who are not equipped to appreciate the ramifications of these technological changes. The Democratising Technology (DemTech) project has been exploring how techniques from performance and live art can be used to help people engage with the potential of ubiquitous digital networks; in particular, how these techniques can be used to enfranchise people with little technical knowledge, but who nonetheless will have to live with the design consequences of technical decisions. This paper describes the iterative development of a performance workshop for use by designers and community workers. These workshops employ a series of simple exercises to emulate possible processes of technological appropriation: turning abstract digital networks into imaginable, meaningful webs. They were specifically designed to target a technologically excluded group, older people, but can also be used with other groups. We describe the process of workshop development and discuss what succeeded with our test groups and what failed. In offering our recommendations for working in this space, we consider the methodological issues of collaborating across science/art/design borders and how this impacted on evaluation. And we describe the final result: a recipe for a performance workshop, also illustrated on a DVD and associated website, which can be used to explore the dynamics of technical and social change in the context of people’s own lives and concerns. Keywords: Performance; Older People; Marginalisation; Person-Centred; Ubiquitous Digital Networks; Interdisciplinary; Technology; Future; Evaluation</p

Visual Feedback for Players of Multi-Level Capture the Flag Games: Field Usability Study

Capture the Flag games represent a popular method of cybersecurity training. Providing meaningful insight into the training progress is essential for increasing learning impact and supporting participants' motivation, especially in advanced hands-on courses. In this paper, we investigate how to provide valuable post-game feedback to players of serious cybersecurity games through interactive visualizations. In collaboration with domain experts, we formulated user requirements that cover three cognitive perspectives: gameplay overview, person-centric view, and comparative feedback. Based on these requirements, we designed two interactive visualizations that provide complementary views on game results. They combine a known clustering and time-based visual approaches to show game results in a way that is easy to decode for players. The purposefulness of our visual feedback was evaluated in a usability field study with attendees of the Summer School in Cyber Security. The evaluation confirmed the adequacy of the two visualizations for instant post-game feedback. Despite our initial expectations, there was no strong preference for neither of the visualizations in solving different tasks

Assessing the Accuracy of Task Time Prediction of an Emerging Human Performance Modeling Software - CogTool

There is a need for a human performance modeling tool which not only has the ability to accurately estimate skilled user task time for any interface design, but can be used by modelers with little or no programming knowledge and at a minimal cost. To fulfill this need, this research investigated the accuracy of task time prediction of a modeling tool – CogTool - on two versions of an interface design used extensively in the petrochemical industry – DeltaV. CogTool uses the KeyStroke Level Model (KLM) to calculate and generate time predictions based on specified operators. The data collected from a previous study (Koffskey, Ikuma, & Harvey, 2013) that investigated how human participants (24 students and 4 operators) performed on these interfaces (in terms of mean speed in seconds) were compared to CogTool’s numeric time estimate. Three tasks (pump I, pump II and cascade system failures) on each interface for both participant groups were tested on both interfaces (improved and poor), on the general hypothesis that CogTool will make task time predictions for each of the modeled tasks, within a certain range of what actual human participants had demonstrated. The 95% confidence interval (CI) tests of the means were used to determine if the predictions fall within the intervals. The estimated task time from CogTool did not fall within the 95% CI in 9 of 12 cases. Of the 3 that were contained in the acceptable interval, two belonged to the experienced operator group for tasks performed on the improved interface, implying that CogTool was better in predicting the operators’ performance than the students’. A control room monitoring task, by its nature, places great demand on an operator’s mental capacity. This also includes the fact that operators work on multiple screens and/or consoles, sometimes requiring them to commit information to memory that they have to revisit a screen to check on some vital information. In this regard, it is suggested that the one user mental operator for “think time” (estimated as 1.2sec), should be revised in CogTool to accommodate the demand on the operator. For this reason, the present CogTool prediction did not meet expectations in estimating control room operator task time, but it however succeeded in showing where the poor interface could be improved by comparing the detailed steps to the improved interface

Honeywell Enhancing Airplane State Awareness (EASA) Project: Final Report on Refinement and Evaluation of Candidate Solutions for Airplane System State Awareness

The loss of pilot airplane state awareness (ASA) has been implicated as a factor in several aviation accidents identified by the Commercial Aviation Safety Team (CAST). These accidents were investigated to identify precursors to the loss of ASA and develop technologies to address the loss of ASA. Based on a gap analysis, two technologies were prototyped and assessed with a formative pilot-in-the-loop evaluation in NASA Langleys full-motion Research Flight Deck. The technologies address: 1) data source anomaly detection in real-time, and 2) intelligent monitoring aids to provide nominal and predictive awareness of situations to be monitored and a mission timeline to visualize events of interest. The evaluation results indicated favorable impressions of both technologies for mitigating the loss of ASA in terms of operational utility, workload, acceptability, complexity, and usability. The team concludes that there is a feasible retrofit solution for improving ASA that would minimize certification risk, integration costs, and training impact

Attention Allocation for Human Multi-Robot Control: Cognitive Analysis based on Behavior Data and Hidden States

Human multi-robot interaction exploits both the human operator’s high-level decision-making skills and the robotic agents’ vigorous computing and motion abilities. While controlling multi-robot teams, an operator’s attention must constantly shift between individual robots to maintain sufficient situation awareness. To conserve an operator’s attentional resources, a robot with self reflect capability on its abnormal status can help an operator focus her attention on emergent tasks rather than unneeded routine checks. With the proposing self-reflect aids, the human-robot interaction becomes a queuing framework, where the robots act as the clients to request for interaction and an operator acts as the server to respond these job requests. This paper examined two types of queuing schemes, the self-paced Open-queue identifying all robots’ normal/abnormal conditions, whereas the forced-paced shortest-job-first (SJF) queue showing a single robot’s request at one time by following the SJF approach. As a robot may miscarry its experienced failures in various situations, the effects of imperfect automation were also investigated in this paper. The results suggest that the SJF attentional scheduling approach can provide stable performance in both primary (locate potential targets) and secondary (resolve robots’ failures) tasks, regardless of the system’s reliability levels. However, the conventional results (e.g., number of targets marked) only present little information about users’ underlying cognitive strategies and may fail to reflect the user’s true intent. As understanding users’ intentions is critical to providing appropriate cognitive aids to enhance task performance, a Hidden Markov Model (HMM) is used to examine operators’ underlying cognitive intent and identify the unobservable cognitive states. The HMM results demonstrate fundamental differences among the queuing mechanisms and reliability conditions. The findings suggest that HMM can be helpful in investigating the use of human cognitive resources under multitasking environments

Teaching photonic integrated circuits with Jupyter notebooks : design, simulation, fabrication

At Ghent University, we have built a course curriculum on integrated photonics, and in particular silicon photonics, based on interactive Jupyter Notebooks. This has been used in short workshops, specialization courses at PhD level, as well as the M.Sc. Photonics Engineering program at Ghent University and the Free University of Brussels. The course material teaches the concepts of on-chip waveguides, basic building blocks, circuits, the design process, fabrication and measurements. The Jupyter notebook environment provides an interface where static didactic content (text, figures, movies, formulas) is mixed with Python code that the user can modify and execute, and interactive plots and widgets to explore the effect of changes in circuits or components. The Python environment supplies a host of scientific and engineering libraries, while the photonic capabilities are based on IPKISS, a commercial design framework for photonic integrated circuits by Luceda Photonics. The IPKISS framework allows scripting of layout and simulation directly from the Jupyter notebooks, so the teaching modules contain live circuit simulation, as well as integration with electromagnetic solvers. Because this is a complete design framework, students can also use it to tape out a small chip design which is fabricated through a rapid prototyping service and then measured, allowing the students to validate the actual performance of their design against the original simulation. The scripting in Jupyter notebooks also provides a self-documenting design flow, and the use of an established design tool guarantees that the acquired skills can be transferred to larger, real-world design projects