Usage of Interactive Event Timelines in Collaborative Digital Tabletops with Automation

This is a post-peer-review, pre-copyedit version of an article published in [Collaboration Meets Interactive Spaces]. The final authenticated version is available online at: http://dx.doi.org/[https://doi.org/10.1007/978-3-319-45853-3_10]Tabletop computers are increasingly being used for complex scenarios, such as emergency response. In such scenarios, maintaining situation awareness of dynamic changes automated by the system is crucial for users to make optimal decisions. If the system does not provide collaborators with appropriate feedback, they can become confused and “out-of-the-loop” about the current system state, leading to non-optimal decisions or actions. To enhance situation awareness of dynamic changes occurring in the collaborative tabletop environment, we designed an interactive event timeline to enable exploration of historical system events. We conducted a user study to understand how various design alternatives of interactive event timelines impacted situation awareness in the context of a co-operative tabletop game. Our initial results showed that, on average, all groups scored high on their combined level of situation awareness, regardless of the given timeline designs. To better understand what role the timelines played for the groups, we conducted an in-depth video analysis. Participants used the timelines mostly for perceiving new changes by interacting with the detailed information. The high-level information was beneficial for projecting future system states. The information presented in the timeline was considered as the correct historical account and was used for negotiating participants’ knowledge of the changes. We also report on how other system components, in addition to the interactive timelines, were used for situation awareness maintenance. Finally, we discuss implications for designing interactive event timelines for co-located collaborative systems involving automated events.Funder 1, NSERC Discovery Grant 2016-04422 || Funder 2, NSERC Discovery Accelerator Grant 492970-201

Supporting Situation Awareness and Workspace Awareness in Co-located Collaborative Systems Involving Dynamic Data

Co-located technologies can provide digital functionality to support collaborative work for multiple users in the same physical space. For example, digital tabletop computers — large interactive tables that allow users to directly interact with the content — can provide the most up-to-date map information while users can work together face-to-face. Combinations of interactive devices, large and small, can also be used together in a multi-device environment to support collaborative work of large groups. This environment allows individuals to utilize different networked devices. In some co-located group work, integrating automation into the available technologies can provide benefits such as automatically switching between different data views or updating map information based on underlying changes in deployed field agents’ locations. However, dynamic changes in the system state can create confusion for users and lead to low situation awareness. Furthermore, with the large size of a tabletop system or with multiple devices being used in the workspace, users may not be able to observe collaborators’ actions due to physical separations between users. Consequently, workspace awareness — knowledge of collaborators’ up-to-the-moment actions — can be difficult to maintain. As a result, users may be frustrated, and the collaboration may become inefficient or ineffective. The current tabletop applications involving dynamic data focus on interaction and information sharing techniques for collaboration rather than providing situation awareness support. Moreover, the situation awareness literature focuses primarily on single-user applications, whereas, the literature in workspace awareness primarily focuses on remote collaborative work. The aim of this dissertation was in supporting situation awareness of system-automated dynamic changes and workspace awareness of collaborators’ actions. The first study (Timeline Study) presented in this dissertation used tabletop systems to investigate supporting situation awareness of automated changes and workspace awareness, and the second study (Callout Bubble Study) followed up to further investigate workspace awareness support in the context of multi-device classrooms. Digital tabletop computers are increasingly being used for complex domains involving dynamic data, such as coastal surveillance and emergency response. Maintaining situation awareness of these changes driven by the system is crucial for quick and appropriate response when problems arise. However, distractors in the environment can make users miss the changes and negatively impact their situation awareness, e.g., the large size of the table and conversations with team members. As interactive event timelines have been shown to improve response time and decision accuracy after interruptions, in this dissertation they were adapted to the context of collaborative tabletop applications to address the lack of situation awareness due to dynamic changes. A user study was conducted to understand design factors related to the adaption and their impacts on situation awareness and workspace awareness. The Callout Bubble Study investigated workspace awareness support for multi-device classrooms, where students were co-located with their personal devices and were connected through a large shared virtual canvas. This context was chosen due to the environment’s ability to support work in large groups and the increasing prevalence of individual devices in co-located collaborative workspaces. By studying another co-located context, this research also sought to combine the lessons learned and provide a set of more generalized design recommendations for co-located technologies. Existing work on workspace awareness focuses on remote collaboration; however, the co-located users may not need all the information beneficial for remote work. This study aimed to balance awareness and distraction to improve students’ workspace awareness maintenance while minimizing distraction to their learning. A Callout Bubble was designed to augment students’ interactions in the shared online workspace, and a field study was conducted to understand how it impacted the students’ collaboration behaviour. Overall, the research presented in this dissertation aimed to investigate information visualizations for supporting situation awareness and workspace awareness in co-located collaborative environments. The contributions included the design of an interactive event timeline and an investigation of how the control placement (how many timelines and where they should be located) and feedback location (whether to display feedback to the group or to individuals when users interact with timelines) factors affected situation awareness. The empirical results revealed that individual timelines were more effective in facilitating situation awareness maintenance and the timelines were used mainly for perceiving new changes. Furthermore, this dissertation contributed in the design of a workspace awareness cue, Callout Bubble. The field study revealed that Callout Bubbles were effective in improving students’ coordination and self-monitoring behaviours, which in turn reduced teachers’ workloads. The dissertation provided overall design lessons learned for supporting awareness in co-located collaborative environments

ISAR: Ein Autorensystem für Interaktive Tische

Developing augmented reality systems involves several challenges, that prevent end users and experts from non-technical domains, such as education, to experiment with this technology. In this research we introduce ISAR, an authoring system for augmented reality tabletops targeting users from non-technical domains. ISAR allows non-technical users to create their own interactive tabletop applications and experiment with the use of this technology in domains such as educations, industrial training, and medical rehabilitation.Die Entwicklung von Augmented-Reality-Systemen ist mit mehreren Herausforderungen verbunden, die Endbenutzer und Experten aus nicht-technischen Bereichen, wie z.B. dem Bildungswesen, daran hindern, mit dieser Technologie zu experimentieren. In dieser Forschung stellen wir ISAR vor, ein Autorensystem für Augmented-Reality-Tabletops, das sich an Benutzer aus nicht-technischen Bereichen richtet. ISAR ermöglicht es nicht-technischen Anwendern, ihre eigenen interaktiven Tabletop-Anwendungen zu erstellen und mit dem Einsatz dieser Technologie in Bereichen wie Bildung, industrieller Ausbildung und medizinischer Rehabilitation zu experimentieren

Analysing, visualising and supporting collaborative learning using interactive tabletops

The key contribution of this thesis is a novel approach to design, implement and evaluate the conceptual and technological infrastructure that captures student’s activity at interactive tabletops and analyses these data through Interaction Data Analytics techniques to provide support to teachers by enhancing their awareness of student’s collaboration. To achieve the above, this thesis presents a series of carefully designed user studies to understand how to capture, analyse and distil indicators of collaborative learning. We perform this in three steps: the exploration of the feasibility of the approach, the construction of a novel solution and the execution of the conceptual proposal, both under controlled conditions and in the wild. A total of eight datasets were analysed for the studies that are described in this thesis. This work pioneered in a number of areas including the application of data mining techniques to study collaboration at the tabletop, a plug-in solution to add user-identification to a regular tabletop using a depth sensor and the first multi-tabletop classroom used to run authentic collaborative activities associated with the curricula. In summary, while the mechanisms, interfaces and studies presented in this thesis were mostly explored in the context of interactive tabletops, the findings are likely to be relevant to other forms of groupware and learning scenarios that can be implemented in real classrooms. Through the mechanisms, the studies conducted and our conceptual framework this thesis provides an important research foundation for the ways in which interactive tabletops, along with data mining and visualisation techniques, can be used to provide support to improve teacher’s understanding about student’s collaboration and learning in small groups

Spatial Interaction for Immersive Mixed-Reality Visualizations

Growing amounts of data, both in personal and professional settings, have caused an increased interest in data visualization and visual analytics. Especially for inherently three-dimensional data, immersive technologies such as virtual and augmented reality and advanced, natural interaction techniques have been shown to facilitate data analysis. Furthermore, in such use cases, the physical environment often plays an important role, both by directly influencing the data and by serving as context for the analysis. Therefore, there has been a trend to bring data visualization into new, immersive environments and to make use of the physical surroundings, leading to a surge in mixed-reality visualization research. One of the resulting challenges, however, is the design of user interaction for these often complex systems. In my thesis, I address this challenge by investigating interaction for immersive mixed-reality visualizations regarding three core research questions: 1) What are promising types of immersive mixed-reality visualizations, and how can advanced interaction concepts be applied to them? 2) How does spatial interaction benefit these visualizations and how should such interactions be designed? 3) How can spatial interaction in these immersive environments be analyzed and evaluated? To address the first question, I examine how various visualizations such as 3D node-link diagrams and volume visualizations can be adapted for immersive mixed-reality settings and how they stand to benefit from advanced interaction concepts. For the second question, I study how spatial interaction in particular can help to explore data in mixed reality. There, I look into spatial device interaction in comparison to touch input, the use of additional mobile devices as input controllers, and the potential of transparent interaction panels. Finally, to address the third question, I present my research on how user interaction in immersive mixed-reality environments can be analyzed directly in the original, real-world locations, and how this can provide new insights. Overall, with my research, I contribute interaction and visualization concepts, software prototypes, and findings from several user studies on how spatial interaction techniques can support the exploration of immersive mixed-reality visualizations.Zunehmende Datenmengen, sowohl im privaten als auch im beruflichen Umfeld, führen zu einem zunehmenden Interesse an Datenvisualisierung und visueller Analyse. Insbesondere bei inhärent dreidimensionalen Daten haben sich immersive Technologien wie Virtual und Augmented Reality sowie moderne, natürliche Interaktionstechniken als hilfreich für die Datenanalyse erwiesen. Darüber hinaus spielt in solchen Anwendungsfällen die physische Umgebung oft eine wichtige Rolle, da sie sowohl die Daten direkt beeinflusst als auch als Kontext für die Analyse dient. Daher gibt es einen Trend, die Datenvisualisierung in neue, immersive Umgebungen zu bringen und die physische Umgebung zu nutzen, was zu einem Anstieg der Forschung im Bereich Mixed-Reality-Visualisierung geführt hat. Eine der daraus resultierenden Herausforderungen ist jedoch die Gestaltung der Benutzerinteraktion für diese oft komplexen Systeme. In meiner Dissertation beschäftige ich mich mit dieser Herausforderung, indem ich die Interaktion für immersive Mixed-Reality-Visualisierungen im Hinblick auf drei zentrale Forschungsfragen untersuche: 1) Was sind vielversprechende Arten von immersiven Mixed-Reality-Visualisierungen, und wie können fortschrittliche Interaktionskonzepte auf sie angewendet werden? 2) Wie profitieren diese Visualisierungen von räumlicher Interaktion und wie sollten solche Interaktionen gestaltet werden? 3) Wie kann räumliche Interaktion in diesen immersiven Umgebungen analysiert und ausgewertet werden? Um die erste Frage zu beantworten, untersuche ich, wie verschiedene Visualisierungen wie 3D-Node-Link-Diagramme oder Volumenvisualisierungen für immersive Mixed-Reality-Umgebungen angepasst werden können und wie sie von fortgeschrittenen Interaktionskonzepten profitieren. Für die zweite Frage untersuche ich, wie insbesondere die räumliche Interaktion bei der Exploration von Daten in Mixed Reality helfen kann. Dabei betrachte ich die Interaktion mit räumlichen Geräten im Vergleich zur Touch-Eingabe, die Verwendung zusätzlicher mobiler Geräte als Controller und das Potenzial transparenter Interaktionspanels. Um die dritte Frage zu beantworten, stelle ich schließlich meine Forschung darüber vor, wie Benutzerinteraktion in immersiver Mixed-Reality direkt in der realen Umgebung analysiert werden kann und wie dies neue Erkenntnisse liefern kann. Insgesamt trage ich mit meiner Forschung durch Interaktions- und Visualisierungskonzepte, Software-Prototypen und Ergebnisse aus mehreren Nutzerstudien zu der Frage bei, wie räumliche Interaktionstechniken die Erkundung von immersiven Mixed-Reality-Visualisierungen unterstützen können

Tools and Methods to Analyze Multimodal Data in Collaborative Design Ideation

Collaborative design ideation is typically characterized by informal acts of sketching, annotation, and discussion. Designers have always used the pencil-and-paper medium for this activity, partly because of the flexibility of the medium, and partly because the ambiguous and ill-defined nature of conceptual design cannot easily be supported by computers. However, recent computational tools for conceptual design have leveraged the availability of hand-held computing devices for creating and sharing ideas. In order to provide computer support for collaborative ideation in a way that augments traditional media rather than imitates it, it is necessary to study the affordances made available by digital media for this process, and to study designers\u27 cognitive and collaborative processes when using such media. In this thesis, we present tools and methods to help make sense of unstructured verbal and sketch data generated during collaborative design, with a view to better understand these collaborative and cognitive processes. This thesis has three main contributions

Investigation and development of a tangible technology framework for highly complex and abstract concepts

The ubiquitous integration of computer-supported learning tools within the educational domain has led educators to continuously seek effective technological platforms for teaching and learning. Overcoming the inherent limitations of traditional educational approaches, interactive and tangible computing platforms have consequently garnered increased interest in the pursuit of embedding active learning pedagogies within curricula. However, whilst Tangible User Interface (TUI) systems have been successfully developed to edutain children in various research contexts, TUI architectures have seen limited deployment towards more advanced educational pursuits. Thus, in contrast to current domain research, this study investigates the effectiveness and suitability of adopting TUI systems for enhancing the learning experience of abstract and complex computational science and technology-based concepts within higher educational institutions (HEI)s. Based on the proposal of a contextually apt TUI architecture, the research describes the design and development of eight distinct TUI frameworks embodying innovate interactive paradigms through tabletop peripherals, graphical design factors, and active tangible manipulatives. These computationally coupled design elements are evaluated through summative and formative experimental methodologies for their ability to aid in the effective teaching and learning of diverse threshold concepts experienced in computational science. In addition, through the design and adoption of a technology acceptance model for educational technology (TAM4Edu), the suitability of TUI frameworks in HEI education is empirically evaluated across a myriad of determinants for modelling students’ behavioural intention. In light of the statistically significant results obtained in both academic knowledge gain (μ = 25.8%) and student satisfaction (μ = 12.7%), the study outlines the affordances provided through TUI design for various constituents of active learning theories and modalities. Thus, based on an empirical and pedagogical analyses, a set of design guidelines is defined within this research to direct the effective development of TUI design elements for teaching and learning abstract threshold concepts in HEI adaptations

Artful Systems: Investigating everyday practices of family life to inform the design of information technology for the home

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.The research in this thesis was motivated by an interest in understanding the work and effort that goes into organising family homes, with the aim of informing the design of novel information technology for the home. It was undertaken to address a notable absence of in-depth research into domestic information and communication technology in the fields of Human Computer Interaction (HCI) and Computer Supported Cooperative Work (CSCW). To that end, this thesis presents an ethnographic study of everyday routines in thirteen family homes. Following an established tradition within HCI and CSCW, the study applies qualitative fieldwork methods as a means to investigate and interpret the empirical materials. Periods of extended observation and semi-structured interviews with the thirteen families over a three-year period form the basis of the empirical material. The materials are analysed using a hybrid perspective composed of a combination of influences from the study of material culture, to interaction analysis and ethnography. The hybrid analytical perspective draws out insights regarding the families’ mundane practices and the artfully devised solutions they use to organise daily life. Four household activities and artefacts are given specific focus: (i) household list making, (ii) the display qualities of refrigerator doors, (iii) the organisation of household clutter, and (iv) the devising of bespoke solutions in organising home life. Broader findings include the observations that people tailor solutions to meet their needs, that optimum efficiency is not the pre-eminent determinant in what method or artefact people choose to organise themselves and their homes, and that homes determine their individual characters in part by how everyday tasks and organisation are accomplished. In short, the personal qualities of these mundane practices are part of what makes a home a home. These findings are used to elicit implications for information technology design, with the aim of encouraging designers of domestic technology to be aware of and respectful towards the idiosyncratic nature of the home, and, wherever possible, to design in such a way as to allow the technology to be appropriated for families’ bespoke tailoring. To evaluate and address this point, two design projects, one on augmented magnets and another on a “media bowl”, are used to develop and test out this approach. Both projects are critically examined to reflect on the efficacy of the design approach and what lessons might be learnt for future studies and design exercises. The combination of detailed ethnographic fieldwork on family homes combined with the development of experimental design projects is intended to deepen the understanding of the mundane behaviours and everyday routines of family homes, in order to better inform the design of information technology for the home

Multitouch-based collaborative pre-visualisation for computer animation

Computer animated pre-visualisation occurs at an early stage of visualising scenes in low-fidelity. This is a collaborative process, in which directors communicate with animators how shot sequences will occur. Producers also take notes to approximate costs and other stakeholders may give further input. The problem with this approach is that the improvement cycles can take a long time, making the process take exponentially longer with more iterations of improvement. Our aim is to create a system that reduces this time, while keeping every stakeholder of the animation on the same page. We have constructed a multitouch-enabled system for low-fidelity, animated 3D pre-visualisation. This tool runs on a single, large multitouch table and caters for simultaneous input from multiple users, to better support collaboration. Users can navigate the virtual environment, place and manipulate 3d objects in the scene, as well as animate them, all using multitouch. The system was constructed using the user-centred systems design (UCSD) methodology. After several iterations of development, we performed a qualitative evaluation of the _nal system using two groups, one consisting of film makers and the other consisting of software developers, and concluded with interviews to get qualitative feedback about our pre-visualisation tool. Both groups suggested that the system's setup promoted collaboration and communication, which is important early on in the planning phase of film creation. However, both groups agreed that such a tool is only useful for low-fidelity pre-visualisation, as it might become "cumbersome" to perform detailed animations using multitouch input. Furthermore, the system was often too dependent on the viewpoint, which was a single user task, effectively minimizing the amount of work that could actually be done by collaborative users simultaneously. This study highlights the potential of a multitouch, collaborative pre-visualisation tool

Interaction design for situated media production teams

PhD ThesisMedia production teams are the backbone of many media industries including television, sport gatherings and live music events. These domains are characterised by a key set of situational factors which significantly impact on the collaborative production workflow, such as temporality, professional concerns and mission criticality. The availability of new interaction technologies presents an opportunity to design systems to support these teams in these complex environments, leveraging the affordances of interaction technologies in response to the situated factors that impact specifically on these types of domains. StoryCrate and ProductionCrate, two large-scale real-world prototype systems for supporting situated media production teams were designed and deployed to explore the interaction design considerations that could support these teams in specific scenarios. Through an extensive analysis of these deployments, key design considerations, interaction techniques and modalities are presented that can be developed in response to the situational factors found in collaborative media production environments