Group reaching over digital tabletops with digital arm embodiments

In almost all collaborative tabletop tasks, groups require coordinated access to the shared objects on the table’s surface. The physical social norms of close-proximity interactions built up over years of interacting around other physical bodies cause people to avoid interfering with other people (e.g., avoiding grabbing the same object simultaneously). However, some digital tabletop situations require the use of indirect input (e.g., when using mice, and when supporting remote users). With indirect input, people are no longer physically embodied during their reaching gestures, so most systems provide digital embodiments – visual representations of each person – to provide feedback to both the person who is reaching and to the other group members. Tabletop arm embodiments have been shown to better support group interactions than simple visual designs, providing awareness of actions to the group. However, researchers and digital tabletop designers know little of how the design of digital arm embodiments affects the fundamental group tabletop interaction of reaching for objects. Therefore, in this thesis, we evaluate how people coordinate their interactions over digital tabletops when using different types of embodiments. Specifically, in a series of studies, we investigate how the visual design (what they look like) and interaction design (how they work) of digital arm embodiments affects a group’s coordinative behaviours in an open- ended parallel tabletop task. We evaluated visual factors of size, transparency, and realism (through pictures and videos of physical arms), as well as interaction factors of input and augmentations (feedback of interactions), in both a co-located and distributed environment. We found that the visual design had little effect on a group’s ability to coordinate access to shared tabletop items, that embodiment augmentations are useful to support group coordinative actions, and that there are large differences when the person is not physically co-present. Our results demonstrate an initial exploration into the design of digital arm embodiments, providing design guidelines for future researchers and designers to use when designing the next generation of shared digital spaces

Robot mediated communication: Enhancing tele-presence using an avatar

In the past few years there has been a lot of development in the field of tele-presence. These developments have caused tele-presence technologies to become easily accessible and also for the experience to be enhanced. Since tele-presence is not only used for tele-presence assisted group meetings but also in some forms of Computer Supported Cooperative Work (CSCW), these activities have also been facilitated. One of the lingering issues has to do with how to properly transmit presence of non-co-located members to the rest of the group. Using current commercially available tele-presence technology it is possible to exhibit a limited level of social presence but no physical presence. In order to cater for this lack of presence a system is implemented here using tele-operated robots as avatars for remote team members and had its efficacy tested. This testing includes both the level of presence that can be exhibited by robot avatars but also how the efficacy of these robots for this task changes depending on the morphology of the robot. Using different types of robots, a humanoid robot and an industrial robot arm, as tele-presence avatars, it is found that the humanoid robot using an appropriate control system is better at exhibiting a social presence. Further, when compared to a voice only scenario, both robots proved significantly better than with only voice in terms of both cooperative task solving and social presence. These results indicate that using an appropriate control system, a humanoid robot can be better than an industrial robot in these types of tasks and the validity of aiming for a humanoid design behaving in a human-like way in order to emulate social interactions that are closer to human norms. This has implications for the design of autonomous socially interactive robot systems

The Effects of Sharing Awareness Cues in Collaborative Mixed Reality

Augmented and Virtual Reality provide unique capabilities for Mixed Reality collaboration. This paper explores how different combinations of virtual awareness cues can provide users with valuable information about their collaborator's attention and actions. In a user study (n = 32, 16 pairs), we compared different combinations of three cues: Field-of-View (FoV) frustum, Eye-gaze ray, and Head-gaze ray against a baseline condition showing only virtual representations of each collaborator's head and hands. Through a collaborative object finding and placing task, the results showed that awareness cues significantly improved user performance, usability, and subjective preferences, with the combination of the FoV frustum and the Head-gaze ray being best. This work establishes the feasibility of room-scale MR collaboration and the utility of providing virtual awareness cues

Using natural user interfaces to support synchronous distributed collaborative work

Synchronous Distributed Collaborative Work (SDCW) occurs when group members work together at the same time from different places together to achieve a common goal. Effective SDCW requires good communication, continuous coordination and shared information among group members. SDCW is possible because of groupware, a class of computer software systems that supports group work. Shared-workspace groupware systems are systems that provide a common workspace that aims to replicate aspects of a physical workspace that is shared among group members in a co-located environment. Shared-workspace groupware systems have failed to provide the same degree of coordination and awareness among distributed group members that exists in co-located groups owing to unintuitive interaction techniques that these systems have incorporated. Natural User Interfaces (NUIs) focus on reusing natural human abilities such as touch, speech, gestures and proximity awareness to allow intuitive human-computer interaction. These interaction techniques could provide solutions to the existing issues of groupware systems by breaking down the barrier between people and technology created by the interaction techniques currently utilised. The aim of this research was to investigate how NUI interaction techniques could be used to effectively support SDCW. An architecture for such a shared-workspace groupware system was proposed and a prototype, called GroupAware, was designed and developed based on this architecture. GroupAware allows multiple users from distributed locations to simultaneously view and annotate text documents, and create graphic designs in a shared workspace. Documents are represented as visual objects that can be manipulated through touch gestures. Group coordination and awareness is maintained through document updates via immediate workspace synchronization, user action tracking via user labels and user availability identification via basic proxemic interaction. Members can effectively communicate via audio and video conferencing. A user study was conducted to evaluate GroupAware and determine whether NUI interaction techniques effectively supported SDCW. Ten groups of three members each participated in the study. High levels of performance, user satisfaction and collaboration demonstrated that GroupAware was an effective groupware system that was easy to learn and use, and effectively supported group work in terms of communication, coordination and information sharing. Participants gave highly positive comments about the system that further supported the results. The successful implementation of GroupAware and the positive results obtained from the user evaluation provides evidence that NUI interaction techniques can effectively support SDCW

Enabling Collaborative Visual Analysis across Heterogeneous Devices

We are surrounded by novel device technologies emerging at an unprecedented pace. These devices are heterogeneous in nature: in large and small sizes with many input and sensing mechanisms. When many such devices are used by multiple users with a shared goal, they form a heterogeneous device ecosystem. A device ecosystem has great potential in data science to act as a natural medium for multiple analysts to make sense of data using visualization. It is essential as today's big data problems require more than a single mind or a single machine to solve them. Towards this vision, I introduce the concept of collaborative, cross-device visual analytics (C2-VA) and outline a reference model to develop user interfaces for C2-VA. This dissertation covers interaction models, coordination techniques, and software platforms to enable full stack support for C2-VA. Firstly, we connected devices to form an ecosystem using software primitives introduced in the early frameworks from this dissertation. To work in a device ecosystem, we designed multi-user interaction for visual analysis in front of large displays by finding a balance between proxemics and mid-air gestures. Extending these techniques, we considered the roles of different devices–large and small–to present a conceptual framework for utilizing multiple devices for visual analytics. When applying this framework, findings from a user study showcase flexibility in the analytic workflow and potential for generation of complex insights in device ecosystems. Beyond this, we supported coordination between multiple users in a device ecosystem by depicting the presence, attention, and data coverage of each analyst within a group. Building on these parts of the C2-VA stack, the culmination of this dissertation is a platform called Vistrates. This platform introduces a component model for modular creation of user interfaces that work across multiple devices and users. A component is an analytical primitive–a data processing method, a visualization, or an interaction technique–that is reusable, composable, and extensible. Together, components can support a complex analytical activity. On top of the component model, the support for collaboration and device ecosystems comes for granted in Vistrates. Overall, this enables the exploration of new research ideas within C2-VA

Co-located Collaborative Information-based Ideation through Embodied Cross-Surface Curation

We develop an embodied cross-surface curation environment to support co-located, collaborative information-based ideation. Information-based ideation (IBI) refers to tasks and activities in which people generate and develop significant new ideas while working with information. Curation is the process of gathering and assembling objects in order to express ideas. The linear media and separated screens of prior curation environments constrain expression. This research utilizes information composition of rich bookmarks as the medium of curation. Visual representation of elements and ability to combine them in a freeform, spatial manner mimics how objects appear and can be manipulated in the physical world. Metadata of rich bookmarks leverages capabilities of the WWW. We equip participants with personal IBI environments, each on a mobile device, as a base for contributing to curation on a larger, collaborative surface. We hypothesize that physical representations for the elements and assemblage of curation, layered with physical techniques of interaction, will facilitate co-located IBI. We hypothesize that consistent physical and spatial representations of information and means for manipulating rich bookmarks on and across personal and collaborative surfaces will support IBI. We hypothesize that the small size and weight of personal devices will facilitate participants shifting their attention from their own work to each other and collaboration. We evaluated the curation environment by inviting couples to participate in a home makeover design task in a living-room lab. We demonstrated that our embodied cross-surface curation environment supports creative thinking, facilitates communication, and stimulates engagement and creativity in collaborative IBI

Designing for Shareable Interfaces in the Wild

Despite excitement about the potential of interactive tabletops to support collaborative work, there have been few empirical demonstrations of their effectiveness (Marshall et al., 2011). In particular, while lab-based studies have explored the effects of individual design features, there has been a dearth of studies evaluating the success of systems in the wild. For this technology to be of value, designers and systems builders require a better understanding of how to develop and evaluate tabletop applications to be deployed in real world settings. This dissertation reports on two systems designed through a process that incorporated ethnography-style observations, iterative design and in the wild evaluation. The first study focused on collaborative learning in a medical setting. To address the fact that visitors to a hospital emergency ward were leaving with an incomplete understanding of their diagnosis and treatment, a system was prototyped in a working Emergency Room (ER) with doctors and patients. The system was found to be helpful but adoption issues hampered its impact. The second study focused on a planning application for visitors to a tourist information centre. Issues and opportunities for a successful, contextually-fitted system were addressed and it was found to be effective in supporting group planning activities by novice users, in particular, facilitating users’ first experiences, providing effective signage and offering assistance to guide the user through the application. This dissertation contributes to understanding of multi-user systems through literature review of tabletop systems, collaborative tasks, design frameworks and evaluation of prototypes. Some support was found for the claim that tabletops are a useful technology for collaboration, and several issues were discussed. Contributions to understanding in this field are delivered through design guidelines, heuristics, frameworks, and recommendations, in addition to the two case studies to help guide future tabletop system creators

Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces

This paper contributes to a taxonomy of augmented reality and robotics based on a survey of 460 research papers. Augmented and mixed reality (AR/MR) have emerged as a new way to enhance human-robot interaction (HRI) and robotic interfaces (e.g., actuated and shape-changing interfaces). Recently, an increasing number of studies in HCI, HRI, and robotics have demonstrated how AR enables better interactions between people and robots. However, often research remains focused on individual explorations and key design strategies, and research questions are rarely analyzed systematically. In this paper, we synthesize and categorize this research field in the following dimensions: 1) approaches to augmenting reality; 2) characteristics of robots; 3) purposes and benefits; 4) classification of presented information; 5) design components and strategies for visual augmentation; 6) interaction techniques and modalities; 7) application domains; and 8) evaluation strategies. We formulate key challenges and opportunities to guide and inform future research in AR and robotics

Tabletop Tangible Interfaces for Music Performance: Design and Evaluation

This thesis investigates a new generation of collaborative systems: tabletop tangible interfaces (TTIs) for music performance or musical tabletops. Musical tabletops are designed for professional musical performance, as well as for casual interaction in public settings. These systems support co-located collaboration, offered by a shared interface. However, we still know little about their challenges and opportunities for collaborative musical practice: in particular, how to best support beginners or experts or both. This thesis explores the nature of collaboration on TTIs for music performance between beginners, experts, or both. Empirical work was done in two stages: 1) an exploratory stage; and 2) an experimental stage. In the exploratory stage we studied the Reactable, a commercial musical tabletop designed for beginners and experts. In particular, we explored its use in two environments: a multi-session study with expert musicians in a casual lab setting; and a field study with casual visitors in a science centre. In the experimental stage we conducted a controlled experiment for mixed groups using a bespoke musical tabletop interface, SoundXY4. The design of this study was informed by the previous stage about a need to support better real-time awareness of the group activity (workspace awareness) in early interactions. For the three studies, groups musical improvisation was video-captured unobtrusively with the aim of understanding natural uses during group musical practice. Rich video data was carefully analysed focusing on the nature of social interaction and how workspace awareness was manifested. The findings suggest that musical tabletops can support peer learning during multiple sessions; fluid between-group social interaction in public settings; and a democratic and ecological approach to music performance. The findings also point to how workspace awareness can be enhanced in early interactions with TTIs using auditory feedback with ambisonics spatialisation. The thesis concludes with theoretical, methodological, and practical implications for future research in New Interfaces for Musical Expression (NIME), tabletop studies, and Human-Computer Interaction (HCI)

Supporting Situation Awareness and Workspace Awareness in Co-located Collaborative Systems Involving Dynamic Data

Co-located technologies can provide digital functionality to support collaborative work for multiple users in the same physical space. For example, digital tabletop computers — large interactive tables that allow users to directly interact with the content — can provide the most up-to-date map information while users can work together face-to-face. Combinations of interactive devices, large and small, can also be used together in a multi-device environment to support collaborative work of large groups. This environment allows individuals to utilize different networked devices. In some co-located group work, integrating automation into the available technologies can provide benefits such as automatically switching between different data views or updating map information based on underlying changes in deployed field agents’ locations. However, dynamic changes in the system state can create confusion for users and lead to low situation awareness. Furthermore, with the large size of a tabletop system or with multiple devices being used in the workspace, users may not be able to observe collaborators’ actions due to physical separations between users. Consequently, workspace awareness — knowledge of collaborators’ up-to-the-moment actions — can be difficult to maintain. As a result, users may be frustrated, and the collaboration may become inefficient or ineffective. The current tabletop applications involving dynamic data focus on interaction and information sharing techniques for collaboration rather than providing situation awareness support. Moreover, the situation awareness literature focuses primarily on single-user applications, whereas, the literature in workspace awareness primarily focuses on remote collaborative work. The aim of this dissertation was in supporting situation awareness of system-automated dynamic changes and workspace awareness of collaborators’ actions. The first study (Timeline Study) presented in this dissertation used tabletop systems to investigate supporting situation awareness of automated changes and workspace awareness, and the second study (Callout Bubble Study) followed up to further investigate workspace awareness support in the context of multi-device classrooms. Digital tabletop computers are increasingly being used for complex domains involving dynamic data, such as coastal surveillance and emergency response. Maintaining situation awareness of these changes driven by the system is crucial for quick and appropriate response when problems arise. However, distractors in the environment can make users miss the changes and negatively impact their situation awareness, e.g., the large size of the table and conversations with team members. As interactive event timelines have been shown to improve response time and decision accuracy after interruptions, in this dissertation they were adapted to the context of collaborative tabletop applications to address the lack of situation awareness due to dynamic changes. A user study was conducted to understand design factors related to the adaption and their impacts on situation awareness and workspace awareness. The Callout Bubble Study investigated workspace awareness support for multi-device classrooms, where students were co-located with their personal devices and were connected through a large shared virtual canvas. This context was chosen due to the environment’s ability to support work in large groups and the increasing prevalence of individual devices in co-located collaborative workspaces. By studying another co-located context, this research also sought to combine the lessons learned and provide a set of more generalized design recommendations for co-located technologies. Existing work on workspace awareness focuses on remote collaboration; however, the co-located users may not need all the information beneficial for remote work. This study aimed to balance awareness and distraction to improve students’ workspace awareness maintenance while minimizing distraction to their learning. A Callout Bubble was designed to augment students’ interactions in the shared online workspace, and a field study was conducted to understand how it impacted the students’ collaboration behaviour. Overall, the research presented in this dissertation aimed to investigate information visualizations for supporting situation awareness and workspace awareness in co-located collaborative environments. The contributions included the design of an interactive event timeline and an investigation of how the control placement (how many timelines and where they should be located) and feedback location (whether to display feedback to the group or to individuals when users interact with timelines) factors affected situation awareness. The empirical results revealed that individual timelines were more effective in facilitating situation awareness maintenance and the timelines were used mainly for perceiving new changes. Furthermore, this dissertation contributed in the design of a workspace awareness cue, Callout Bubble. The field study revealed that Callout Bubbles were effective in improving students’ coordination and self-monitoring behaviours, which in turn reduced teachers’ workloads. The dissertation provided overall design lessons learned for supporting awareness in co-located collaborative environments