Designing to Support Workspace Awareness in Remote Collaboration using 2D Interactive Surfaces

Increasing distributions of the global workforce are leading to collaborative workamong remote coworkers. The emergence of such remote collaborations is essentiallysupported by technology advancements of screen-based devices ranging from tabletor laptop to large displays. However, these devices, especially personal and mobilecomputers, still suffer from certain limitations caused by their form factors, that hinder supporting workspace awareness through non-verbal communication suchas bodily gestures or gaze. This thesis thus aims to design novel interfaces andinteraction techniques to improve remote coworkers’ workspace awareness throughsuch non-verbal cues using 2D interactive surfaces.The thesis starts off by exploring how visual cues support workspace awareness infacilitated brainstorming of hybrid teams of co-located and remote coworkers. Basedon insights from this exploration, the thesis introduces three interfaces for mobiledevices that help users maintain and convey their workspace awareness with their coworkers. The first interface is a virtual environment that allows a remote person to effectively maintain his/her awareness of his/her co-located collaborators’ activities while interacting with the shared workspace. To help a person better express his/her hand gestures in remote collaboration using a mobile device, the second interfacepresents a lightweight add-on for capturing hand images on and above the device’sscreen; and overlaying them on collaborators’ device to improve their workspace awareness. The third interface strategically leverages the entire screen space of aconventional laptop to better convey a remote person’s gaze to his/her co-locatedcollaborators. Building on the top of these three interfaces, the thesis envisions an interface that supports a person using a mobile device to effectively collaborate with remote coworkers working with a large display.Together, these interfaces demonstrate the possibilities to innovate on commodity devices to offer richer non-verbal communication and better support workspace awareness in remote collaboration

Metro Environmental: The impact of training HVAC technicians using the SightPros-VirTechs system for remote, wireless, Internet video assistance.

This qualitative study explored the overall impact of training HVAC technicians using the SightPros-VirTechs system for remote, wireless, internet video assistance at a small HVAC company, Metro Environmental. John Thomason, the president/co-owner developed a website and a new SightPros communication tool that allows wireless, one-on-one, just-in-time, high-quality, video-monitored instructions between an expert at one site and a technician at another site. Metro Environmental successfully used the SightPros-VirTechs system to train a new apprentice remotely. The apprentice and expert changed their normal and routine physical activities because the expert worked remotely and the apprentice worked on-site. Within just a few months, the apprentice proved competent enough to go to customer accounts without more experienced technicians nearby. The technicians express excitement about the SightPros communication tool as a way to contact remote experts whenever needed. The customer and business contacts also give good reviews and suggest other benefits. The expert permanently captures the communications so the company can use the saved video for many applications, especially training. The dissertation provides a list of recommendations to trainers/educators for similar applications

Social Interactions in Immersive Virtual Environments: People, Agents, and Avatars

Immersive virtual environments (IVEs) have received increased popularity with applications in many fields. IVEs aim to approximate real environments, and to make users react similarly to how they would in everyday life. An important use case is the users-virtual characters (VCs) interaction. We interact with other people every day, hence we expect others to appropriately act and behave, verbally and non-verbally (i.e., pitch, proximity, gaze, turn-taking). These expectations also apply to interactions with VCs in IVEs, and this thesis tackles some of these aspects. We present three projects that inform the area of social interactions with a VC in IVEs, focusing on non-verbal behaviours. In our first study on interactions between people, we collaborated with the Social Neuroscience group at the Institute of Cognitive Neuroscience from UCL on a dyad multi-modal interaction. This aims to understand the conversation dynamics, focusing on gaze and turn-taking. The results show that people have a higher frequency of gaze change (from averted to direct and vice versa) when they are being looked at compared to when they are not. When they are not being looked at, they are also directing their gaze to their partners more compared to when they are being looked at. Another contribution of this work is the automated method of annotating speech and gaze data. Next, we consider agents’ higher-level non-verbal behaviours, covering social attitudes. We present a pipeline to collect data and train a machine learning (ML) model that detects social attitudes in a user-VC interaction. Here we collaborated with two game studios: Dream Reality Interaction and Maze Theory. We present a case study for the ML pipeline on social engagement recognition for the Peaky Blinders narrative VR game from Maze Theory studio. We use a reinforcement learning algorithm with imitation learning rewards and a temporal memory element. The results show that the model trained with raw data does not generalise and performs worse (60% accuracy) than the one trained with socially meaningful data (83% accuracy). In IVEs, people embody avatars and their appearance can impact social interactions. In collaboration with Microsoft Research, we report a longitudinal study in mixed-reality on avatar appearance in real-work meetings between co-workers comparing personalised full-body realistic and cartoon avatars. The results imply that when participants use realistic avatars first, they may have higher expectations and they perceive their colleagues’ emotional states with less accuracy. Participants may also become more accustomed to cartoon avatars as time passes and the overall use of avatars may lead to less accurately perceiving negative emotions. The work presented here contributes towards the field of detecting and generating nonverbal cues for VCs in IVEs. These are also important building blocks for creating autonomous agents for IVEs. Additionally, this work contributes to the games and work industry fields through an immersive ML pipeline for detecting social attitudes and through insights into using different avatar styles over time in real-world meetings

Enhanced Virtuality: Increasing the Usability and Productivity of Virtual Environments

Mit stetig steigender Bildschirmauflösung, genauerem Tracking und fallenden Preisen stehen Virtual Reality (VR) Systeme kurz davor sich erfolgreich am Markt zu etablieren. Verschiedene Werkzeuge helfen Entwicklern bei der Erstellung komplexer Interaktionen mit mehreren Benutzern innerhalb adaptiver virtueller Umgebungen. Allerdings entstehen mit der Verbreitung der VR-Systeme auch zusätzliche Herausforderungen: Diverse Eingabegeräte mit ungewohnten Formen und Tastenlayouts verhindern eine intuitive Interaktion. Darüber hinaus zwingt der eingeschränkte Funktionsumfang bestehender Software die Nutzer dazu, auf herkömmliche PC- oder Touch-basierte Systeme zurückzugreifen. Außerdem birgt die Zusammenarbeit mit anderen Anwendern am gleichen Standort Herausforderungen hinsichtlich der Kalibrierung unterschiedlicher Trackingsysteme und der Kollisionsvermeidung. Beim entfernten Zusammenarbeiten wird die Interaktion durch Latenzzeiten und Verbindungsverluste zusätzlich beeinflusst. Schließlich haben die Benutzer unterschiedliche Anforderungen an die Visualisierung von Inhalten, z.B. Größe, Ausrichtung, Farbe oder Kontrast, innerhalb der virtuellen Welten. Eine strikte Nachbildung von realen Umgebungen in VR verschenkt Potential und wird es nicht ermöglichen, die individuellen Bedürfnisse der Benutzer zu berücksichtigen. Um diese Probleme anzugehen, werden in der vorliegenden Arbeit Lösungen in den Bereichen Eingabe, Zusammenarbeit und Erweiterung von virtuellen Welten und Benutzern vorgestellt, die darauf abzielen, die Benutzerfreundlichkeit und Produktivität von VR zu erhöhen. Zunächst werden PC-basierte Hardware und Software in die virtuelle Welt übertragen, um die Vertrautheit und den Funktionsumfang bestehender Anwendungen in VR zu erhalten. Virtuelle Stellvertreter von physischen Geräten, z.B. Tastatur und Tablet, und ein VR-Modus für Anwendungen ermöglichen es dem Benutzer reale Fähigkeiten in die virtuelle Welt zu übertragen. Des Weiteren wird ein Algorithmus vorgestellt, der die Kalibrierung mehrerer ko-lokaler VR-Geräte mit hoher Genauigkeit und geringen Hardwareanforderungen und geringem Aufwand ermöglicht. Da VR-Headsets die reale Umgebung der Benutzer ausblenden, wird die Relevanz einer Ganzkörper-Avatar-Visualisierung für die Kollisionsvermeidung und das entfernte Zusammenarbeiten nachgewiesen. Darüber hinaus werden personalisierte räumliche oder zeitliche Modifikationen vorgestellt, die es erlauben, die Benutzerfreundlichkeit, Arbeitsleistung und soziale Präsenz von Benutzern zu erhöhen. Diskrepanzen zwischen den virtuellen Welten, die durch persönliche Anpassungen entstehen, werden durch Methoden der Avatar-Umlenkung (engl. redirection) kompensiert. Abschließend werden einige der Methoden und Erkenntnisse in eine beispielhafte Anwendung integriert, um deren praktische Anwendbarkeit zu verdeutlichen. Die vorliegende Arbeit zeigt, dass virtuelle Umgebungen auf realen Fähigkeiten und Erfahrungen aufbauen können, um eine vertraute und einfache Interaktion und Zusammenarbeit von Benutzern zu gewährleisten. Darüber hinaus ermöglichen individuelle Erweiterungen des virtuellen Inhalts und der Avatare Einschränkungen der realen Welt zu überwinden und das Erlebnis von VR-Umgebungen zu steigern

Annotations of maps in collaborative work at a distance

This thesis inquires how map annotations can be used to sustain remote collaboration. Maps condense the interplay of space and communication, solving linguistic references by linking conversational content to the actual places to which it refers. This is a mechanism people are accustomed to. When we are face-to-face, we can point to things around us. However, at a distance, we need to recreate a context that can help disambiguate what we mean. A map can help recreate this context. However other technological solutions are required to allow deictic gestures over a shared map when collaborators are not co-located. This mechanism is here termed Explicit Referencing. Several systems that allow sharing maps annotations are reviewed critically. A taxonomy is then proposed to compare their features. Two filed experiments were conducted to investigate the production of collaborative annotations of maps with mobile devices, looking for the reasons why people might want to produce these notes and how they might do so. Both studies led to very disappointing results. The reasons for this failure are attributed to the lack of a critical mass of users (social network), the lack of useful content, and limited social awareness. More importantly, the study identified a compelling effect of the way messages were organized in the tested application, which caused participants to refrain from engaging in content-driven explorations and synchronous discussions. This last qualitative observation was refined in a controlled experiment where remote participants had to solve a problem collaboratively, using chat tools that differed in the way a user could relate an utterance to a shared map. Results indicated that team performance is improved by the Explicit Referencing mechanisms. However, when this is implemented in a way that is detrimental to the linearity of the conversation, resulting in the visual dispersion or scattering of messages, its use has negative consequences for collaborative work at a distance. Additionally, an analysis of the eye movements of the participants over the map helped to ascertain the interplay of deixis and gaze in collaboration. A primary relation was found between the pair's recurrence of eye movements and their task performance. Finally, this thesis presents an algorithm that detects misunderstandings in collaborative work at a distance. It analyses the movements of collaborators' eyes over the shared map, their utterances containing references to this workspace, and the availability of "remote" deictic gestures. The algorithm associates the distance between the gazes of the emitter and gazes of the receiver of a message with the probability that the recipient did not understand the message

Design revolutions: IASDR 2019 Conference Proceedings. Volume 4: Learning, Technology, Thinking

In September 2019 Manchester School of Art at Manchester Metropolitan University was honoured to host the bi-annual conference of the International Association of Societies of Design Research (IASDR) under the unifying theme of DESIGN REVOLUTIONS. This was the first time the conference had been held in the UK. Through key research themes across nine conference tracks – Change, Learning, Living, Making, People, Technology, Thinking, Value and Voices – the conference opened up compelling, meaningful and radical dialogue of the role of design in addressing societal and organisational challenges. This Volume 4 includes papers from Learning, Technology and Thinking tracks of the conference

Context-based multimodal interpretation : an integrated approach to multimodal fusion and discourse processing

This thesis is concerned with the context-based interpretation of verbal and nonverbal contributions to interactions in multimodal multiparty dialogue systems. On the basis of a detailed analysis of context-dependent multimodal discourse phenomena, a comprehensive context model is developed. This context model supports the resolution of a variety of referring and elliptical expressions as well as the processing and reactive generation of turn-taking signals and the identification of the intended addressee(s) of a contribution. A major goal of this thesis is the development of a generic component for multimodal fusion and discourse processing. Based on the integration of this component into three distinct multimodal dialogue systems, the generic applicability of the approach is shown.Diese Dissertation befasst sich mit der kontextbasierten Interpretation von verbalen und nonverbalen Gesprächsbeiträgen im Rahmen von multimodalen Dialogsystemen. Im Rahmen dieser Arbeit wird, basierend auf einer detaillierten Analyse multimodaler Diskursphänomene, ein umfassendes Modell des Gesprächskontextes erarbeitet. Dieses Modell soll sowohl die Verarbeitung einer Vielzahl von referentiellen und elliptischen Ausdrücken, als auch die Erzeugung reaktiver Aktionen wie sie für den Sprecherwechsel benötigt werden unterstützen. Ein zentrales Ziel dieser Arbeit ist die Entwicklung einer generischen Komponente zur multimodalen Fusion und Diskursverarbeitung. Anhand der Integration dieser Komponente in drei unterschiedliche Dialogsysteme soll der generische Charakter dieser Komponente gezeigt werden

Actions and Words: Meaning Making in a Bilingual Arabic/English Kindergarten

Much research relating to bilingual education in the early years has focused on how young children are learning a new language, viewed from a monolingual paradigm of language use (Tabors 1997, Clarke 1999, Cummins 2001, Drury 2007, Genesee 2008). Recent sociocultural research on meaning-making in communities (García and Sylvan 2011, Gort and Sembiante 2015, Wei 2018, Creese and Blackledge 2019) reveals a less rigid boundary between named languages, and an emerging theory of translanguaging. This theory offers a new perspective from which to view pedagogical practices in bilingual classrooms. The multimodal nature of young children’s meaning-making has long been acknowledged, founded on the work of Vygotsky in the 1920s, and is evident in the variety of pedagogical practices used by early years teachers to support meaning-making (Rogoff 1990, Siraj-Blatchford et al 2002). More recently, multimodal meaning-making has been explored in different communities (Kress 2000, Flewitt 2013, Bezemer and Kress 2014), revealing the many different modes of communication regularly employed to share meaning. However, in studies of early bilingual education, the significance of non-verbal behaviours has often been overlooked. This thesis therefore sets out to explore the significance of multimodal meaning-making and a pedagogy for translingual practice (Canagarajah 2013) in an early years bilingual context. Using ethnographic tools, it analyses the pedagogical practices of two teachers working in a co-teaching situation in a kindergarten in Abu Dhabi. By an iterative process of data analysis, it explores how those practices contribute to children’s development as confident bilingual learners. The findings indicate that, as teachers create new spaces for learning using action and gesture as well as speech, children are empowered to draw on both verbal and non-verbal modes of meaning-making in a fluid process of bilingual and multimodal languaging. Emergent bilingualism, translanguaging, multimodality, sociocultural theory, early years, co-teaching, Arabian Gulf

Promoting Andean children's learning of science through cultural and digital tools

Conference Theme: To see the world and a grain of sand: Learning across levels of space, time, and scaleIn Peru, there is a large achievement gap in rural schools. In order to overcome this problem, the study aims to design environments that enhance science learning through the integration of ICT with cultural artifacts, respecting the Andean culture and empower rural children to pursue lifelong learning. This investigation employs the Cultural-Historical Activity Theory (CHAT) framework, and the Design-Based Research (DBR) methodology using an iterative process of design, implementation and evaluation of the innovative practice.published_or_final_versio

Integrating BIM and GIS for design collaboration in railway projects

Collaboration is essential to achieve project targets and minimising rework in any project including railway projects. The railway project is considered as a megaproject that requires effective collaboration in order to achieve efficiency and effectiveness. To ensure that the railway continues to provide safe, reliable, cost-effective services, and remains environmentally friendly while driving economic growth, engaging new technologies and new types of work models are required. Among these technologies, Building Information Modelling (BIM) and Geographic Information Systems (GIS) are recent technologies that support collaboration. However, using these technologies to achieve effective collaboration is challenging, especially in railway projects as they are amongst the most complicated projects and often numerous parties are involved in making important decisions. Currently, there is a lack of evidence-based guidelines or processes for effective collaboration in railway projects throughout their design stage. Therefore, this thesis has focused on developing a process model to improve collaboration in the design stage of railway projects using BIM and GIS. This research adopted a mixed-methods approach to examine and identify the issues that hinder collaboration in railway projects to assist in developing theBIM and GIS-enabled collaboration process model. An online questionnaire was designed and distributed to professionals to assess the state-of-the-art in BIM and GIS followed by two rounds of in-depth interviews with experts. The first round aimed to identify collaboration issues and consisted of 15 in-depth, face to face and videoconference/telephone interviews; while the second round consisted of 10 in-depth interviews to identify the process model components of the collaborative process using IDEF technique.The questionnaire data were analysed using descriptive statistics and statistical tests (for example, Regression analysis, Wilcoxon Signed Ranks and Kruskal-Wallis Test). The results showed a lack of training in BIM and GIS and identified collaboration as a significant factor for railway projects, but there were many challenges to achieve effective collaboration. These challenges have been further investigated during the first round of interviews using content and thematic analysis. The results revealed that the most common challenges were getting the right information at the right time for the right purposes followed by resistance to change. Furthermore, the findings indicated that developing a process model, based on a clear plan of work demonstrating the collaboration process, is a potential solution to tackle these challenges. Thus, a Collaborative Plan of Work (CPW) has been developed through combining the RIBA (Royal Institute of British Architects) Plan of Work and the GRIP (Governance for Railway Investment Projects) stages. This CPW will be the basis to develop a process model for BIM and GIS-enabled collaboration. The results from the second round of the interviews identified the process model components which are: key players’ roles and responsibilities, tasks (BIM and GIS Uses), BIM and GIS-based deliverables, and critical decision points for collaborative process design. Moreover, this process model was formulated utilising Integrated DEFinition (IDEF) structured diagramming techniques (IDEF0 and IDEF3).In conclusion, the process model of the collaboration process and the integrated implementation of BIM and GIS sets out role and responsibilities, deliverables, and key decision points. Finally, the research outcomes have been validated through a focus group and interviews with professionals in the biggest Railway company where the proposed process model was operationalised using a commercial Common Data Environment platform (viewpoint 4project). From their discussion, feedback and recommendations the IDEF processes model have been refined. It is concluded that such a process is crucial for effective collaboration in railway projects as it enables the management of the design process in terms of technologies used, activities, deliverables, and decision points. Therefore, the research findings support the notion that BIM and GIS can help to achieve effective collaboration by delivering the right information at the right time for the right purposes. As a result, they help to achieve the projects’ objectives efficiently in terms of time, cost and effort.</div