Augmented Reality to Facilitate a Conceptual Understanding of Statics in Vocational Education

At the core of the contribution of this dissertation there is an augmented reality (AR) environment, StaticAR, that supports the process of learning the fundamentals of statics in vocational classrooms, particularly in carpentry ones. Vocational apprentices are expected to develop an intuition of these topics rather than a formal comprehension. We have explored the potentials of the AR technology for this pedagogical challenge. Furthermore, we have investigated the role of physical objects in mixed-reality systems when they are implemented as tangible user interfaces (TUIs) or when they serve as a background for the augmentation in handheld AR. This thesis includes four studies. In the first study, we used eye-tracking methods to look for evidences of the benefits associated to TUIs in the learning context. We designed a 3D modelling task and compared users' performance when they completed it using a TUI or a GUI. The gaze measures that we analysed further confirmed the positive impact that TUIs can have on the learners' experience and enforced the empirical basis for their adoption in learning applications. The second study evaluated whether the physical interaction with models of carpentry structures could lead to a better understanding of statics principles. Apprentices engaged in a learning activity in which they could manipulate physical models that were mechanically augmented, allowing for exploring how structures react to external loads. The analysis of apprentices' performance and their gaze behaviors highlighted the absence of clear advantages in exploring statics through manipulation. This study also showed that the manipulation might prevent students from noticing aspects relevant for solving statics problems. From the second study we obtained guidelines to design StaticAR which implements the magic-lens metaphor: a tablet augments a small-scale structure with information about its structural behavior. The structure is only a background for the augmentation and its manipulation does not trigger any function, so in the third study we asked to what extent it was important to have it. We rephrased this question to whether users would look directly at the structure instead of seeing it only through a tablet. Our findings suggested that a shift of attention from the screen to the physical object (a structure in our case) might occur in order to sustain users' spatial orientation when they change positions. In addition, the properties of the gaze shift (e.g. duration) could depend on the features of the task (e.g. difficulty) and of the setup (e.g. stability of the augmentation). The focus of our last study was the digital representation of the forces that act in a loaded structure. From the second study we observed that the physical manipulation failed to help apprentices understanding the way the forces interact with each other. To overcome this issue, our solution was to combine an intuitive representation (springs) with a slightly more formal one (arrows) which would show both the nature of the forces and the interaction between them. In this study apprentices used the two representations to collaboratively solve statics problems. Even though apprentices had difficulties in interpreting the two representations, there were cases in which they gained a correct intuition of statics principles from them. In this thesis, besides describing the designed system and the studies, implications for future directions are discussed

Teaching Analytics: Towards Automatic Extraction of Orchestration Graphs Using Wearable Sensors

"Teaching analytics" is the application of learning analytics techniques to understand teaching and learning processes, and eventually enable supportive interventions. However, in the case of (often, half-improvised) teaching in face-to-face classrooms, such interventions would require first an understanding of what the teacher actually did, as the starting point for teacher reflection and inquiry. Currently, such teacher enactment characterization requires costly manual coding by researchers. This paper presents a case study exploring the potential of machine learning techniques to automatically extract teaching actions during classroom enactment, from five data sources collected using wearable sensors (eye-tracking, EEG, accelerometer, audio and video). Our results highlight the feasibility of this approach, with high levels of accuracy in determining the social plane of interaction (90%, k=0.8). The reliable detection of concrete teaching activity (e.g., explanation vs. questioning) accurately still remains challenging (67%, k=0.56), a fact that will prompt further research on multimodal features and models for teaching activity extraction, as well as the collection of a larger multimodal dataset to improve the accuracy and generalizability of these methods

Design-activity-sequence: A case study and polyphonic analysis of learning in a digital design thinking workshop

In this case study, we report on the outcomes of a one-day workshop on design thinking attended by participants from the Computer-Supported Collaborative Learning conference in Philadelphia in 2017. We highlight the interactions between the workshop design, structured as a design thinking process around the design of a digital environment for design thinking, and the diverse backgrounds and interests of its participants. Data from in-workshop reflections and post-workshop interviews were analyzed using a novel set of analytical approaches, a combination the facilitators made by possible by welcoming participants as coresearchers

Analysing, visualising and supporting collaborative learning using interactive tabletops

The key contribution of this thesis is a novel approach to design, implement and evaluate the conceptual and technological infrastructure that captures student’s activity at interactive tabletops and analyses these data through Interaction Data Analytics techniques to provide support to teachers by enhancing their awareness of student’s collaboration. To achieve the above, this thesis presents a series of carefully designed user studies to understand how to capture, analyse and distil indicators of collaborative learning. We perform this in three steps: the exploration of the feasibility of the approach, the construction of a novel solution and the execution of the conceptual proposal, both under controlled conditions and in the wild. A total of eight datasets were analysed for the studies that are described in this thesis. This work pioneered in a number of areas including the application of data mining techniques to study collaboration at the tabletop, a plug-in solution to add user-identification to a regular tabletop using a depth sensor and the first multi-tabletop classroom used to run authentic collaborative activities associated with the curricula. In summary, while the mechanisms, interfaces and studies presented in this thesis were mostly explored in the context of interactive tabletops, the findings are likely to be relevant to other forms of groupware and learning scenarios that can be implemented in real classrooms. Through the mechanisms, the studies conducted and our conceptual framework this thesis provides an important research foundation for the ways in which interactive tabletops, along with data mining and visualisation techniques, can be used to provide support to improve teacher’s understanding about student’s collaboration and learning in small groups

Supporting Situation Awareness and Workspace Awareness in Co-located Collaborative Systems Involving Dynamic Data

Co-located technologies can provide digital functionality to support collaborative work for multiple users in the same physical space. For example, digital tabletop computers — large interactive tables that allow users to directly interact with the content — can provide the most up-to-date map information while users can work together face-to-face. Combinations of interactive devices, large and small, can also be used together in a multi-device environment to support collaborative work of large groups. This environment allows individuals to utilize different networked devices. In some co-located group work, integrating automation into the available technologies can provide benefits such as automatically switching between different data views or updating map information based on underlying changes in deployed field agents’ locations. However, dynamic changes in the system state can create confusion for users and lead to low situation awareness. Furthermore, with the large size of a tabletop system or with multiple devices being used in the workspace, users may not be able to observe collaborators’ actions due to physical separations between users. Consequently, workspace awareness — knowledge of collaborators’ up-to-the-moment actions — can be difficult to maintain. As a result, users may be frustrated, and the collaboration may become inefficient or ineffective. The current tabletop applications involving dynamic data focus on interaction and information sharing techniques for collaboration rather than providing situation awareness support. Moreover, the situation awareness literature focuses primarily on single-user applications, whereas, the literature in workspace awareness primarily focuses on remote collaborative work. The aim of this dissertation was in supporting situation awareness of system-automated dynamic changes and workspace awareness of collaborators’ actions. The first study (Timeline Study) presented in this dissertation used tabletop systems to investigate supporting situation awareness of automated changes and workspace awareness, and the second study (Callout Bubble Study) followed up to further investigate workspace awareness support in the context of multi-device classrooms. Digital tabletop computers are increasingly being used for complex domains involving dynamic data, such as coastal surveillance and emergency response. Maintaining situation awareness of these changes driven by the system is crucial for quick and appropriate response when problems arise. However, distractors in the environment can make users miss the changes and negatively impact their situation awareness, e.g., the large size of the table and conversations with team members. As interactive event timelines have been shown to improve response time and decision accuracy after interruptions, in this dissertation they were adapted to the context of collaborative tabletop applications to address the lack of situation awareness due to dynamic changes. A user study was conducted to understand design factors related to the adaption and their impacts on situation awareness and workspace awareness. The Callout Bubble Study investigated workspace awareness support for multi-device classrooms, where students were co-located with their personal devices and were connected through a large shared virtual canvas. This context was chosen due to the environment’s ability to support work in large groups and the increasing prevalence of individual devices in co-located collaborative workspaces. By studying another co-located context, this research also sought to combine the lessons learned and provide a set of more generalized design recommendations for co-located technologies. Existing work on workspace awareness focuses on remote collaboration; however, the co-located users may not need all the information beneficial for remote work. This study aimed to balance awareness and distraction to improve students’ workspace awareness maintenance while minimizing distraction to their learning. A Callout Bubble was designed to augment students’ interactions in the shared online workspace, and a field study was conducted to understand how it impacted the students’ collaboration behaviour. Overall, the research presented in this dissertation aimed to investigate information visualizations for supporting situation awareness and workspace awareness in co-located collaborative environments. The contributions included the design of an interactive event timeline and an investigation of how the control placement (how many timelines and where they should be located) and feedback location (whether to display feedback to the group or to individuals when users interact with timelines) factors affected situation awareness. The empirical results revealed that individual timelines were more effective in facilitating situation awareness maintenance and the timelines were used mainly for perceiving new changes. Furthermore, this dissertation contributed in the design of a workspace awareness cue, Callout Bubble. The field study revealed that Callout Bubbles were effective in improving students’ coordination and self-monitoring behaviours, which in turn reduced teachers’ workloads. The dissertation provided overall design lessons learned for supporting awareness in co-located collaborative environments

Investigating Real-time Touchless Hand Interaction and Machine Learning Agents in Immersive Learning Environments

The recent surge in the adoption of new technologies and innovations in connectivity, interaction technology, and artificial realities can fundamentally change the digital world. eXtended Reality (XR), with its potential to bridge the virtual and real environments, creates new possibilities to develop more engaging and productive learning experiences. Evidence is emerging that thissophisticated technology offers new ways to improve the learning process for better student interaction and engagement. Recently, immersive technology has garnered much attention as an interactive technology that facilitates direct interaction with virtual objects in the real world. Furthermore, these virtual objects can be surrogates for real-world teaching resources, allowing for virtual labs. Thus XR could enable learning experiences that would not bepossible in impoverished educational systems worldwide. Interestingly, concepts such as virtual hand interaction and techniques such as machine learning are still not widely investigated in immersive learning. Hand interaction technologies in virtual environments can support the kinesthetic learning pedagogical approach, and the need for its touchless interaction nature hasincreased exceptionally in the post-COVID world. By implementing and evaluating real-time hand interaction technology for kinesthetic learning and machine learning agents for self-guided learning, this research has addressed these underutilized technologies to demonstrate the efficiency of immersive learning. This thesis has explored different hand-tracking APIs and devices to integrate real-time hand interaction techniques. These hand interaction techniques and integrated machine learning agents using reinforcement learning are evaluated with different display devices to test compatibility. The proposed approach aims to provide self-guided, more productive, and interactive learning experiences. Further, this research has investigated ethics, privacy, and security issues in XR and covered the future of immersive learning in the Metaverse.<br/

Investigating Real-time Touchless Hand Interaction and Machine Learning Agents in Immersive Learning Environments

The recent surge in the adoption of new technologies and innovations in connectivity, interaction technology, and artificial realities can fundamentally change the digital world. eXtended Reality (XR), with its potential to bridge the virtual and real environments, creates new possibilities to develop more engaging and productive learning experiences. Evidence is emerging that thissophisticated technology offers new ways to improve the learning process for better student interaction and engagement. Recently, immersive technology has garnered much attention as an interactive technology that facilitates direct interaction with virtual objects in the real world. Furthermore, these virtual objects can be surrogates for real-world teaching resources, allowing for virtual labs. Thus XR could enable learning experiences that would not bepossible in impoverished educational systems worldwide. Interestingly, concepts such as virtual hand interaction and techniques such as machine learning are still not widely investigated in immersive learning. Hand interaction technologies in virtual environments can support the kinesthetic learning pedagogical approach, and the need for its touchless interaction nature hasincreased exceptionally in the post-COVID world. By implementing and evaluating real-time hand interaction technology for kinesthetic learning and machine learning agents for self-guided learning, this research has addressed these underutilized technologies to demonstrate the efficiency of immersive learning. This thesis has explored different hand-tracking APIs and devices to integrate real-time hand interaction techniques. These hand interaction techniques and integrated machine learning agents using reinforcement learning are evaluated with different display devices to test compatibility. The proposed approach aims to provide self-guided, more productive, and interactive learning experiences. Further, this research has investigated ethics, privacy, and security issues in XR and covered the future of immersive learning in the Metaverse.<br/

Tangible Interfaces for Learning:Training Spatial Skills in Vocational Classrooms

There have been many claims that Tangible User Interfaces (TUIs) can have a positive impact on learning. Alleged benefits include increasing usability, improving engagement and collaboration of students, and providing a better perception of the task, especially spatial ones. However, there exists little empirical data to back up these claims. Moreover, for all their potential benefits for learning, TUIs are still scarcely used in schools. This thesis explores these two issues in the specific context of vocational education and training of carpenter apprentices. The learning objectives concern spatial skills and in particular, the mapping between 2D and 3D representations. We study (1) whether TUIs can support the training of spatial skills, and if so what features allow them to do so, and (2) what kinds of classroom pedagogical scenarios TUIs can support. We follow a design-based research approach and run empirical studies, mostly in classrooms. The contributions of this thesis touch on three research domains: 1. Spatial skills. Our results show that TUIs can help teach spatial skills to carpenter apprentices. The tangible nature of TUIs can help the learner relate multiple representations of an object, especially for difficult problems. It can also lower the barrier to entry into a learning domain for beginners. 2. Learning with TUIs. According to our results, TUIs can benefit learning, but the mere fact of using TUIs does not guarantee learning. Instead, special attention needs to be given to the design of the TUI. Small design variations, such as the physical correspondence between the tangible object and its virtual representations, or the type and timing of feedback given to the user, can have a significant impact on learning. 3. Classroom technologies and orchestration. We explore several classroom pedagogical scenarios that TUIs can support. The most promising one is to use a TUI as part of a hybrid classroom learning activity that includes both TUI and non TUI steps. Additionally, we devise two ways to promote the integration of TUIs in classroom. First, we introduce 5 design principles that reduce the classroom orchestration load. Second, we show how new web technologies can be used to deploy TUIs in schools at a lower cost

Investigation and development of a tangible technology framework for highly complex and abstract concepts

The ubiquitous integration of computer-supported learning tools within the educational domain has led educators to continuously seek effective technological platforms for teaching and learning. Overcoming the inherent limitations of traditional educational approaches, interactive and tangible computing platforms have consequently garnered increased interest in the pursuit of embedding active learning pedagogies within curricula. However, whilst Tangible User Interface (TUI) systems have been successfully developed to edutain children in various research contexts, TUI architectures have seen limited deployment towards more advanced educational pursuits. Thus, in contrast to current domain research, this study investigates the effectiveness and suitability of adopting TUI systems for enhancing the learning experience of abstract and complex computational science and technology-based concepts within higher educational institutions (HEI)s. Based on the proposal of a contextually apt TUI architecture, the research describes the design and development of eight distinct TUI frameworks embodying innovate interactive paradigms through tabletop peripherals, graphical design factors, and active tangible manipulatives. These computationally coupled design elements are evaluated through summative and formative experimental methodologies for their ability to aid in the effective teaching and learning of diverse threshold concepts experienced in computational science. In addition, through the design and adoption of a technology acceptance model for educational technology (TAM4Edu), the suitability of TUI frameworks in HEI education is empirically evaluated across a myriad of determinants for modelling students’ behavioural intention. In light of the statistically significant results obtained in both academic knowledge gain (μ = 25.8%) and student satisfaction (μ = 12.7%), the study outlines the affordances provided through TUI design for various constituents of active learning theories and modalities. Thus, based on an empirical and pedagogical analyses, a set of design guidelines is defined within this research to direct the effective development of TUI design elements for teaching and learning abstract threshold concepts in HEI adaptations