Recent Advancements in Augmented Reality for Robotic Applications: A Survey

Robots are expanding from industrial applications to daily life, in areas such as medical robotics, rehabilitative robotics, social robotics, and mobile/aerial robotics systems. In recent years, augmented reality (AR) has been integrated into many robotic applications, including medical, industrial, human–robot interactions, and collaboration scenarios. In this work, AR for both medical and industrial robot applications is reviewed and summarized. For medical robot applications, we investigated the integration of AR in (1) preoperative and surgical task planning; (2) image-guided robotic surgery; (3) surgical training and simulation; and (4) telesurgery. AR for industrial scenarios is reviewed in (1) human–robot interactions and collaborations; (2) path planning and task allocation; (3) training and simulation; and (4) teleoperation control/assistance. In addition, the limitations and challenges are discussed. Overall, this article serves as a valuable resource for working in the field of AR and robotic research, offering insights into the recent state of the art and prospects for improvement

A Review of Pneumatic Actuators Used for the Design of Medical Simulators and Medical Tools

International audienc

VR welding kit: welding training simulation in mobile virtual reality using multiple marker tracking method

Welding simulation design using virtual reality (VR) is a challenge, as numerous developments and research in the mechanical engineering fields are involved. One of the key challenges is the improvement of realism by considering a mixed system of real and virtual equipment. A conceptual design and research management framework is currently lacking which leveraging the combination of VR and marker tracking techniques. This study seeks to examine and evaluating the use of mobile VR in welding training and how multiple markers tracking methods can be incorporated to overcome the current problems in VR for welding training simulation. In this study, the VR Welding Kit application is created by utilizing the Vuforia tracking engine to provide an alternative interaction for mobile devices. The results of the experiment revealed a benchmark comparison with Oculus Quest, the high-end VR system, to investigate the efficiency of the proposed multiple marker interaction technique. Performance for both devices was recorded. The System Usability Scales (SUS) have also been used to obtain users' acceptance rates using these devices. The Simulator Sickness Questionnaire (SSQ) was used to assess the cybersickness of participants. The performance results show that mobile VR have a moderate gap completion time in seconds if compared to Oculus Quest. The SUS scored a satisfactory result which is 73.33. Besides, SSQ surveys result shows that most of the participant felt the simulation sickness was minimal

Multimodal Mixed Reality Impact on a Hand Guiding Task with a Holographic Cobot

In the context of industrial production, a worker that wants to program a robot using the hand-guidance technique needs that the robot is available to be programmed and not in operation. This means that production with that robot is stopped during that time. A way around this constraint is to perform the same manual guidance steps on a holographic representation of the digital twin of the robot, using augmented reality technologies. However, this presents the limitation of a lack of tangibility of the visual holograms that the user tries to grab. We present an interface in which some of the tangibility is provided through ultrasound-based mid-air haptics actuation. We report a user study that evaluates the impact that the presence of such haptic feedback may have on a pick-and-place task of the wrist of a holographic robot arm which we found to be beneficial

Prefrontal cortex activation upon a demanding virtual hand-controlled task: A new frontier for neuroergonomics

open9noFunctional near-infrared spectroscopy (fNIRS) is a non-invasive vascular-based functional neuroimaging technology that can assess, simultaneously from multiple cortical areas, concentration changes in oxygenated-deoxygenated hemoglobin at the level of the cortical microcirculation blood vessels. fNIRS, with its high degree of ecological validity and its very limited requirement of physical constraints to subjects, could represent a valid tool for monitoring cortical responses in the research field of neuroergonomics. In virtual reality (VR) real situations can be replicated with greater control than those obtainable in the real world. Therefore, VR is the ideal setting where studies about neuroergonomics applications can be performed. The aim of the present study was to investigate, by a 20-channel fNIRS system, the dorsolateral/ventrolateral prefrontal cortex (DLPFC/VLPFC) in subjects while performing a demanding VR hand-controlled task (HCT). Considering the complexity of the HCT, its execution should require the attentional resources allocation and the integration of different executive functions. The HCT simulates the interaction with a real, remotely-driven, system operating in a critical environment. The hand movements were captured by a high spatial and temporal resolution 3-dimensional (3D) hand-sensing device, the LEAP motion controller, a gesture-based control interface that could be used in VR for tele-operated applications. Fifteen University students were asked to guide, with their right hand/forearm, a virtual ball (VB) over a virtual route (VROU) reproducing a 42 m narrow road including some critical points. The subjects tried to travel as long as possible without making VB fall. The distance traveled by the guided VB was 70.2 ± 37.2 m. The less skilled subjects failed several times in guiding the VB over the VROU. Nevertheless, a bilateral VLPFC activation, in response to the HCT execution, was observed in all the subjects. No correlation was found between the distance traveled by the guided VB and the corresponding cortical activation. These results confirm the suitability of fNIRS technology to objectively evaluate cortical hemodynamic changes occurring in VR environments. Future studies could give a contribution to a better understanding of the cognitive mechanisms underlying human performance either in expert or non-expert operators during the simulation of different demanding/fatiguing activities.openCarrieri, Marika; Petracca, Andrea; Lancia, Stefania; Basso Moro, Sara; Brigadoi, Sabrina; Spezialetti, Matteo; Ferrari, Marco; Placidi, Giuseppe; Quaresima, ValentinaCarrieri, Marika; Petracca, Andrea; Lancia, Stefania; BASSO MORO, Sara; Brigadoi, Sabrina; Spezialetti, Matteo; Ferrari, Marco; Placidi, Giuseppe; Quaresima, Valentin

A Scoping Review on Virtual Reality-Based Industrial Training

The fourth industrial revolution has forced most companies to technologically evolve, applying new digital tools, so that their workers can have the necessary skills to face changing work environments. This article presents a scoping review of the literature on virtual reality-based training systems. The methodology consisted of four steps, which pose research questions, document search, paper selection, and data extraction. From a total of 350 peer-reviewed database articles, such as SpringerLink, IEEEXplore, MDPI, Scopus, and ACM, 44 were eventually chosen, mostly using the virtual reality haptic glasses and controls from Oculus Rift and HTC VIVE. It was concluded that, among the advantages of using this digital tool in the industry, is the commitment, speed, measurability, preservation of the integrity of the workers, customization, and cost reduction. Even though several research gaps were found, virtual reality is presented as a present and future alternative for the efficient training of human resources in the industrial field.This work was supported by Instituto Superior Tecnológico Victoria Vásconez Cuvi. The authors appreciate the opportunity to analyze topics related to this paper. The authors must also recognize the supported bringing by Universidad Tecnica de Ambato (UTA) and their Research and Development Department (DIDE) under project CONIN-P-256-2019, and SENESCYT by grants “Convocatoria Abierta 2011” and “Convocatoria Abierta 2013”

Model-based training of manual procedures in automated production systems

Maintenance engineers deal with increasingly complex automated production systems (aPSs). Such systems are characterized by an increasing computerization or the addition of robots that collaborate with human workers. The effects of changing or replacing components of such systems are difficult to assess since there are complex interdependencies between process parameters and the state of the components. This paper proposes a model-based training system that visualizes these interdependencies using domain-independent SysML models. The training system consists of a virtual training system for initial training and an online support system for assistance during maintenance or changeover procedures. Both systems use structural SysML models to visualize the state of the machine at a certain step of a procedure. An evaluation of the system in a changeover procedure against a paper-based manual showed promising results regarding effectiveness, usability and attractiveness.Comment: 25 pages, https://www.sciencedirect.com/science/article/pii/S095741581830080

Exploring virtual reality to improve engineering students' spatial abilities pilot study

A Virtual Reality pilot study is conducted to improve the spatial ability of engineering students based on solid geometry scenarios. The investigation focused on the Graphic Expression and Computer-Aided Design (GECAD) course, specifically on the study of the spatial abilities developed and the assessment of the academic results in the solid geometry module. A total of 20 participants completed three activities (6 h) in an immersive virtual learning environment (IVLE), using head-mounted display (HMD) glasses. Modeling exercises of three-dimensional geometric shapes are proposed, based on concepts of solid geometry. The scenarios are built step by step and the students can regulate the progress between stages while observing the geometric components at the scale and in the point of view they wish. Beyond academic results, the assessment of student improvement is based on spatial abilities tests: the Differential Aptitude Test: Spatial Relations Subset DAT-SR, Purdue Spatial Visualisation Test: Rotations PSVT:R and Mental Cutting Test MCT. Those tests are applied for evaluating different skills: mental folding, mental rotation and section by a plane. In summary, a methodology is proposed developing activities in an (IVLE) with 3D modelling software applied in solid geometry, in order to promote the development of spatial ability (SA). Spatial abilities are measured before and after the classroom activities and looking for correlations between the spatial perception tests (DAT:SR, PSVT:R and MCT) and academic results in solid geometry. In addition, we also wish to determine the students' opinion with regard to the proposed activities. The results obtained confirm the interest in using IVLE to develop spatial abilities in engineering students. Substantial increases of 10,9% in DAT:SR, 8,8 % in PSVT:R and 9,5% in MCT between pre- and post-tests were found. Moreover, the students' opinion of IVLE/HMD activities is positive. The methodology can be summarized in the following steps: 1. Students take the DAT:SR , PSVT:R and MCT prior to the activities. They also answer the survey on other variables that can affect SA (1 h). 2. The students individually complete the exercises with the 3D modelling software SolidWorks (10 h). 3. The IVLE activities consist of the guided reading by the professor of the completed exercise. The professor addresses the concepts of solid geometry used in each step. The students have a few minutes to view with HMD the animation showing the construction of the geometric shape, and once the representation is finished, they can move freely throughout the scenario, using the keyboard options (6 h). 4. Students solve the (DAT:SR, PSVT:R and MCT after the IVLE activities. At the end, the groups answer the satisfaction survey (1 h). 5. All the students are evaluated on their knowledge of the solid geometry contents by means of a test and 3D modelling exercises similar to those done in class and those described in the IVRL (1h). 6. Finally, the analysis of the spatial abilities test data, the controlled variables survey, satisfaction surveys and the academic results obtained in the solid geometry module enable us to examine the correlations and the strongest determining factors in order to obtain good academic results and propose IVLE activities to improve the levels of spatial ability obtained on the tests. This paper describes the exploratory methodology used and its results.Postprint (published version