467 research outputs found

    Virtual and Mixed Reality in Telerobotics: A Survey

    Get PDF

    Development and evaluation of mixed reality-enhanced robotic systems for intuitive tele-manipulation and telemanufacturing tasks in hazardous conditions

    Get PDF
    In recent years, with the rapid development of space exploration, deep-sea discovery, nuclear rehabilitation and management, and robotic-assisted medical devices, there is an urgent need for humans to interactively control robotic systems to perform increasingly precise remote operations. The value of medical telerobotic applications during the recent coronavirus pandemic has also been demonstrated and will grow in the future. This thesis investigates novel approaches to the development and evaluation of a mixed reality-enhanced telerobotic platform for intuitive remote teleoperation applications in dangerous and difficult working conditions, such as contaminated sites and undersea or extreme welding scenarios. This research aims to remove human workers from the harmful working environments by equipping complex robotic systems with human intelligence and command/control via intuitive and natural human-robot- interaction, including the implementation of MR techniques to improve the user's situational awareness, depth perception, and spatial cognition, which are fundamental to effective and efficient teleoperation. The proposed robotic mobile manipulation platform consists of a UR5 industrial manipulator, 3D-printed parallel gripper, and customized mobile base, which is envisaged to be controlled by non-skilled operators who are physically separated from the robot working space through an MR-based vision/motion mapping approach. The platform development process involved CAD/CAE/CAM and rapid prototyping techniques, such as 3D printing and laser cutting. Robot Operating System (ROS) and Unity 3D are employed in the developing process to enable the embedded system to intuitively control the robotic system and ensure the implementation of immersive and natural human-robot interactive teleoperation. This research presents an integrated motion/vision retargeting scheme based on a mixed reality subspace approach for intuitive and immersive telemanipulation. An imitation-based velocity- centric motion mapping is implemented via the MR subspace to accurately track operator hand movements for robot motion control, and enables spatial velocity-based control of the robot tool center point (TCP). The proposed system allows precise manipulation of end-effector position and orientation to readily adjust the corresponding velocity of maneuvering. A mixed reality-based multi-view merging framework for immersive and intuitive telemanipulation of a complex mobile manipulator with integrated 3D/2D vision is presented. The proposed 3D immersive telerobotic schemes provide the users with depth perception through the merging of multiple 3D/2D views of the remote environment via MR subspace. The mobile manipulator platform can be effectively controlled by non-skilled operators who are physically separated from the robot working space through a velocity-based imitative motion mapping approach. Finally, this thesis presents an integrated mixed reality and haptic feedback scheme for intuitive and immersive teleoperation of robotic welding systems. By incorporating MR technology, the user is fully immersed in a virtual operating space augmented by real-time visual feedback from the robot working space. The proposed mixed reality virtual fixture integration approach implements hybrid haptic constraints to guide the operator’s hand movements following the conical guidance to effectively align the welding torch for welding and constrain the welding operation within a collision-free area. Overall, this thesis presents a complete tele-robotic application space technology using mixed reality and immersive elements to effectively translate the operator into the robot’s space in an intuitive and natural manner. The results are thus a step forward in cost-effective and computationally effective human-robot interaction research and technologies. The system presented is readily extensible to a range of potential applications beyond the robotic tele- welding and tele-manipulation tasks used to demonstrate, optimise, and prove the concepts

    Recent Advancements in Augmented Reality for Robotic Applications: A Survey

    Get PDF
    Robots are expanding from industrial applications to daily life, in areas such as medical robotics, rehabilitative robotics, social robotics, and mobile/aerial robotics systems. In recent years, augmented reality (AR) has been integrated into many robotic applications, including medical, industrial, human–robot interactions, and collaboration scenarios. In this work, AR for both medical and industrial robot applications is reviewed and summarized. For medical robot applications, we investigated the integration of AR in (1) preoperative and surgical task planning; (2) image-guided robotic surgery; (3) surgical training and simulation; and (4) telesurgery. AR for industrial scenarios is reviewed in (1) human–robot interactions and collaborations; (2) path planning and task allocation; (3) training and simulation; and (4) teleoperation control/assistance. In addition, the limitations and challenges are discussed. Overall, this article serves as a valuable resource for working in the field of AR and robotic research, offering insights into the recent state of the art and prospects for improvement

    Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces

    Get PDF
    This paper contributes to a taxonomy of augmented reality and robotics based on a survey of 460 research papers. Augmented and mixed reality (AR/MR) have emerged as a new way to enhance human-robot interaction (HRI) and robotic interfaces (e.g., actuated and shape-changing interfaces). Recently, an increasing number of studies in HCI, HRI, and robotics have demonstrated how AR enables better interactions between people and robots. However, often research remains focused on individual explorations and key design strategies, and research questions are rarely analyzed systematically. In this paper, we synthesize and categorize this research field in the following dimensions: 1) approaches to augmenting reality; 2) characteristics of robots; 3) purposes and benefits; 4) classification of presented information; 5) design components and strategies for visual augmentation; 6) interaction techniques and modalities; 7) application domains; and 8) evaluation strategies. We formulate key challenges and opportunities to guide and inform future research in AR and robotics

    An Investigation of Skill Acquisition under Conditions of Augmented Reality

    Get PDF
    Augmented reality is a virtual environment that integrates rendered content with the experience of the real world. There is evidence suggesting that augmented reality provides for important spatial constancy of objects relative to the real world coordinate system and that this quality contributes to rapid skill acquisition. The qualities of simulation, through the use of augmented reality, may be incorporated into actual job activities to produce a condition of just-in-time learning. This may make possible the rapid acquisition of information and reliable completion of novel or infrequently performed tasks by individuals possessing a basic skill-set. The purpose of this research has been to investigate the degree to which the acquisition of a skill is enhanced through the use of an augmented reality training device

    Context-aware learning for robot-assisted endovascular catheterization

    Get PDF
    Endovascular intervention has become a mainstream treatment of cardiovascular diseases. However, multiple challenges remain such as unwanted radiation exposures, limited two-dimensional image guidance, insufficient force perception and haptic cues. Fast evolving robot-assisted platforms improve the stability and accuracy of instrument manipulation. The master-slave system also removes radiation to the operator. However, the integration of robotic systems into the current surgical workflow is still debatable since repetitive, easy tasks have little value to be executed by the robotic teleoperation. Current systems offer very low autonomy, potential autonomous features could bring more benefits such as reduced cognitive workloads and human error, safer and more consistent instrument manipulation, ability to incorporate various medical imaging and sensing modalities. This research proposes frameworks for automated catheterisation with different machine learning-based algorithms, includes Learning-from-Demonstration, Reinforcement Learning, and Imitation Learning. Those frameworks focused on integrating context for tasks in the process of skill learning, hence achieving better adaptation to different situations and safer tool-tissue interactions. Furthermore, the autonomous feature was applied to next-generation, MR-safe robotic catheterisation platform. The results provide important insights into improving catheter navigation in the form of autonomous task planning, self-optimization with clinical relevant factors, and motivate the design of intelligent, intuitive, and collaborative robots under non-ionizing image modalities.Open Acces

    A Magnetic Laser Scanner for Endoscopic Microsurgery

    Get PDF
    Laser scanners increase the quality of the laser microsurgery enabling fast tissue ablation with less thermal damage. Such technology is part of state-of-the-art freebeam surgical laser systems. However, laser scanning has not been incorporated to fiber-based lasers yet. This is a combination that has potential to greatly improve the quality of laser microsurgeries on difficult-to-reach surgical sites. Current fiberbased tissue ablations are performed in contact with the tissue, resulting in excessive thermal damage to healthy tissue in the vicinity of the ablated tissue. This is far from ideal for delicate microsurgeries, which require high-quality tissue incisions without any thermal damage or char formation. However, the possibility to perform scanning laser microsurgery in confined workspaces is restricted by the large size of currently available actuators, which are typically located outside the patient and require direct line-of-sight to the microsurgical area. Thus, it is desired to have the laser scanning feature in an endoscopic system to provide high incision quality in hard-to-reach surgical sites. This thesis aims to introduce a new endoscopic laser scanner to perform 2D position control and high-speed scanning of a fiber-based laser for operation in narrow workspaces. It also presents a technology concept aimed at assisting in incision depth control during soft-tissue microsurgery. The main objective of the work presented in this thesis is to bring the benefits of free-beam lasers to laser-based endoscopic surgery by designing an end-effector module to be placed at the distal tip of a flexible robot arm. To this end, the design and control of a magnetic laser scanner for endoscopic microsurgeries is presented. The system involves an optical fiber, electromagnetic coils, a permanent magnet and optical lenses in a compact system for laser beam deflection. The actuation mechanism is based on the interaction between the electromagnetic field and the permanent magnets. A cantilevered optical fiber is bended with the magnetic field induced by the electromagnetic coils by creating magnetic torque on the permanent magnet. The magnetic laser scanner provides 2D position control and high-speed scanning of the laser beam. The device includes laser focusing optics to allow non-contact incisions. A proof-of-concept device was manufactured and evaluated. It includes four electromagnetic coils and two plano-convex lenses, and has an external diameter of 13 mm. A 4 74 mm2 scanning range was achieved at a 30 mm distance from the scanner tip. Computer-controlled trajectory executions demonstrated repeatable results with 75 m precision for challenging trajectories. Frequency analysis demonstrated stable response up to 33 Hz for 3 dB limit. The system is able to ablate tissue substitutes with a 1940 nm wavelength surgical diode laser. Tablet-based control interface has been developed for intuitive teleoperation. The performance of the proof-of-concept device is analysed through control accuracy and usability studies. Teleoperation user trials consisting in trajectory-following tasks involved 12 subjects. Results demonstrated users could achieve an accuracy of 39 m with the magnetic laser scanner system. For minimally invasive surgeries, it is essential to perform accurate laser position control. Therefore, a model based feed-forward position control of magnetic laser scanner was developed for automated trajectory executions. First, the dynamical model of the system was identified using the electromagnets current (input) and the laser position (output). Then, the identified model was used to perform feedforward control. Validation experiments were performed with different trajectory types, frequencies and amplitudes. Results showed that desired trajectories can be executed in high-speed scanning mode with less than 90 m (1.4 mrad bending angle) accuracy for frequencies up to 15 Hz. State-of-the-art systems do not provide incision depth control, thus the quality of such control relies entirely on the experience and visual perception of the surgeons. In order to provide intuitive incision depth control in endoscopic microsurgeries, the concept of a technology was presented for the automated laser incisions given a desired depth based on a commercial laser scanner. The technology aims at automatically controlling laser incisions based on high-level commands from the surgeon, i.e. desired incision shape, length and depth. A feed-forward controller provides (i) commands to the robotic laser system and (ii) regulates the parameters of the laser source to achieve the desired results. The controller for the incision depth is extracted from experimental data. The required energy density and the number of passes are calculated to reach the targeted depth. Experimental results demonstrate that targeted depths can be achieved with \ub1100 m accuracy, which proves the feasibility of this approach. The proposed technology has the potential to facilitate the surgeon\u2019s control over laser incisions. The magnetic laser scanner enables high-speed laser positioning in narrow and difficult-to-reach workspaces, promising to bring the benefits of scanning laser microsurgery to flexible endoscopic procedures. In addition, the same technology can be potentially used for optical fiber based imaging, enabling for example the creation of new family of scanning endoscopic OCT or hyperspectral probes
    • 

    corecore