Sensory Manipulation as a Countermeasure to Robot Teleoperation Delays: System and Evidence

In the field of robotics, robot teleoperation for remote or hazardous environments has become increasingly vital. A major challenge is the lag between command and action, negatively affecting operator awareness, performance, and mental strain. Even with advanced technology, mitigating these delays, especially in long-distance operations, remains challenging. Current solutions largely focus on machine-based adjustments. Yet, there's a gap in using human perceptions to improve the teleoperation experience. This paper presents a unique method of sensory manipulation to help humans adapt to such delays. Drawing from motor learning principles, it suggests that modifying sensory stimuli can lessen the perception of these delays. Instead of introducing new skills, the approach uses existing motor coordination knowledge. The aim is to minimize the need for extensive training or complex automation. A study with 41 participants explored the effects of altered haptic cues in delayed teleoperations. These cues were sourced from advanced physics engines and robot sensors. Results highlighted benefits like reduced task time and improved perceptions of visual delays. Real-time haptic feedback significantly contributed to reduced mental strain and increased confidence. This research emphasizes human adaptation as a key element in robot teleoperation, advocating for improved teleoperation efficiency via swift human adaptation, rather than solely optimizing robots for delay adjustment.Comment: Submitted to Scientific Report

Development and evaluation of mixed reality-enhanced robotic systems for intuitive tele-manipulation and telemanufacturing tasks in hazardous conditions

In recent years, with the rapid development of space exploration, deep-sea discovery, nuclear rehabilitation and management, and robotic-assisted medical devices, there is an urgent need for humans to interactively control robotic systems to perform increasingly precise remote operations. The value of medical telerobotic applications during the recent coronavirus pandemic has also been demonstrated and will grow in the future. This thesis investigates novel approaches to the development and evaluation of a mixed reality-enhanced telerobotic platform for intuitive remote teleoperation applications in dangerous and difficult working conditions, such as contaminated sites and undersea or extreme welding scenarios. This research aims to remove human workers from the harmful working environments by equipping complex robotic systems with human intelligence and command/control via intuitive and natural human-robot- interaction, including the implementation of MR techniques to improve the user's situational awareness, depth perception, and spatial cognition, which are fundamental to effective and efficient teleoperation. The proposed robotic mobile manipulation platform consists of a UR5 industrial manipulator, 3D-printed parallel gripper, and customized mobile base, which is envisaged to be controlled by non-skilled operators who are physically separated from the robot working space through an MR-based vision/motion mapping approach. The platform development process involved CAD/CAE/CAM and rapid prototyping techniques, such as 3D printing and laser cutting. Robot Operating System (ROS) and Unity 3D are employed in the developing process to enable the embedded system to intuitively control the robotic system and ensure the implementation of immersive and natural human-robot interactive teleoperation. This research presents an integrated motion/vision retargeting scheme based on a mixed reality subspace approach for intuitive and immersive telemanipulation. An imitation-based velocity- centric motion mapping is implemented via the MR subspace to accurately track operator hand movements for robot motion control, and enables spatial velocity-based control of the robot tool center point (TCP). The proposed system allows precise manipulation of end-effector position and orientation to readily adjust the corresponding velocity of maneuvering. A mixed reality-based multi-view merging framework for immersive and intuitive telemanipulation of a complex mobile manipulator with integrated 3D/2D vision is presented. The proposed 3D immersive telerobotic schemes provide the users with depth perception through the merging of multiple 3D/2D views of the remote environment via MR subspace. The mobile manipulator platform can be effectively controlled by non-skilled operators who are physically separated from the robot working space through a velocity-based imitative motion mapping approach. Finally, this thesis presents an integrated mixed reality and haptic feedback scheme for intuitive and immersive teleoperation of robotic welding systems. By incorporating MR technology, the user is fully immersed in a virtual operating space augmented by real-time visual feedback from the robot working space. The proposed mixed reality virtual fixture integration approach implements hybrid haptic constraints to guide the operator’s hand movements following the conical guidance to effectively align the welding torch for welding and constrain the welding operation within a collision-free area. Overall, this thesis presents a complete tele-robotic application space technology using mixed reality and immersive elements to effectively translate the operator into the robot’s space in an intuitive and natural manner. The results are thus a step forward in cost-effective and computationally effective human-robot interaction research and technologies. The system presented is readily extensible to a range of potential applications beyond the robotic tele- welding and tele-manipulation tasks used to demonstrate, optimise, and prove the concepts

Design and Development of Assistive Robots for Close Interaction with People with Disabilities

People with mobility and manipulation impairments wish to live and perform tasks as independently as possible; however, for many tasks, compensatory technology does not exist, to do so. Assistive robots have the potential to address this need. This work describes various aspects of the development of three novel assistive robots: the Personal Mobility and Manipulation Appliance (PerMMA), the Robotic Assisted Transfer Device (RATD), and the Mobility Enhancement Robotic Wheelchair (MEBot). PerMMA integrates mobility with advanced bi-manual manipulation to assist people with both upper and lower extremity impairments. The RATD is a wheelchair mounted robotic arm that can lift higher payloads and its primary aim is to assist caregivers of people who cannot independently transfer from their electric powered wheelchair to other surfaces such as a shower bench or toilet. MEBot is a wheeled robot that has highly reconfigurable kinematics, which allow it to negotiate challenging terrain, such as steep ramps, gravel, or stairs. A risk analysis was performed on all three robots which included a Fault Tree Analysis (FTA) and a Failure Mode Effect Analysis (FMEA) to identify potential risks and inform strategies to mitigate them. Identified risks or PerMMA include dropping sharp or hot objects. Critical risks identified for RATD included tip over, crush hazard, and getting stranded mid-transfer, and risks for MEBot include getting stranded on obstacles and tip over. Lastly, several critical factors, such as early involvement of people with disabilities, to guide future assistive robot design are presented

Surgical Subtask Automation for Intraluminal Procedures using Deep Reinforcement Learning

Intraluminal procedures have opened up a new sub-field of minimally invasive surgery that use flexible instruments to navigate through complex luminal structures of the body, resulting in reduced invasiveness and improved patient benefits. One of the major challenges in this field is the accurate and precise control of the instrument inside the human body. Robotics has emerged as a promising solution to this problem. However, to achieve successful robotic intraluminal interventions, the control of the instrument needs to be automated to a large extent. The thesis first examines the state-of-the-art in intraluminal surgical robotics and identifies the key challenges in this field, which include the need for safe and effective tool manipulation, and the ability to adapt to unexpected changes in the luminal environment. To address these challenges, the thesis proposes several levels of autonomy that enable the robotic system to perform individual subtasks autonomously, while still allowing the surgeon to retain overall control of the procedure. The approach facilitates the development of specialized algorithms such as Deep Reinforcement Learning (DRL) for subtasks like navigation and tissue manipulation to produce robust surgical gestures. Additionally, the thesis proposes a safety framework that provides formal guarantees to prevent risky actions. The presented approaches are evaluated through a series of experiments using simulation and robotic platforms. The experiments demonstrate that subtask automation can improve the accuracy and efficiency of tool positioning and tissue manipulation, while also reducing the cognitive load on the surgeon. The results of this research have the potential to improve the reliability and safety of intraluminal surgical interventions, ultimately leading to better outcomes for patients and surgeons

An early-stage decision-support framework for the implementation of intelligent automation

The constant pressure on manufacturing companies to improve productivity, reduce the lead time and progress in quality requires new technological developments and adoption.The rapid development of smart technology and robotics and autonomous systems (RAS) technology has a profound impact on manufacturing automation and might determine winners and losers of the next generation’s manufacturing competition. Simultaneously, recent smart technology developments in the areas enable an automation response to new production paradigms such as mass customisation and product-lifecycle considerations in the context of Industry 4.0. New paradigms, like mass customisation, increased both the complexity of the tasks and the risk due to smart technology integration. From a manufacturing automation perspective, intelligent automation has been identified as a possible response to arising demands. The presented research aims to support the industrial uptake of intelligent automation into manufacturing businesses by quantifying risks at the early design stage and business case development. An early-stage decision-support framework for the implementation of intelligent automation in manufacturing businesses is presented in this thesis.The framework is informed by an extensive literature review, updated and verified with surveys and workshops to add to the knowledge base due to the rapid development of the associated technologies. A paradigm shift from cost to a risk-modelling perspective is proposed to provide a more flexible and generic approach applicable throughout the current technology landscape. The proposed probabilistic decision-support framework consists of three parts:• A clustering algorithm to identify the manufacturing functions in manual processes from task analysis to mitigate early-stage design uncertainties• A Bayesian Belief Network (BBN) informed by an expert elicitation via the DELPHI method, where the identified functions become the unit of analysis.• A Markov-Chain Monte-Carlo method modelling the effects of uncertainties on the critical success factors to address issues of factor interdependencies after expert elicitation.Based on the overall decision framework a toolbox was developed in Microsoft Excel. Five different case studies are used to test and validate the framework. Evaluation of the results derived from the toolbox from the industrial feedback suggests a positive validation for commercial use. The main contributions to knowledge in the presented thesis arise from the following four points:• Early-stage decision-support framework for business case evaluation of intelligent automation.• Translating manual tasks to automation function via a novel clustering approach• Application of a Markov-Chain Monte-Carlo Method to simulate correlation between decision criteria• Causal relationship among Critical Success Factors has been established from business and technical perspectives.The implications on practise might be promising. The feedback arising from the created tool was promising from the industry, and a practical realisation of the decision-support tool seems to be desired from an industrial point of view.With respect to further work, the decision-support tool might have established a ground to analyse a human task automatically for automation purposes. The established clustering mechanisms and the related attributes could be connected to sensorial data and analyse a manufacturing task autonomously without the subjective input of task analysis experts. To enable such an autonomous process, however, the psychophysiological understanding must be increased in the future.</div

Flexible Automation and Intelligent Manufacturing: The Human-Data-Technology Nexus

This is an open access book. It gathers the first volume of the proceedings of the 31st edition of the International Conference on Flexible Automation and Intelligent Manufacturing, FAIM 2022, held on June 19 – 23, 2022, in Detroit, Michigan, USA. Covering four thematic areas including Manufacturing Processes, Machine Tools, Manufacturing Systems, and Enabling Technologies, it reports on advanced manufacturing processes, and innovative materials for 3D printing, applications of machine learning, artificial intelligence and mixed reality in various production sectors, as well as important issues in human-robot collaboration, including methods for improving safety. Contributions also cover strategies to improve quality control, supply chain management and training in the manufacturing industry, and methods supporting circular supply chain and sustainable manufacturing. All in all, this book provides academicians, engineers and professionals with extensive information on both scientific and industrial advances in the converging fields of manufacturing, production, and automation

Flexible Automation and Intelligent Manufacturing: The Human-Data-Technology Nexus

This is an open access book. It gathers the first volume of the proceedings of the 31st edition of the International Conference on Flexible Automation and Intelligent Manufacturing, FAIM 2022, held on June 19 – 23, 2022, in Detroit, Michigan, USA. Covering four thematic areas including Manufacturing Processes, Machine Tools, Manufacturing Systems, and Enabling Technologies, it reports on advanced manufacturing processes, and innovative materials for 3D printing, applications of machine learning, artificial intelligence and mixed reality in various production sectors, as well as important issues in human-robot collaboration, including methods for improving safety. Contributions also cover strategies to improve quality control, supply chain management and training in the manufacturing industry, and methods supporting circular supply chain and sustainable manufacturing. All in all, this book provides academicians, engineers and professionals with extensive information on both scientific and industrial advances in the converging fields of manufacturing, production, and automation

Minimally Invasive Expeditionary Surgical Care Using Human-Inspired Robots

This technical report serves as an updated collection of subject matter experts on surgical care using human-inspired robotics for human exploration. It is a summary of the Blue Sky Meeting, organized by the Florida Institute for Human and Machine Cognition (IHMC), Pensacola, Florida, and held on October 2-3, 2018. It contains an executive summary, the final report, all of the presentation materials, and an updated reference list

Robot Assisted Laser Osteotomy

In the scope of this thesis world\u27s first robot system was developed, which facilitates osteotomy using laser in arbitrary geometries with an overall accuracy below 0.5mm. Methods of computer and robot assisted surgery were reconsidered and composed to a workflow. Adequate calibration and registration methods are proposed. Further a methodology for transferring geometrically defined cutting trajectories into pulse sequences and optimized execution plans is developed