Automatic extraction of constraints in manipulation tasks for autonomy and interaction

Tasks routinely executed by humans involve sequences of actions performed with high dexterity and coordination. Fully specifying these actions such that a robot could replicate the task is often difficult. Furthermore the uncertainties introduced by the use of different tools or changing configurations demand the specification to be generic, while enhancing the important task aspects, i.e. the constraints. Therefore the first challenge of this thesis is inferring these constraints from repeated demonstrations. In addition humans explaining a task to another person rely on the person's ability to apprehend missing or implicit information. Therefore observations contain user-specific cues, alongside knowledge on performing the task. Thus our second challenge is correlating the task constraints with the user behavior for improving the robot's performance. We address these challenges using a Programming by Demonstration framework. In the first part of the thesis we describe an approach for decomposing demonstrations into actions and extracting task-space constraints as continuous features that apply throughout each action. The constraints consist of: (1) the reference frame for performing manipulation, (2) the variables of interest relative to this frame, allowing a decomposition in force and position control, and (3) a stiffness gain modulating the contribution of force and position. We then extend this approach to asymmetrical bimanual tasks by extracting features that enable arm coordination: the master--slave role that enables precedence, and the motion--motion or force--motion coordination that facilitates the physical interaction through an object. The set of constraints and the time-independent encoding of each action form a task prototype, used to execute the task. In the second part of the thesis we focus on discovering additional features implicit in the demonstrations with respect to two aspects of the teaching interactions: (1) characterizing the user performance and (2) improving the user behavior. For the first goal we assess the skill of the user and implicitly the quality of the demonstrations by using objective task--specific metrics, related directly to the constraints. We further analyze ways of making the user aware of the robot's state during teaching by providing task--related feedback. The feedback has a direct influence on both the teaching efficiency and the user's perception of the interaction. We evaluated our approaches on robotic experiments that encompass daily activities using two 7 degrees of freedom Kuka LWR robotic arms, and a 53 degrees of freedom iCub humanoid robot

Quantifying Demonstration Quality for Robot Learning and Generalization

Learning from Demonstration (LfD) seeks to democratize robotics by enabling diverse end-users to teach robots to perform a task by providing demonstrations. However, most LfD techniques assume users provide optimal demonstrations. This is not always the case in real applications where users are likely to provide demonstrations of varying quality, that may change with expertise and other factors. Demonstration quality plays a crucial role in robot learning and generalization. Hence, it is important to quantify the quality of the provided demonstrations before using them for robot learning. In this paper, we propose quantifying the quality of the demonstrations based on how well they perform in the learned task. We hypothesize that task performance can give an indication of the generalization performance on similar tasks. The proposed approach is validated in a user study (N = 27). Users with different robotics expertise levels were recruited to teach a PR2 robot a generic task (pressing a button) under different task constraints. They taught the robot in two sessions on two different days to capture their teaching behaviour across sessions. The task performance was utilized to classify the provided demonstrations into high-quality and low-quality sets. The results show a significant Pearson correlation coefficient (R = 0.85, p < 0.0001) between the task performance and generalization performance across all participants. We also found that users clustered into two groups: Users who provided high-quality demonstrations from the first session, assigned to the fast-adapters group, and users who provided low-quality demonstrations in the first session and then improved with practice, assigned to the slow-adapters group. These results highlight the importance of quantifying demonstration quality, which can be indicative of the adaptation level of the user to the task

Shared Control Based on Extended Lipschitz Analysis With Application to Human-Superlimb Collaboration

This paper presents a quantitative method to construct voluntary manual control and sensor-based reactive control in human-robot collaboration based on Lipschitz conditions. To collaborate with a human, the robot observes the human's motions and predicts a desired action. This predictor is constructed from data of human demonstrations observed through the robot's sensors. Analysis of demonstration data based on Lipschitz quotients evaluates a) whether the desired action is predictable and b) to what extent the action is predictable. If the quotients are low for all the input-output pairs of demonstration data, a predictor can be constructed with a smooth function. In dealing with human demonstration data, however, the Lipschitz quotients tend to be very high in some situations due to the discrepancy between the information that humans use and the one robots can obtain. This paper a) presents a method for seeking missing information or a new variable that can lower the Lipschitz quotients by adding the new variable to the input space, and b) constructs a human-robot shared control system based on the Lipschitz analysis. Those predictable situations are assigned to the robot's reactive control, while human voluntary control is assigned to those situations where the Lipschitz quotients are high even after the new variable is added. The latter situations are deemed unpredictable and are rendered to the human. This human-robot shared control method is applied to assist hemiplegic patients in a bimanual eating task with a Supernumerary Robotic Limb, which works in concert with an unaffected functional hand

Measures of Performance and Proficiency in Robotic-Assisted Surgery : A Systematic Review

The first author received a research grant from RCS England and Health Education England in November 2021 until present to complete the study.Peer reviewedPostprin

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND: There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula. METHODS: A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias. RESULTS: Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent. CONCLUSIONS: Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901

Neuroplastic Changes Following Brain Ischemia and their Contribution to Stroke Recovery: Novel Approaches in Neurorehabilitation

Ischemic damage to the brain triggers substantial reorganization of spared areas and pathways, which is associated with limited, spontaneous restoration of function. A better understanding of this plastic remodeling is crucial to develop more effective strategies for stroke rehabilitation. In this review article, we discuss advances in the comprehension of post-stroke network reorganization in patients and animal models. We first focus on rodent studies that have shed light on the mechanisms underlying neuronal remodeling in the perilesional area and contralesional hemisphere after motor cortex infarcts. Analysis of electrophysiological data has demonstrated brain-wide alterations in functional connectivity in both hemispheres, well beyond the infarcted area. We then illustrate the potential use of non-invasive brain stimulation (NIBS) techniques to boost recovery. We finally discuss rehabilitative protocols based on robotic devices as a tool to promote endogenous plasticity and functional restoration

Clinical Considerations for Flexible Access Surgery

The expectation of excellence in health care in modern times continues to be challenged. Government and patients alike continue to demand superior health care with excellent treatment outcomes at minimal expense to their time and convenience. Although surgery is the most definitive treatment option in modern medicine, it can be the most demanding both physically and psychologically. The less invasive the procedure offered the more acceptable it has been shown to be to the patient more often with fewer complications attributed and a faster return to health (1). The positive impact of the minimally invasive concept on the healthcare system has been unfathomable. The domino effect created by the early results from laparoscopic surgery was felt not only across the surgical community but also the medical. Across different specialties, alternative novel therapeutic techniques were devised to overcome problems relating to the large operative procedures which struggled to cross over to the laparoscopic approach. The best example of this is in cardiovascular surgery, where image guided endovascular techniques have overcome the need for many of the once extensive operative procedures including the abdominal aortic aneurysm repair and the coronary bypass procedure. The risks and complications from these operative interventions remain significant and are still performed, though far less frequently than in the past. Selective aneurysms as well as primary coronary events are managed routinely through the endovascular technique with surgery being retained for the complex cases or the non-responders. It seems obvious in hindsight that given the choice of a small 5mm groin incision over a large 30cm open chest or abdominal incision which the public would choose, even with the greater long term benefits sometimes favoring the open approaches. Gastrointestinal endoscopy has the potential to move in the same direction. The use of the endoscope as a surgical tool rather than simply an investigative device has only recently been recognized, promoted through the concept of Natural Orifice Translumenal Endoscopic Surgery (NOTES). The technique aims to provide a cosmetic enhancement to routine surgical procedures by creating the access incision within a natural orifice. The endoscope provides the vision and the biopsy channels in-built are able to guide operative instruments to the target site to enable a therapeutic procedure to be undertaken. However, it would be naïve to believe that in the current state NOTES is anything but a fashionable research technique and far from routine clinical use. However, it’s most superior element, which has the potential to extend the boundaries of surgery aside from all else, is the flexibility of the platform. This thesis provides a detailed investigation into the use of the flexible endoscope as a surgical platform. It defines Flexible Access Surgery (FAS) as an all encompassing surgical technique which utilizes flexible platforms at its heart, describes some novel applications representative as examplars of the technique and explores the significant challenges which would hinder clinical translation. These challenges are described and integrated into two novel enhanced mechatronic flexible access surgical platforms which are further validated and trialed within the pre-clinical in-vivo setting as the future of flexible surgery. The major original contributions of this thesis include the description and definition of the flexible access technique with novel clinical applications. The design, construction and validation of a flexible access box simulator for describing flexible endoscopic navigation within a spatial environment highlighting the challenge this encompasses for many clinicians. The instrumental requirements are explored through the evaluation of the force requirements within the preclinical setting and the instrument refinement both in design and practice that can be adopted to optimize the force delivery particularly when relating to novel flexible platform designs. Finally, the thesis describes the integrated clinical design and validation of two enhanced mechatronic flexible access platforms and describes their clinically driven construction through a series of pre-clinical live in-vivo trials. The evolution of each device is described with performance evaluation and clinical exemplars undertaken. The impact of the results presented within this thesis and the potential for further high impact research is centered on the design and integration of future flexible robotic platforms for minimally invasive surgery. The clinical and mechanical requirements essential for optimal clinical performance will enable designs to be more clinically relevant and ultimately more clinically translatable in the future. Defining these requirements has entailed the use of mapping and sensing the relevant tools which has in turn exposed future potential research avenues to be opened into the perhaps more relevant real time evaluation of the surgical workflow, enabling clinical skills to be more reliably quantified during laparoscopic and endoscopic procedures

A Novel Haptic Simulator for Evaluating and Training Salient Force-Based Skills for Laparoscopic Surgery

Laparoscopic surgery has evolved from an \u27alternative\u27 surgical technique to currently being considered as a mainstream surgical technique. However, learning this complex technique holds unique challenges to novice surgeons due to their \u27distance\u27 from the surgical site. One of the main challenges in acquiring laparoscopic skills is the acquisition of force-based or haptic skills. The neglect of popular training methods (e.g., the Fundamentals of Laparoscopic Surgery, i.e. FLS, curriculum) in addressing this aspect of skills training has led many medical skills professionals to research new, efficient methods for haptic skills training. The overarching goal of this research was to demonstrate that a set of simple, simulator-based haptic exercises can be developed and used to train users for skilled application of forces with surgical tools. A set of salient or core haptic skills that underlie proficient laparoscopic surgery were identified, based on published time-motion studies. Low-cost, computer-based haptic training simulators were prototyped to simulate each of the identified salient haptic skills. All simulators were tested for construct validity by comparing surgeons\u27 performance on the simulators with the performance of novices with no previous laparoscopic experience. An integrated, \u27core haptic skills\u27 simulator capable of rendering the three validated haptic skills was built. To examine the efficacy of this novel salient haptic skills training simulator, novice participants were tested for training improvements in a detailed study. Results from the study demonstrated that simulator training enabled users to significantly improve force application for all three haptic tasks. Research outcomes from this project could greatly influence surgical skills simulator design, resulting in more efficient training