Intelligent strategies for mobile robotics in laboratory automation

In this thesis a new intelligent framework is presented for the mobile robots in laboratory automation, which includes: a new multi-floor indoor navigation method is presented and an intelligent multi-floor path planning is proposed; a new signal filtering method is presented for the robots to forecast their indoor coordinates; a new human feature based strategy is proposed for the robot-human smart collision avoidance; a new robot power forecasting method is proposed to decide a distributed transportation task; a new blind approach is presented for the arm manipulations for the robots

Visual Tracking of Instruments in Minimally Invasive Surgery

Reducing access trauma has been a focal point for modern surgery and tackling the challenges that arise from new operating techniques and instruments is an exciting and open area of research. Lack of awareness and control from indirect manipulation and visualization has created a need to augment the surgeon's understanding and perception of how their instruments interact with the patient's anatomy but current methods of achieving this are inaccurate and difficult to integrate into the surgical workflow. Visual methods have the potential to recover the position and orientation of the instruments directly in the reference frame of the observing camera without the need to introduce additional hardware to the operating room and perform complex calibration steps. This thesis explores how this problem can be solved with the fusion of coarse region and fine scale point features to enable the recovery of both the rigid and articulated degrees of freedom of laparoscopic and robotic instruments using only images provided by the surgical camera. Extensive experiments on different image features are used to determine suitable representations for reliable and robust pose estimation. Using this information a novel framework is presented which estimates 3D pose with a region matching scheme while using frame-to-frame optical flow to account for challenges due to symmetry in the instrument design. The kinematic structure of articulated robotic instruments is also used to track the movement of the head and claspers. The robustness of this method was evaluated on calibrated ex-vivo images and in-vivo sequences and comparative studies are performed with state-of-the-art kinematic assisted tracking methods

Deep representation learning for marker-less human posture analysis

This thesis presents a holistic human posture analysis system. The proposed system leverages the state-of-the-art deep learning techniques to feature a comprehensive pipeline. Moreover, a new nonlinear computational layer is proposed to the deep convolutional neural network architectures to incorporate human perception capabilities into the deep learning architectures

Wearable fusion system for assessment of motor function in lesion-symptom mapping studies

Lesion-symptom mapping studies are a critical component of addressing the relationship between brain and behaviour. Recent developments have yielded significant improvements in the imaging and detection of lesion profiles, but the quantification of motor outcomes is still largely performed by subjective and low-resolution standard clinical rating scales. This mismatch means than lesion-symptom mapping studies are limited in scope by scores which lack the necessary accuracy to fully quantify the subcomponents of motor function. The first study conducted aimed to develop a new automated system of motor function which addressed the limitations inherent in the clinical rating scales. A wearable fusion system was designed that included the attachment of inertial sensors to record the kinematics of upper extremity. This was combined with the novel application of mechanomyographic sensors in this field, to enable the quantification of hand/wrist function. Novel outputs were developed for this system which aimed to combine the validity of the clinical rating scales with the high accuracy of measurements possible with a wearable sensor system. This was achieved by the development of a sophisticated classification model which was trained on series of kinematic and myographic measures to classify the clinical rating scale. These classified scores were combined with a series of fine-grained clinical features derived from higher-order sensor metrics. The developed automated system graded the upper-extremity tasks of the Fugl-Meyer Assessment with a mean accuracy of 75\% for gross motor tasks and 66\% for the wrist/hand tasks. This accuracy increased to 85\% and 74\% when distinguishing between healthy and impaired function for each of these tasks. Several clinical features were computed to describe the subcomponents of upper extremity motor function. This fine-grained clinical feature set offers a novel means to complement the low resolution but well-validated standardised clinical rating scales. A second study was performed to utilise the fine-grained clinical feature set calculated in the previous study in a large-scale region-of-interest lesion-symptom mapping study. Statistically significant regions of motor dysfunction were found in the corticospinal tract and the internal capsule, which are consistent with other motor-based lesion-symptom mapping studies. In addition, the cortico-ponto-cerebellar tract was found to be statistically significant when testing with a clinical feature of hand/wrist motor function. This is a novel finding, potentially due to prior studies being limited to quantifying this subcomponent of motor function using standard clinical rating scales. These results indicate the validity and potential of the clinical feature set to provide a more detailed picture of motor dysfunction in lesion-symptom mapping studies.Open Acces

Scene understanding by robotic interactive perception

This thesis presents a novel and generic visual architecture for scene understanding by robotic interactive perception. This proposed visual architecture is fully integrated into autonomous systems performing object perception and manipulation tasks. The proposed visual architecture uses interaction with the scene, in order to improve scene understanding substantially over non-interactive models. Specifically, this thesis presents two experimental validations of an autonomous system interacting with the scene: Firstly, an autonomous gaze control model is investigated, where the vision sensor directs its gaze to satisfy a scene exploration task. Secondly, autonomous interactive perception is investigated, where objects in the scene are repositioned by robotic manipulation. The proposed visual architecture for scene understanding involving perception and manipulation tasks has four components: 1) A reliable vision system, 2) Camera-hand eye calibration to integrate the vision system into an autonomous robot’s kinematic frame chain, 3) A visual model performing perception tasks and providing required knowledge for interaction with scene, and finally, 4) A manipulation model which, using knowledge received from the perception model, chooses an appropriate action (from a set of simple actions) to satisfy a manipulation task. This thesis presents contributions for each of the aforementioned components. Firstly, a portable active binocular robot vision architecture that integrates a number of visual behaviours are presented. This active vision architecture has the ability to verge, localise, recognise and simultaneously identify multiple target object instances. The portability and functional accuracy of the proposed vision architecture is demonstrated by carrying out both qualitative and comparative analyses using different robot hardware configurations, feature extraction techniques and scene perspectives. Secondly, a camera and hand-eye calibration methodology for integrating an active binocular robot head within a dual-arm robot are described. For this purpose, the forward kinematic model of the active robot head is derived and the methodology for calibrating and integrating the robot head is described in detail. A rigid calibration methodology has been implemented to provide a closed-form hand-to-eye calibration chain and this has been extended with a mechanism to allow the camera external parameters to be updated dynamically for optimal 3D reconstruction to meet the requirements for robotic tasks such as grasping and manipulating rigid and deformable objects. It is shown from experimental results that the robot head achieves an overall accuracy of fewer than 0.3 millimetres while recovering the 3D structure of a scene. In addition, a comparative study between current RGB-D cameras and our active stereo head within two dual-arm robotic test-beds is reported that demonstrates the accuracy and portability of our proposed methodology. Thirdly, this thesis proposes a visual perception model for the task of category-wise objects sorting, based on Gaussian Process (GP) classification that is capable of recognising objects categories from point cloud data. In this approach, Fast Point Feature Histogram (FPFH) features are extracted from point clouds to describe the local 3D shape of objects and a Bag-of-Words coding method is used to obtain an object-level vocabulary representation. Multi-class Gaussian Process classification is employed to provide a probability estimate of the identity of the object and serves the key role of modelling perception confidence in the interactive perception cycle. The interaction stage is responsible for invoking the appropriate action skills as required to confirm the identity of an observed object with high confidence as a result of executing multiple perception-action cycles. The recognition accuracy of the proposed perception model has been validated based on simulation input data using both Support Vector Machine (SVM) and GP based multi-class classifiers. Results obtained during this investigation demonstrate that by using a GP-based classifier, it is possible to obtain true positive classification rates of up to 80\%. Experimental validation of the above semi-autonomous object sorting system shows that the proposed GP based interactive sorting approach outperforms random sorting by up to 30\% when applied to scenes comprising configurations of household objects. Finally, a fully autonomous visual architecture is presented that has been developed to accommodate manipulation skills for an autonomous system to interact with the scene by object manipulation. This proposed visual architecture is mainly made of two stages: 1) A perception stage, that is a modified version of the aforementioned visual interaction model, 2) An interaction stage, that performs a set of ad-hoc actions relying on the information received from the perception stage. More specifically, the interaction stage simply reasons over the information (class label and associated probabilistic confidence score) received from perception stage to choose one of the following two actions: 1) An object class has been identified with high confidence, so remove from the scene and place it in the designated basket/bin for that particular class. 2) An object class has been identified with less probabilistic confidence, since from observation and inspired from the human behaviour of inspecting doubtful objects, an action is chosen to further investigate that object in order to confirm the object’s identity by capturing more images from different views in isolation. The perception stage then processes these views, hence multiple perception-action/interaction cycles take place. From an application perspective, the task of autonomous category based objects sorting is performed and the experimental design for the task is described in detail