Gesture Recognition in Robotic Surgery: a Review

OBJECTIVE: Surgical activity recognition is a fundamental step in computer-assisted interventions. This paper reviews the state-of-the-art in methods for automatic recognition of fine-grained gestures in robotic surgery focusing on recent data-driven approaches and outlines the open questions and future research directions. METHODS: An article search was performed on 5 bibliographic databases with combinations of the following search terms: robotic, robot-assisted, JIGSAWS, surgery, surgical, gesture, fine-grained, surgeme, action, trajectory, segmentation, recognition, parsing. Selected articles were classified based on the level of supervision required for training and divided into different groups representing major frameworks for time series analysis and data modelling. RESULTS: A total of 52 articles were reviewed. The research field is showing rapid expansion, with the majority of articles published in the last 4 years. Deep-learning-based temporal models with discriminative feature extraction and multi-modal data integration have demonstrated promising results on small surgical datasets. Currently, unsupervised methods perform significantly less well than the supervised approaches. CONCLUSION: The development of large and diverse open-source datasets of annotated demonstrations is essential for development and validation of robust solutions for surgical gesture recognition. While new strategies for discriminative feature extraction and knowledge transfer, or unsupervised and semi-supervised approaches, can mitigate the need for data and labels, they have not yet been demonstrated to achieve comparable performance. Important future research directions include detection and forecast of gesture-specific errors and anomalies. SIGNIFICANCE: This paper is a comprehensive and structured analysis of surgical gesture recognition methods aiming to summarize the status of this rapidly evolving field

Contact aware robust semi-autonomous teleoperation of mobile manipulators

In the context of human-robot collaboration, cooperation and teaming, the use of mobile manipulators is widespread on applications involving unpredictable or hazardous environments for humans operators, like space operations, waste management and search and rescue on disaster scenarios. Applications where the manipulator's motion is controlled remotely by specialized operators. Teleoperation of manipulators is not a straightforward task, and in many practical cases represent a common source of failures. Common issues during the remote control of manipulators are: increasing control complexity with respect the mechanical degrees of freedom; inadequate or incomplete feedback to the user (i.e. limited visualization or knowledge of the environment); predefined motion directives may be incompatible with constraints or obstacles imposed by the environment. In the latter case, part of the manipulator may get trapped or blocked by some obstacle in the environment, failure that cannot be easily detected, isolated nor counteracted remotely. While control complexity can be reduced by the introduction of motion directives or by abstraction of the robot motion, the real-time constraint of the teleoperation task requires the transfer of the least possible amount of data over the system's network, thus limiting the number of physical sensors that can be used to model the environment. Therefore, it is of fundamental to define alternative perceptive strategies to accurately characterize different interaction with the environment without relying on specific sensory technologies. In this work, we present a novel approach for safe teleoperation, that takes advantage of model based proprioceptive measurement of the robot dynamics to robustly identify unexpected collisions or contact events with the environment. Each identified collision is translated on-the-fly into a set of local motion constraints, allowing the exploitation of the system redundancies for the computation of intelligent control laws for automatic reaction, without requiring human intervention and minimizing the disturbance of the task execution (or, equivalently, the operator efforts). More precisely, the described system consist in two different building blocks. The first, for detecting unexpected interactions with the environment (perceptive block). The second, for intelligent and autonomous reaction after the stimulus (control block). The perceptive block is responsible of the contact event identification. In short, the approach is based on the claim that a sensorless collision detection method for robot manipulators can be extended to the field of mobile manipulators, by embedding it within a statistical learning framework. The control deals with the intelligent and autonomous reaction after the contact or impact with the environment occurs, and consist on an motion abstraction controller with a prioritized set of constrains, where the highest priority correspond to the robot reconfiguration after a collision is detected; when all related dynamical effects have been compensated, the controller switch again to the basic control mode

Automatic induction of verb classes using clustering

Verb classifications have attracted a great deal of interest in both linguistics and natural language processing (NLP). They have proved useful for important tasks and applications, including e.g. computational lexicography, parsing, word sense disambiguation, semantic role labelling, information extraction, question-answering, and machine translation (Swier and Stevenson, 2004; Dang, 2004; Shi and Mihalcea, 2005; Kipper et al., 2008; Zapirain et al., 2008; Rios et al., 2011). Particularly useful are classes which capture generalizations about a range of linguistic properties (e.g. lexical, (morpho-)syntactic, semantic), such as those proposed by Beth Levin (1993). However, full exploitation of such classes in real-world tasks has been limited because no comprehensive or domain-specific lexical classification is available. This thesis investigates how Levin-style lexical semantic classes could be learned automatically from corpus data. Automatic acquisition is cost-effective when it involves either no or minimal supervision and it can be applied to any domain of interest where adequate corpus data is available. We improve on earlier work on automatic verb clustering. We introduce new features and new clustering methods to improve the accuracy and coverage. We evaluate our methods and features on well-established cross-domain datasets in English, on a specific domain of English (the biomedical) and on another language (French), reporting promising results. Finally, our task-based evaluation demonstrates that the automatically acquired lexical classes enable new approaches to some NLP tasks (e.g. metaphor identification) and help to improve the accuracy of existing ones (e.g. argumentative zoning).This work was supported by a Dorothy Hodgkin PhD Scholarship

EVALITA Evaluation of NLP and Speech Tools for Italian Proceedings of the Final Workshop

Editor of the proceedings of EVALITA 2016

The effectiveness of an upper extremity neuromuscular training program on the shoulder function of military members with a rotator cuff tendinopathy : a pilot randomized controlled trial

INTRODUCTION: La tendinopathie de la coiffe des rotateurs (TCR) entraine au quotidien des douleurs et faiblesses musculaires et une diminution du contrôle moteur à l'épaule. OBJECTIFS: Les objectifs de cette étude étaient i) d'effectuer une revue de littérature pour identifier les méthodes de quantification de la proprioception de l'épaule utilisées en laboratoire et en clinique et d’en présenter les qualités métrologiques, ii) d'évaluer l'efficacité d’un programme d’entrainement neuro-musculaire en comparant son efficacité à réduire la douleur à l’épaule et en améliorer la fonction à celle obtenue par des soins usuels de physiothérapie. MÉTHODES: i) Une revue de 5 bases de données a été conduite d’octobre 2015 à juillet 2016 pour documenter les propriétés métrologiques de protocoles d’évaluation de la proprioception à l'épaule. Les études incluses ont été évaluées à l'aide de l’outil de contrôle QualSyst et de l'échelle COSMIN à 4 points. ii) Trente-trois soldats en service actif au sein des Forces armées canadiennes ont été assignés au hasard à 1) programme standardisé supervisé d’entrainement neuromusculaire et contrôle moteur (Exp) ou à 2) soins usuels de physiothérapie (Ctl). Les variables principales étaient les symptômes, la capacité fonctionnelle et les limitations physiques évalués avec le questionnaire Disabilities of the Arm, Shoulder and Hand (DASH) et la variable secondaire était l'indice Western Ontario Rotator Cuff (WORC). Toutes les variables ont été mesurées au départ (T0) et à 6 (T6) et 12 (T12) semaines après l'intervention. La comparaison des effets des interventions a été évaluée à l'aide d’une analyse per protocole (APP), analyse intention-traitement (AIT) et avec une analyse de variance à mesures répétées à 2 voies. RÉSULTATS: i) Vingt et une études (n = 407 participants, 553 épaules) ont été retenues. Les études analysées confirment d'excellents scores méthodologiques avec l’outil QualSyst (88,1 ± 9,9%) et de bons scores avec le COSMIN pour la fidélité (71,1%) et un score de qualité modérée à faible (50%) pour la validité de critère. Les coefficients de corrélation intraclasse (CCI) pondérés pour la fidélité intraévaluateur étaient les plus élevés pour le sens du positionnement articulaire passif et la kinesthésie soit 0,92 ± 0,07 (n = 214) et 0,92 ± 0,04 (n = 74), respectivement. Le mouvement et l'outil les plus fidèles sont la rotation interne à 90 ° d'abduction (CCI = 0,88 ± 0,01 (n = 53)) et le dynamomètre (CCI = 0,92 ± 0,88 (n = 225)). Aucune étude n’a rapporté d’indices de sensibilité au changement. ii) Aucune interaction significative (p ≥ 0,101) de groupe × temps (p ≥ 0,101) n'a été démontrée. Par contre, nous avons observé un effet de temps significatif (p <0,001) pour le questionnaire DASH et l'indice WORC. CONCLUSION: Ces données préliminaires suggèrent que les deux approches proposées conduisent à des améliorations comparables. L'utilisation d'une intervention de groupe axée sur l'exercice a le potentiel d'être aussi efficace qu'une approche un à un plus exigeante en terme de temps de traitement. Ces résultats permettront de fournir aux cliniciens des lignes directrices pour la mesure de la proprioception à l'épaule et l’utilisation d’une approche novatrice de traitement en groupe pour la TCR. Mots clés : Épaule, tendinopathie, contrôle moteur, proprioception, programme d'exercices, soins en physiothérapieINTRODUCTION: The shoulder is the most mobile joint of the body which means that it heavily relies of an important level of neuromuscular control at all times. A rotator cuff (RC) complex provides stability to the shoulder and often times falls victim to injury, which can produce functional limitations during activities of daily living and work tasks. Individuals affected by an RC tendinopathy often have neuromuscular and proprioceptive deficits. OBJECTIVES: The objectives of this study are to (i) conduct a systematic review to identify methods of quantifying shoulder proprioception in a laboratory and clinical setting and to present the associated psychometric properties. (ii) To evaluate the effectiveness of a novel neuromuscular training program for the upper extremities versus one-on-one physiotherapy care (manual therapy, range of motion exercises, strengthening) for the reduction of shoulder pain and improvement in function with soldiers affected by an RC tendinopathy. METHODS: (i) A review of five databases was conducted from conception to July 2016 to identify studies that reported at least one psychometric property of a shoulder proprioception protocol. The included studies were evaluated using the QualSyst checklist and the 4-point COSMIN scale. (ii) Thirty-three military personnel with the Canadian Armed Forces were randomly assigned to one of the following interventions: 1) Upper Extremity Neuromuscular Training Program; (2) usual physiotherapy care. The main outcomes included symptoms and functional capacity assessed using the Disability of the Arm, Shoulder, and Hand (DASH) questionnaire. A secondary outcome included the Western Ontario Rotator Cuff (WORC) Index. Outcome measures were evaluated at baseline (T0) and 6 (T6) and 12 (T12) weeks post-intervention. The effects of the interventions were evaluated using repeated 2-way variance measures (ANOVAs) for a per-protocol analysis and intention-to-treat. RESULTS: i) Twenty-one studies were included, resulting in 407 participants and 553 evaluated shoulders (n). The weighed intraclass correlation coefficients (ICC) for intra-rater reliability were highest for passive joint position sense and kinesthesia, ICC = 0.92 ± 0.07 (n = 214) and ICC = 0.92 ± 0.04 (n = 74), respectively. The most reliable direction of movement and equipment used were internal rotation at 90° abduction, ICC = 0.88 ± 0.01 (n = 53), and the dynamometer, ICC = 0.92 ± 0.88 (N = 225). ii) No significant group (p ≥ 0.1) or group × time interactions (p ≥ 0.1) were found; though a statistically significant time effect (p < 0.001) was established for the DASH questionnaire and WORC Index. Our preliminary data suggests a marginally better improvement with the control group with all outcomes over 12 weeks. CONCLUSION: The evaluation of shoulder proprioception is most reliable when using a passive protocol with an isokinetic dynamometer for internal rotation at 90° shoulder abduction. The preliminary results of our pilot RCT suggest that both groups statistically improved with a time effect, but that the usual care group further demonstrated clinically significant gains. The results of this study will provide clinicians with potential guidelines for measuring shoulder proprioception in a clinical setting, as well as an innovative approach to group therapy that is potentially less costly and equally as effective as conventional one-on-one physiotherapy. Key words (4-6) : Shoulder, tendinopathy, motor control, proprioception, exercise program, physiotherapy car

Designing Statistical Language Learners: Experiments on Noun Compounds

The goal of this thesis is to advance the exploration of the statistical language learning design space. In pursuit of that goal, the thesis makes two main theoretical contributions: (i) it identifies a new class of designs by specifying an architecture for natural language analysis in which probabilities are given to semantic forms rather than to more superficial linguistic elements; and (ii) it explores the development of a mathematical theory to predict the expected accuracy of statistical language learning systems in terms of the volume of data used to train them. The theoretical work is illustrated by applying statistical language learning designs to the analysis of noun compounds. Both syntactic and semantic analysis of noun compounds are attempted using the proposed architecture. Empirical comparisons demonstrate that the proposed syntactic model is significantly better than those previously suggested, approaching the performance of human judges on the same task, and that the proposed semantic model, the first statistical approach to this problem, exhibits significantly better accuracy than the baseline strategy. These results suggest that the new class of designs identified is a promising one. The experiments also serve to highlight the need for a widely applicable theory of data requirements.Comment: PhD thesis (Macquarie University, Sydney; December 1995), LaTeX source, xii+214 page

Principled Approaches to Automatic Text Summarization

Automatic text summarization is a particularly challenging Natural Language Processing (NLP) task involving natural language understanding, content selection and natural language generation. In this thesis, we concentrate on the content selection aspect, the inherent problem of summarization which is controlled by the notion of information Importance. We present a simple and intuitive formulation of the summarization task as two components: a summary scoring function θ measuring how good a text is as a summary of the given sources, and an optimization technique O extracting a summary with a high score according to θ. This perspective offers interesting insights over previous summarization efforts and allows us to pinpoint promising research directions. In particular, we realize that previous works heavily constrained the summary scoring function in order to solve convenient optimization problems (e.g., Integer Linear Programming). We question this assumption and demonstrate that General Purpose Optimization (GPO) techniques like genetic algorithms are practical. These GPOs do not require mathematical properties from the objective function and, thus, the summary scoring function can be relieved from its previously imposed constraints. Additionally, the summary scoring function can be evaluated on its own based on its ability to correlate with humans. This offers a principled way of examining the inner workings of summarization systems and complements the traditional evaluations of the extracted summaries. In fact, evaluation metrics are also summary scoring functions which should correlate well with humans. Thus, the two main challenges of summarization, the evaluation and the development of summarizers, are unified within the same setup: discovering strong summary scoring functions. Hence, we investigated ways of uncovering such functions. First, we conducted an empirical study of learning the summary scoring function from data. The results show that an unconstrained summary scoring function is better able to correlate with humans. Furthermore, an unconstrained summary scoring function optimized approximately with GPO extracts better summaries than a constrained summary scoring function optimized exactly with, e.g., ILP. Along the way, we proposed techniques to leverage the small and biased human judgment datasets. Additionally, we released a new evaluation metric explicitly trained to maximize its correlation with humans. Second, we developed a theoretical formulation of the notion of Importance. In a framework rooted in information theory, we defined the quantities: Redundancy, Relevance and Informativeness. Importance arises as the notion unifying these concepts. More generally, Importance is the measure that guides which choices to make when information must be discarded. Finally, evaluation remains an open-problem with a massive impact on summarization progress. Thus, we conducted experiments on available human judgment datasets commonly used to compare evaluation metrics. We discovered that these datasets do not cover the high-quality range in which summarization systems and evaluation metrics operate. This motivates efforts to collect human judgments for high-scoring summaries as this would be necessary to settle the debate over which metric to use. This would also be greatly beneficial for improving summarization systems and metrics alike

Seamless Multimodal Biometrics for Continuous Personalised Wellbeing Monitoring

Artificially intelligent perception is increasingly present in the lives of every one of us. Vehicles are no exception, (...) In the near future, pattern recognition will have an even stronger role in vehicles, as self-driving cars will require automated ways to understand what is happening around (and within) them and act accordingly. (...) This doctoral work focused on advancing in-vehicle sensing through the research of novel computer vision and pattern recognition methodologies for both biometrics and wellbeing monitoring. The main focus has been on electrocardiogram (ECG) biometrics, a trait well-known for its potential for seamless driver monitoring. Major efforts were devoted to achieving improved performance in identification and identity verification in off-the-person scenarios, well-known for increased noise and variability. Here, end-to-end deep learning ECG biometric solutions were proposed and important topics were addressed such as cross-database and long-term performance, waveform relevance through explainability, and interlead conversion. Face biometrics, a natural complement to the ECG in seamless unconstrained scenarios, was also studied in this work. The open challenges of masked face recognition and interpretability in biometrics were tackled in an effort to evolve towards algorithms that are more transparent, trustworthy, and robust to significant occlusions. Within the topic of wellbeing monitoring, improved solutions to multimodal emotion recognition in groups of people and activity/violence recognition in in-vehicle scenarios were proposed. At last, we also proposed a novel way to learn template security within end-to-end models, dismissing additional separate encryption processes, and a self-supervised learning approach tailored to sequential data, in order to ensure data security and optimal performance. (...)Comment: Doctoral thesis presented and approved on the 21st of December 2022 to the University of Port

Hydrogeological engineering approaches to investigate and characterize heterogeneous aquifers

This dissertation presents a compilation of five stand-alone manuscripts (Chapters 2 through 5 and Appendix A). Chapters 2 through 5 present hydrogeological analysis approaches, while Appendix A is utilized within the dissertation introduction as an example of a non-physically based modeling approach, albeit demonstrated on a non-hydrogeologically based application. Chapter 2 presents an inverse approach to decompose pumping influences from water-level fluctuations observed at a monitoring location. Chapter 3 presents an inferencing approach to identify effective aquifer properties at the interwell scale that can be applied to highly transient datasets. Chapter 4 introduces the use of a Markov-chain model of spatial correlation to an automated geostatistical inverse framework, demonstrating the approach on a 2-D two-stratigraphic-unit synthetic aquifer. Chapter 5 utilizes the inverse framework introduced in Chapter 4 to develop a stochastic analysis approach to identify the most plausible geostatistical model given the available data. The dissertation introduction reconciles these hydrogeological engineering approaches within the context of the current hydrogeological perspective, discussing where these approaches within the often conflicting goals of providing operational decision support based on modeling and advancing the science of hydrogeology beyond its current limitations