Détection automatique de déviations chirurgicales et identification de comportements chirurgicaux par modélisation et analyse des processus chirurgicaux

L'auteur n'a pas fourni de résumé en anglaisLes événements indésirables (EIs) sont devenus une vraie préoccupation du monde médical, leur réduction étant recherchée pour assurer la meilleure sécurité possible pour les patients. Les EIs sont, selon la HAS, ‘‘ des situations qui s'écartent de procédures ou de résultats escomptés dans une situation habituelle et qui sont ou qui seraient potentiellement sources de dommages’’. Alors que les EIs postopératoires sont étudiés depuis de nombreuses années, ceux ayant lieu au cours des opérations ne le sont que depuis récemment, comme le montre la récente classification des EIs intraopératoires par Kaafarani et al. publié en 2014. Cependant, la classification d'EIs intraopératoires n'est que la première étape pour comprendre les comportements chirurgicaux qui les entraînent.Dans cette thèse, nous présenterons des méthodes pour détecter l'apparition de déviations dues à l'apparition d'EIs intraopératoires et pour identifier des comportements chirurgicaux à partir de modèle de processus chirurgicaux.Ce travail a nécessité de concevoir et développer une modélisation formelle de la rectopexie et des événements indésirables qui sont associés à cette procédure chirurgicale grâceà la mise en place d'ontologies. Cette modélisation formelle nous a permis de bien appréhender le principe de cette opération et de fournir un vocabulaire permettant une annotation détaillé de vidéos endoscopiques de rectopexies, afin de créer des modèles de processus chirurgicaux en jeu.Grâce à l'annotation des vidéos chirurgicales basée sur cette modélisation, nous avons développé une une méthode de détection automatique des déviations dues à l'apparition d'événements indésirables Cette méthode est basée sur un alignement temporel non-linéaire multi-dimensionnel, que nous avons développé, suivi d'un modèle semi-Markovien caché que nous avons entraîné pour déterminer s'il existe des déviations par rapport à une chirurgie de référence et si celles-ci sont dues à des événements indésirables.Cette détection de déviations dues aux événements indésirables est la première étape afin de comprendre les raisons de leurs apparitions. Nous émettons l'hypothèse que leurs apparitions peuvent être expliquées par une succession d’activités, c'est-à-dire un pattern. Pour répondre à cette hypothèse, nous avons mis en place une méthode de découverte de patterns permettant d'identifier les comportements chirurgicaux spécifiques à différents critères. Cette identification de comportements chirurgicaux est réalisée par une classification ascendante hiérarchique avec la mise en place d'une nouvelle métrique basée sur les patterns partagés entre les chirurgies. Afin de valider notre méthode, nous l'avons comparé à deux études mettant en évidence des différences de comportements chirurgicaux, comme par exemple entre différents sites chirurgicaux ou entre deux types de procédure de la même opération. Une fois la méthode validée, nous avons utilisé notre méthode afin de montrer s'il existait des comportements chirurgicaux spécifiques à des données préopératoires et à l'apparition d'événements indésirables.Pour finir, nous revenons sur les contributions les plus importantes de ces travaux à travers une discussion générale et nous proposons différentes pistes pour améliorer nos résultat

Détection automatique de déviations chirurgicales et identification de comportements chirurgicaux par modélisation et analyse des processus chirurgicaux.

Adverse events are an important concern for medical domain, their reduction is searched to allow the best safety for patients. The adverse events are, according to the HAS (Haute Autorité de santé), ‘‘situations which divert from procedures or from expected results in a usual situation and which are or which would be potentially sources of damage’’. Even though postoperative adverse events have been studied for many years, the ones which occur during the operation are recently studied, for example the first classification of intraoperative adverse events is the classification of Kaafarani et al. published in 2014. Nevertheless, the classification of intraoperative adverse events is only the first step to understand the surgical behaviors to their sources.In this thesis, we will present methods to detect the apparition of deviations due to intraoperative adverse events and to identify surgical behaviors thanks to surgical process model.To allow the development of these methods, the first step was to model the rectopexy and the adverse events related to this surgery thanks to the creation of ontologies. This work has enabled us to understand the principle of this operation and to create a vocabulary. This vocabulary was used to annotate laparoscopic videos of rectopexies, in order to create surgical process models Thanks to the surgical video annotation based on this modelisation of the rectopexy, we have developed a method to automatically detect deviations due to adverse events. This method is based on a multidimensional non-linear temporal scaling, a homemade alignment of sequences, follows by a hidden semi-Markovian model. This Markovian model was trained to detect deviations from a standard surgical process, a reference, and to determine if these deviations are due to adverse events.This deviation detection is the first step in order to understand the reason of their apparitions. We hypothesize that their apparitions could be explained by an activities succession, i.e. a pattern. To verify this hypothesis, we develop a pattern discovery method to allow the identification of specific surgical behaviors. This identification of surgical behaviors was done by a hierarchical clustering thanks to a new metric based on shared pattern between surgeries. To validate our method, we make a comparison with two state of the art article highlighting surgical behaviors, for example, surgical behaviors specific to surgical site or to type of procedures. Once our method has been validated, we have used it to identify surgical behavior specific to preoperative data and to adverse events apparitions.Finally, we come back to the most important contributions of this work through a general discussion and we propose perspectives to improve our results.Les événements indésirables (EIs) sont devenus une vraie préoccupation du monde médical, leur réduction étant recherchée pour assurer la meilleure sécurité possible pour les patients. Les événements indésirables sont, selon la HAS, ‘‘des situations qui s'écartent de procédures ou de résultats escomptés dans une situation habituelle et qui sont ou qui seraient potentiellement sources de dommages’’. Alors que les événements indésirables postopératoires sont étudiés depuis de nombreuses années, ceux ayant lieu au cours des opérations ne le sont que depuis récemment, comme le montre la récente classification des événements indésirables intraopératoires par Kaafarani et al. publiée en 2014. Cependant, la classification d'événements indésirables intraopératoires n'est que la première étape pour comprendre les comportements chirurgicaux qui les entraînent.Dans cette thèse, nous présenterons des méthodes pour détecter l'apparition de déviations dues à l'apparition d'événements indésirables intraopératoires et pour identifier des comportements chirurgicaux à partir de modèle de processus chirurgicaux.Ce travail a nécessité de concevoir et développer une modélisation formelle de la rectopexie et des événements indésirables qui sont associés à cette procédure chirurgicale grâce à la mise en place d'ontologies. Cette modélisation formelle nous a permis de bien appréhender le principe de cette opération et de fournir un vocabulaire permettant une annotation détaillée de vidéos endoscopiques de rectopexies, afin de créer des modèles de processus chirurgicaux.Grâce à l'annotation des vidéos chirurgicales basée sur cette modélisation, nous avons développé une méthode de détection automatique des déviations dues à l'apparition d'événements indésirables. Cette méthode est basée sur un alignement temporel non linéaire multidimensionnel, que nous avons développé, suivi d'un modèle semi-Markovien caché que nous avons entraîné pour déterminer s'il existe des déviations par rapport à une chirurgie de référence et si celles-ci sont dues à des événements indésirables.Cette détection de déviations dues aux événements indésirables est la première étape afin de comprendre les raisons de leurs apparitions. Nous émettons l'hypothèse que leurs apparitions peuvent être expliquées par une succession d’activités, c'est-à-dire un pattern. Pour répondre à cette hypothèse, nous avons mis en place une méthode de découverte de patterns permettant d'identifier les comportements chirurgicaux spécifiques à différents critères. Cette identification de comportements chirurgicaux est réalisée par une classification ascendante hiérarchique avec la mise en place d'une nouvelle métrique basée sur les patterns partagés entre les chirurgies. Afin de valider notre méthode, nous l'avons comparé à deux études mettant en évidence des différences de comportements chirurgicaux, par exemple entre différents sites chirurgicaux ou entre deux types de procédures de la même opération. Une fois la méthode validée, nous avons utilisé notre méthode afin de montrer s'il existait des comportements chirurgicaux spécifiques à des données préopératoires et à l'apparition d'événements indésirables.Pour finir, nous revenons sur les contributions les plus importantes de ces travaux à travers une discussion générale et nous proposons différentes pistes pour améliorer nos résultats

Review of automated performance metrics to assess surgical technical skills in robot-assisted laparoscopy

International audienceIntroduction Robot-assisted laparoscopy is a safe surgical approach with several studies suggesting correlations between complication rates and the surgeon's technical skills. Surgical skills are usually assessed by questionnaires completed by an expert observer. With the advent of surgical robots, automated surgical performance metrics (APMs)-objective measures related to instrument movements-can be computed. The aim of this systematic review was thus to assess APMs use in robot-assisted laparoscopic procedures. The primary outcome was the assessment of surgical skills by APMs and the secondary outcomes were the association between APM and surgeon parameters and the prediction of clinical outcomes. Methods A systematic review following the PRISMA guidelines was conducted. PubMed and Scopus electronic databases were screened with the query "robot-assisted surgery OR robotic surgery AND performance metrics" between January 2010 and January 2021. The quality of the studies was assessed by the medical education research study quality instrument. The study settings, metrics, and applications were analysed. Results The initial search yielded 341 citations of which 16 studies were finally included. The study settings were either simulated virtual reality (VR) (4 studies) or real clinical environment (12 studies). Data to compute APMs were kinematics (motion tracking), and system and specific events data (actions from the robot console). APMs were used to differentiate expertise levels, and thus validate VR modules, predict outcomes, and integrate datasets for automatic recognition models. APMs were correlated with clinical outcomes for some studies. Conclusions APMs constitute an objective approach for assessing technical skills. Evidence of associations between APMs and clinical outcomes remain to be confirmed by further studies, particularly, for non-urological procedures. Concurrent validation is also required

Automatic data-driven real-time segmentation and recognition of surgical workflow

International audiencePurpose With the intention of extending the perception and action of surgical staff inside the operating room, the medical community has expressed a growing interest towards context-aware systems. Requiring an accurate identification of the surgical workflow, such systems make use of data from a diverse set of available sensors. In this paper, we propose a fully data-driven and real-time method for segmentation and recognition of surgical phases using a combination of video data and instrument usage signals, exploiting no prior knowledge. We also introduce new validation metrics for assessment of workflow detection.Methods The segmentation and recognition are based on a four-stage process. Firstly, during the learning time, a Surgical Process Model is automatically constructed from data annotations to guide the following process. Secondly, data samples are described using a combination of low-level visual cues and instrument information. Then, in the third stage, these descriptions are employed to train a set of AdaBoost classifiers capable of distinguishing one surgical phase from others. Finally, AdaBoost responses are used as input to a Hidden semi-Markov Model in order to obtain a final decision.Results On the MICCAI EndoVis challenge laparoscopic dataset we achieved a precision and a recall of 91 % in classification of 7 phases.Conclusion Compared to the analysis based on one data type only, a combination of visual features and instrument signals allows better segmentation, reduction of the detection delay and discovery of the correct phase order

sEMG-based Motion Recognition for Robotic Surgery Training - A Preliminary Study

International audienceRobotic surgery represents a major breakthrough in the evolution of medical technology. Accordingly, efficient skill training and assessment methods should be developed to meet the surgeon’s need of acquiring such robotic skills over a relatively short learning curve in a safe manner. Different from conventional training and assessment methods, we aim to explore the surface electromyography (sEMG) signal during the training process in order to obtain semantic and interpretable information to help the trainee better understand and improve his/her training performance. As a preliminary study, motion primitive recognition based on sEMG signal is studied in this work. Using machine learning (ML) technique, it is shown that the sEMG-based motion recognition method is feasible and promising for hand motions along 3 Cartesian axes in the virtual reality (VR) environment of a commercial robotic surgery training platform, which will hence serve as the basis for new robotic surgical skill assessment criterion and training guidance based on muscle activity information.Considering certain motion patterns were less accurately recognized than others, more data collection and deep learning-based analysis will be carried out to further improve the recognition accuracy in future researc

Stand-up straight!: human pose estimation to evaluate postural skills during orthopedic surgery simulations

International audiencePurpose Surgery simulators can be used to learn technical and non-technical skills and, to analyse posture. Ergonomic skill can be automatically detected with a Human Pose Estimation algorithm to help improve the surgeon's work quality. The objective of this study was to analyse the postural behaviour of surgeons and identify expertise-dependent movements. Our hypothesis was that hesitation and the occurrence of surgical instruments interfering with movement (defined as interfering movements) decrease with expertise. Material and methods Sixty surgeons with three expertise levels (novice, intermediate, and expert) were recruited. During a training session using an arthroscopic simulator, each participant's movements were video-recorded with an RGB camera. A modified OpenPose algorithm was used to detect the surgeon's joints. The detection frequency of each joint in a specific area was visualized with a heatmap-like approach and used to calculate a mobility score. Results This analysis allowed quantifying surgical movements. Overall, the mean mobility score was 0.823, 0.816, and 0.820 for novice, intermediate and expert surgeons, respectively. The mobility score alone was not enough to identify postural behaviour differences. A visual analysis of each participants' movements highlighted expertise-dependent interfering movements. Conclusion Video-recording and analysis of surgeon's movements are a non-invasive approach to obtain quantitative and qualitative ergonomic information in order to provide feedback during training. Our findings suggest that the interfering movements do not decrease with expertise but differ in function of the surgeon's level

Gaze behavior is related to objective technical skills assessment during virtual reality simulator-based surgical training: a proof of concept

International audiencePurposeSimulation-based training allows surgical skills to be learned safely. Most virtual reality-based surgical simulators address technical skills without considering non-technical skills, such as gaze use. In this study, we investigated surgeons' visual behavior during virtual reality-based surgical training where visual guidance is provided. Our hypothesis was that the gaze distribution in the environment is correlated with the simulator's technical skills assessment.MethodsWe recorded 25 surgical training sessions on an arthroscopic simulator. Trainees were equipped with a head-mounted eye-tracking device. A U-net was trained on two sessions to segment three simulator-specific areas of interest (AoI) and the background, to quantify gaze distribution. We tested whether the percentage of gazes in those areas was correlated with the simulator's scores.ResultsThe neural network was able to segment all AoI with a mean Intersection over Union superior to 94% for each area. The gaze percentage in the AoI differed among trainees. Despite several sources of data loss, we found significant correlations between gaze position and the simulator scores. For instance, trainees obtained better procedural scores when their gaze focused on the virtual assistance (Spearman correlation test, N = 7, r = 0.800, p = 0.031).ConclusionOur findings suggest that visual behavior should be quantified for assessing surgical expertise in simulation-based training environments, especially when visual guidance is provided. Ultimately visual behavior could be used to quantitatively assess surgeons' learning curve and expertise while training on VR simulators, in a way that complements existing metrics

Offline identification of surgical deviations in laparoscopic rectopexy

International audienceObjective: According to a meta-analysis of 7 studies, the median number of patients with at least one adverse event during the surgery is 14.4%, and a third of those adverse events were preventable. The occurrence of adverse events forces surgeons to implement corrective strategies and, thus, deviate from the standard surgical process. Therefore, it is clear that the automatic identification of adverse events is a major challenge for patient safety. In this paper, we have proposed a method enabling us to identify such deviations. We have focused on identifying surgeons’ deviations from standard surgical processes due to surgical events rather than anatomic specificities. This is particularly challenging, given the high variability in typical surgical procedure workflows.Methods: We have introduced a new approach designed to automatically detect and distinguish surgical process deviations based on multi-dimensional non-linear temporal scaling with a hidden semi-Markov model using manual annotation of surgical processes. The approach was then evaluated using cross-validation.Results: The best results have over 90% accuracy. Recall and precision for event deviations, i.e. related to adverse events, are respectively below 80% and 40%. To understand these results, we have provided a detailed analysis of the incorrectly-detected observations.Conclusion: Multi-dimensional non-linear temporal scaling with a hidden semi-Markov model provides promising results for detecting deviations. Our error analysis of the incorrectly-detected observations offers different leads in order to further improve our method.Significance: Our method demonstrated the feasibility of automatically detecting surgical deviations that could be implemented for both skill analysis and developing situation awareness-based computer-assisted surgical system

Distinguishing surgical behavior by sequential pattern discovery

International audienceObjective: Each surgical procedure is unique due to patient's and also surgeon's particularities. In this study, we propose a new approach to distinguish surgical behaviors between surgical sites, levels of expertise and individual surgeons thanks to a pattern discovery method. Methods: The developed approach aims to distinguish surgical behaviors based on shared longest frequent sequential patterns between surgical process models. To allow clustering, we propose a new metric called SLFSP. The approach is validated by comparison with a clustering method using Dynamic Time Warping as a metric to characterize the similarity between surgical process models. Results: Our method outperformed the existing approach. It was able to make a perfect distinction between surgical sites (accuracy of 100%). We reached an accuracy superior to 90% and 85% for distinguishing levels of expertise and individual surgeons. Conclusion: Clustering based on shared longest frequent sequential patterns outperformed the previous study based on time analysis. Significance: The proposed method shows the feasibility of comparing surgical process models, not only by their duration but also by their structure of activities. Furthermore, patterns may show risky behaviors, which could be an interesting information for surgical training to prevent adverse events

Virtual reality simulation training improve diagnostic knee arthroscopy and meniscectomy skills: a prospective transfer validity study

Abstract Purpose Limited data exist on the actual transfer of skills learned using a virtual reality (VR) simulator for arthroscopy training because studies mainly focused on VR performance improvement and not on transfer to real word (transfer validity). The purpose of this single‐blinded, controlled trial was to objectively investigate transfer validity in the context of initial knee arthroscopy training. Methods For this study, 36 junior resident orthopaedic surgeons (postgraduate year one and year two) without prior experience in arthroscopic surgery were enrolled to receive standard knee arthroscopy surgery training (NON‐VR group) or standard training plus training on a hybrid virtual reality knee arthroscopy simulator (1 h/month) (VR group). At inclusion, all participants completed a questionnaire on their current arthroscopic technical skills. After 6 months of training, both groups performed three exercises that were evaluated independently by two blinded trainers: i) arthroscopic partial meniscectomy on a bench‐top knee simulator; ii) supervised diagnostic knee arthroscopy on a cadaveric knee; and iii) supervised knee partial meniscectomy on a cadaveric knee. Training level was determined with the Arthroscopic Surgical Skill Evaluation Tool (ASSET) score. Results Overall, performance (ASSET scores) was better in the VR group than NON‐VR group (difference in the global scores: p < 0.001, in bench‐top meniscectomy scores: p = 0.03, in diagnostic knee arthroscopy on a cadaveric knee scores: p = 0.04, and in partial meniscectomy on a cadaveric knee scores: p = 0.02). Subgroup analysis by postgraduate year showed that the year‐one NON‐VR subgroup performed worse than the other subgroups, regardless of the exercise. Conclusion This study showed the transferability of the technical skills acquired by novice residents on a hybrid virtual reality simulator to the bench‐top and cadaveric models. Surgical skill acquired with a VR arthroscopy surgical simulator might safely improve arthroscopy competences in the operating room, also helping to standardise resident training and follow their progress. Level of evidence