One-shot domain adaptation in video-based assessment of surgical skills

Deep Learning (DL) has achieved automatic and objective assessment of surgical skills. However, DL models are data-hungry and restricted to their training domain. This prevents them from transitioning to new tasks where data is limited. Hence, domain adaptation is crucial to implement DL in real life. Here, we propose a meta-learning model, A-VBANet, that can deliver domain-agnostic surgical skill classification via one-shot learning. We develop the A-VBANet on five laparoscopic and robotic surgical simulators. Additionally, we test it on operating room (OR) videos of laparoscopic cholecystectomy. Our model successfully adapts with accuracies up to 99.5% in one-shot and 99.9% in few-shot settings for simulated tasks and 89.7% for laparoscopic cholecystectomy. For the first time, we provide a domain-agnostic procedure for video-based assessment of surgical skills. A significant implication of this approach is that it allows the use of data from surgical simulators to assess performance in the operating room.Comment: 12 pages (+9 pages of Supplementary Materials), 4 figures (+2 Supplementary Figures), 2 tables (+5 Supplementary Tables

Understanding whole-body inter-personal dynamics between two players using neural Granger causality as the explainable AI (XAI)

Background: Simultaneously focusing on intra- and inter-individual body dynamics and elucidating how these affect each other will help understand human inter-personal coordination behavior. However, this association has not been investigated previously owing to difficulties in analyzing complex causal relations among several body components.To address this issue, this study proposes a new analytical framework that attempts to understand the underlying causal structures behind each joint movement of individual baseball players using neural Granger causality (NGC) as the explainable AI. Methods: In the NGC analysis, causal relationships were defined as the size of the weight parameters of the first layer of a machine-learning model trained to predict the future state of a specific time-series variable. To verify the approach in a practical context, we conducted an experiment with 16 pairs of expert baseball pitchers and batters; input datasets with 27 joint resultant velocity data (joints of 13 pitchers and 14 batters) were generated and used for model training.Results: NGC analysis revealed significant causal relations among intra- and inter-individual body components such as the batter's hands having a causal effect from the pitcher's throwing arm. Remarkably, although the causality from the batter's body to pitcher's body is much lower than the reverse, it is significantly correlated with batter performance outcomes. Conclusions: The above results suggest the effectiveness of NGC analysis for understanding whole-body inter-personal coordination dynamics and that of the AI technique as a new approach for analyzing complex human behavior from a different perspective than conventional techniques.Comment: 35 pages (including 6 supporting information), 9 figures, 1 tabl

Automatic microsurgical skill assessment based on cross-domain transfer learning

The assessment of microsurgical skills for Robot-Assisted Microsurgery (RAMS) still relies primarily on subjective observations and expert opinions. A general and automated evaluation method is desirable. Deep neural networks can be used for skill assessment through raw kinematic data, which has the advantages of being objective and efficient. However, one of the major issues of deep learning for the analysis of surgical skills is that it requires a large database to train the desired model, and the training process can be time-consuming. This letter presents a transfer learning scheme for training a model with limited RAMS datasets for microsurgical skill assessment. An in-house Microsurgical Robot Research Platform Database (MRRPD) is built with data collected from a microsurgical robot research platform (MRRP). It is used to verify the proposed cross-domain transfer learning for RAMS skill level assessment. The model is fine-tuned after training with the data obtained from the MRRP. Moreover, microsurgical tool tracking is developed to provide visual feedback while task-specific metrics and the other general evaluation metrics are provided to the operator as a reference. The method proposed has shown to offer the potential to guide the operator to achieve a higher level of skills for microsurgical operation