Artificial intelligence and automation in endoscopy and surgery

Modern endoscopy relies on digital technology, from high-resolution imaging sensors and displays to electronics connecting configurable illumination and actuation systems for robotic articulation. In addition to enabling more effective diagnostic and therapeutic interventions, the digitization of the procedural toolset enables video data capture of the internal human anatomy at unprecedented levels. Interventional video data encapsulate functional and structural information about a patient’s anatomy as well as events, activity and action logs about the surgical process. This detailed but difficult-to-interpret record from endoscopic procedures can be linked to preoperative and postoperative records or patient imaging information. Rapid advances in artificial intelligence, especially in supervised deep learning, can utilize data from endoscopic procedures to develop systems for assisting procedures leading to computer-assisted interventions that can enable better navigation during procedures, automation of image interpretation and robotically assisted tool manipulation. In this Perspective, we summarize state-of-the-art artificial intelligence for computer-assisted interventions in gastroenterology and surgery

GastroVision: A Multi-class Endoscopy Image Dataset for Computer Aided Gastrointestinal Disease Detection

Integrating real-time artificial intelligence (AI) systems in clinical practices faces challenges such as scalability and acceptance. These challenges include data availability, biased outcomes, data quality, lack of transparency, and underperformance on unseen datasets from different distributions. The scarcity of large-scale, precisely labeled, and diverse datasets are the major challenge for clinical integration. This scarcity is also due to the legal restrictions and extensive manual efforts required for accurate annotations from clinicians. To address these challenges, we present \textit{GastroVision}, a multi-center open-access gastrointestinal (GI) endoscopy dataset that includes different anatomical landmarks, pathological abnormalities, polyp removal cases and normal findings (a total of 27 classes) from the GI tract. The dataset comprises 8,000 images acquired from B{\ae}rum Hospital in Norway and Karolinska University Hospital in Sweden and was annotated and verified by experienced GI endoscopists. Furthermore, we validate the significance of our dataset with extensive benchmarking based on the popular deep learning based baseline models. We believe our dataset can facilitate the development of AI-based algorithms for GI disease detection and classification. Our dataset is available at \url{https://osf.io/84e7f/}

Redes neurais convolucionais para deteção de landmarks gástricas

Gastric cancer is the fifth most incident cancer in the world and, when diagnosed at an advanced stage, its survival rate is only 5%-25%, providing that it is essential that the cancer is detected at an early stage. However, physicians specialized in this diagnosis have difficulties in detecting early lesions during a diagnostic examination, esophagogastroduodenoscopy (EGD). Early lesions on the walls of the digestive system are imperceptible and confounded with the stomach mucosa, being difficult to detect. On the other hand, physicians run the risk of not covering all areas of the stomach during diagnosis, especially areas that may have lesions. The introduction of artificial intelligence into this diagnostic method may help to detect gastric cancer at an earlier stage. The implementation of a system capable of monitoring all areas of the digestive system during EGD would be a solution to prevent the diagnosis of gastric cancer in advanced states. This work focuses on the study of upper gastrointestinal (GI) landmarks monitoring, which are anatomical areas of the digestive system more conducive to the appearance of lesions and that allow better control of the missed areas during EGD exam. The use of convolutional neural networks (CNNs) in GI landmarks monitoring has been a great target of study by the scientific community, with such networks having a good capacity to extract features that better characterize EGD images. The aim of this work consisted in testing new automatic algorithms, specifically CNN-based systems able to detect upper GI landmarks to avoid the presence of blind spots during EGD to increase the quality of endoscopic exams. In contrast with related works in the literature, in this work we used upper GI landmarks images closer to real-world environments. In particular, images for each anatomical landmark class include both examples affected by pathologies and healthy tissue. We tested some pre-trained architectures as the ResNet-50, DenseNet-121, and VGG-16. For each pre-trained architecture, we tested different learning approaches, including the use of class weights (CW), the use of batch normalization and dropout layers, and the use of data augmentation to train the network. The CW ResNet-50 achieved an accuracy of 71.79% and a Mathews Correlation Coefficient (MCC) of 65.06%. In current state-of-art studies, only supervised learning approaches were used to classify EGD images. On the other hand, in our work, we tested the use of unsupervised learning to increase classification performance. In particular, convolutional autoencoder architectures to extract representative features from unlabeled GI images and concatenated their outputs withs with the CW ResNet-50 architecture. We achieved an accuracy of 72.45% and an MCC of 65.08%.O cancro gástrico é o quinto cancro mais incidente no mundo e quando diagnosticado numa fase avançada a taxa de sobrevivência é de apenas 5%-25%. Assim, é essencial que este cancro seja detetado numa fase precoce. No entanto, os médicos especializados neste diagnóstico nem sempre são capazes de uma boa performance de deteção durante o exame de diagnóstico, a esofagogastroduodenoscopia (EGD). As lesões precoces nas paredes do sistema digestivo são quase impercetíveis e confundíveis com a mucosa do estômago, sendo difíceis de detetar. Por outro lado, os médicos correm o risco de não cobrirem todas as áreas do estômago durante o diagnóstico, podendo estas áreas ter lesões. A introdução da inteligência artificial neste método de diagnóstico poderá ajudar a detetar o cancro gástrico numa fase mais precoce. A implementação de um sistema capaz de fazer a monitorização de todas as áreas do sistema digestivo durante a EGD seria uma solução de forma a prevenir o diagnóstico de cancro gástrico em estados avançados. Este trabalho tem como foco o estudo da monitorização de landmarks gastrointestinais (GI) superiores, que são zonas anatómicas do sistema digestivo mais propícias ao surgimento de lesões e que permitem fazer um melhor controlo das áreas esquecidas durante a EGD. O uso de redes neurais convolucionais (CNNs) na monitorização de landmarks GI tem sido grande alvo de estudo pela comunidade científica, por serem redes com uma boa capacidade de extração features que melhor caraterizam as imagens da EGD. O objetivo deste trabalho consistiu em testar novos algoritmos automáticos baseados em CNNs capazes de detetar landmarks GI superiores para evitar a presença áreas não cobertas durante a EGD, aumentando a qualidade deste exame. Este trabalho difere de outros estudos porque foram usadas classes de landmarks GI superiores mais próximas do ambiente real da EGD. Dentro de cada classe incluímos imagens com patologias e de tecido saudável da respetiva zona anatómica, ao contrário dos demais estudos. Nos estudos apresentados no estado de arte apenas foram consideradas classes de landmarks com tecido saudável em tarefas de deteção de landmarks GI. Testámos algumas arquiteturas pré-treinadas como a ResNet-50, a DenseNet-121 e a VGG-16. Para cada arquitetura pré-treinada, testámos algumas variáveis: o uso de class weights (CW), o uso das camadas batch normalization e dropout, e o uso de data augmentation. A arquitetura CW ResNet-50 atingiu uma accuracy de 71,79% e um coeficiente de correlação de Mathews (MCC) de 65,06%. Nos estudos apresentados no estado de arte, apenas foram estudados sistemas de supervised learning para classificação de imagens EGD enquanto, que no nosso trabalho, foram também testados sistemas de unsupervised learning para aumentar o desempenho da classificação. Em particular, arquiteturas autoencoder convolucionais para extração de features de imagens GI sem labels. Assim, concatenámos os outputs das arquiteturas autoencoder convolucionais com a arquitetura CW ResNet-50 e alcançamos uma accuracy de 72,45% e um MCC de 65,08%.Mestrado em Engenharia Biomédic

Kvasir-Capsule, a video capsule endoscopy dataset

Artificial intelligence (AI) is predicted to have profound effects on the future of video capsule endoscopy (VCE) technology. The potential lies in improving anomaly detection while reducing manual labour. Existing work demonstrates the promising benefits of AI-based computer-assisted diagnosis systems for VCE. They also show great potential for improvements to achieve even better results. Also, medical data is often sparse and unavailable to the research community, and qualified medical personnel rarely have time for the tedious labelling work. We present Kvasir-Capsule, a large VCE dataset collected from examinations at a Norwegian Hospital. Kvasir-Capsule consists of 117 videos which can be used to extract a total of 4,741,504 image frames. We have labelled and medically verified 47,238 frames with a bounding box around findings from 14 different classes. In addition to these labelled images, there are 4,694,266 unlabelled frames included in the dataset. The Kvasir-Capsule dataset can play a valuable role in developing better algorithms in order to reach true potential of VCE technology

Artificial intelligence in biliopancreatic endoscopy: Is there any role?

Artificial intelligence (AI) research in endoscopy is being translated at rapid pace with a number of approved devices now available for use in luminal endoscopy. However, the published literature for AI in biliopancreatic endoscopy is predominantly limited to early pre-clinical studies including applications for diagnostic EUS and patient risk stratification. Potential future use cases are highlighted in this manuscript including optical characterisation of strictures during cholangioscopy, prediction of post-ERCP acute pancreatitis and selective biliary duct cannulation difficulty, automated report generation and novel AI-based quality key performance metrics. To realise the full potential of AI and accelerate innovation, it is crucial that robust inter-disciplinary collaborations are formed between biliopancreatic endoscopists and AI researchers

Safety and efficacy of endoscopic submucosal dissection for rectal neoplasia: a multicenter North American experience.

Background and aims  Rectal lesions traditionally represent the first lesions approached during endoscopic submucosal dissection (ESD) training in the West. We evaluated the safety and efficacy of rectal ESD in North America. Methods  This is a multicenter retrospective analysis of rectal ESD between January 2010 and September 2018 in 15 centers. End points included: rates of en bloc resection, R0 resection, adverse events, comparison of pre- and post-ESD histology, and factors associated with failed resection. Results  In total, 171 patients (median age 63 years; 56 % men) underwent rectal ESD (median size 43 mm). En bloc resection was achieved in 141 cases (82.5 %; 95 %CI 76.8-88.2), including 24 of 27 (88.9 %) with prior failed endoscopic mucosal resection (EMR). R0 resection rate was 74.9 % (95 %CI 68.4-81.4). Post-ESD bleeding and perforation occurred in 4 (2.3 %) and 7 (4.1 %), respectively. Covert submucosal invasive cancer (SMIC) was identified in 8.6 % of post-ESD specimens. There was one case (1/120; 0.8 %) of recurrence at a median follow-up of 31 weeks; IQR: 19-76 weeks). Older age and higher body mass index (BMI) were predictors of failed R0 resection, whereas submucosal fibrosis was associated with a higher likelihood of both failed en bloc and R0 resection. Conclusion  Rectal ESD in North America is safe and is associated with high en bloc and R0 resection rates. The presence of submucosal fibrosis was the main predictor of failed en bloc and R0 resection. ESD can be considered for select rectal lesions, and serves not only to establish a definitive tissue diagnosis but also to provide curative resection for lesions with covert advanced disease

Hybrid Loss with Network Trimming for Disease Recognition in Gastrointestinal Endoscopy

EndoTect Challenge 2020, which aims at the detection of gastrointestinal diseases and abnormalities, consists of three tasks including Detection, Efficient Detection and Segmentation in endoscopic images. Although pathologies belonging to different classes can be manually separated by experienced experts, however, existing classification models struggle to discriminate them due to low inter-class variability. As a result, the models’ convergence deteriorates. To this end, we propose a hybrid loss function to stabilise model training. For the detection and efficient detection tasks, we utilise ResNet-152 and MobileNetV3 architectures, respectively, along with the hybrid loss function. For the segmentation task, Cascade Mask R-CNN is investigated. In this paper, we report the architecture of our detection and segmentation models and the performance of our methods on HyperKvasir and EndoTect test dataset

Efficient video indexing for monitoring disease activity and progression in the upper gastrointestinal tract

Endoscopy is a routine imaging technique used for both diagnosis and minimally invasive surgical treatment. While the endoscopy video contains a wealth of information, tools to capture this information for the purpose of clinical reporting are rather poor. In date, endoscopists do not have any access to tools that enable them to browse the video data in an efficient and user friendly manner. Fast and reliable video retrieval methods could for example, allow them to review data from previous exams and therefore improve their ability to monitor disease progression. Deep learning provides new avenues of compressing and indexing video in an extremely efficient manner. In this study, we propose to use an autoencoder for efficient video compression and fast retrieval of video images. To boost the accuracy of video image retrieval and to address data variability like multi-modality and view-point changes, we propose the integration of a Siamese network. We demonstrate that our approach is competitive in retrieving images from 3 large scale videos of 3 different patients obtained against the query samples of their previous diagnosis. Quantitative validation shows that the combined approach yield an overall improvement of 5% and 8% over classical and variational autoencoders, respectively.Comment: Accepted at IEEE International Symposium on Biomedical Imaging (ISBI), 201