Handling Missing Annotations for Semantic Segmentation with Deep ConvNets

International audienceAnnotation of medical images for semantic segmentation is a very time consuming and difficult task. Moreover, clinical experts often focus on specific anatomical structures and thus, produce partially annotated images. In this paper, we introduce SMILE, a new deep con-volutional neural network which addresses the issue of learning with incomplete ground truth. SMILE aims to identify ambiguous labels in order to ignore them during training, and don't propagate incorrect or noisy information. A second contribution is SMILEr which uses SMILE as initialization for automatically relabeling missing annotations, using a curriculum strategy. Experiments on 3 organ classes (liver, stomach, pancreas) show the relevance of the proposed approach for semantic seg-mentation: with 70% of missing annotations, SMILEr performs similarly as a baseline trained with complete ground truth annotations

Automatic Recognition of Colon and Esophagogastric Cancer with Machine Learning and Hyperspectral Imaging

There are approximately 1.8 million diagnoses of colorectal cancer, 1 million diagnoses of stomach cancer, and 0.6 million diagnoses of esophageal cancer each year globally. An automatic computer-assisted diagnostic (CAD) tool to rapidly detect colorectal and esophagogastric cancer tissue in optical images would be hugely valuable to a surgeon during an intervention. Based on a colon dataset with 12 patients and an esophagogastric dataset of 10 patients, several state-of-the-art machine learning methods have been trained to detect cancer tissue using hyperspectral imaging (HSI), including Support Vector Machines (SVM) with radial basis function kernels, Multi-Layer Perceptrons (MLP) and 3D Convolutional Neural Networks (3DCNN). A leave-one-patient-out cross-validation (LOPOCV) with and without combining these sets was performed. The ROC-AUC score of the 3DCNN was slightly higher than the MLP and SVM with a difference of 0.04 AUC. The best performance was achieved with the 3DCNN for colon cancer and esophagogastric cancer detection with a high ROC-AUC of 0.93. The 3DCNN also achieved the best DICE scores of 0.49 and 0.41 on the colon and esophagogastric datasets, respectively. These scores were significantly improved using a patient-specific decision threshold to 0.58 and 0.51, respectively. This indicates that, in practical use, an HSI-based CAD system using an interactive decision threshold is likely to be valuable. Experiments were also performed to measure the benefits of combining the colorectal and esophagogastric datasets (22 patients), and this yielded significantly better results with the MLP and SVM models

Prediction of In Vivo Laser-Induced Thermal Damage with Hyperspectral Imaging Using Deep Learning.

Thermal ablation is an acceptable alternative treatment for primary liver cancer, of which laser ablation (LA) is one of the least invasive approaches, especially for tumors in high-risk locations. Precise control of the LA effect is required to safely destroy the tumor. Although temperature imaging techniques provide an indirect measurement of the thermal damage, a degree of uncertainty remains about the treatment effect. Optical techniques are currently emerging as tools to directly assess tissue thermal damage. Among them, hyperspectral imaging (HSI) has shown promising results in image-guided surgery and in the thermal ablation field. The highly informative data provided by HSI, associated with deep learning, enable the implementation of non-invasive prediction models to be used intraoperatively. Here we show a novel paradigm "peak temperature prediction model" (PTPM), convolutional neural network (CNN)-based, trained with HSI and infrared imaging to predict LA-induced damage in the liver. The PTPM demonstrated an optimal agreement with tissue damage classification providing a consistent threshold (50.6 ± 1.5 °C) for the damage margins with high accuracy (~0.90). The high correlation with the histology score (r = 0.9085) and the comparison with the measured peak temperature confirmed that PTPM preserves temperature information accordingly with the histopathological assessment

Shape parametrization of bio-mechanical finite element models based on medical images

The main objective of this study is to combine the statistical shape analysis with a morphing procedure in order to generate shape-parametric finite element models of tissues and organs and to explore the reliability and the limitations of this approach when applied to databases of real medical images. As classical statistical shape models are not always adapted to the morphing procedure, a new registration method was developed in order to maximize the morphing efficiency. The method was compared to the traditional iterative thin plate spline (iTPS). Two data sets of 33 proximal femora shapes and 385 liver shapes were used for the comparison. The principal component analysis was used to get the principal morphing modes. In terms of anatomical shape reconstruction (evaluated through the criteria of generalization, compactness and specificity), our approach compared fairly well to the iTPS method, while performing remarkably better in terms of mesh quality, since it was less prone to generate invalid meshes in the interior. This was particularly true in the liver case. Such methodology offers a potential application for the generation of automated finite element (FE) models from medical images. Parametrized anatomical models can also be used to assess the influence of inter-patient variability on the biomechanical response of the tissues. Indeed, thanks to the shape parametrization the user would easily have access to a valid FE model for any shape belonging to the parameters subspace

A model order reduction approach to create patient-specific mechanical models of human liver in computational medicine applications.

Background and objective: This paper focuses on computer simulation aspects of Digital Twin models in the medical framework. In particular, it addresses the need of fast and accurate simulators for the mechanical response at tissue and organ scale and the capability of integrating patient-specific anatomy from medical images to pinpoint the individual variations from standard anatomical models. Methods: We propose an automated procedure to create mechanical models of the human liver with patient-specific geometry and real time capabilities. The method hinges on the use of Statistical Shape Analysis to extract the relevant anatomical features from a database of medical images and Model Order Reduction to compute an explicit parametric solution for the mechanical response as a function of such features. The Sparse Subspace Learning, coupled with a Finite Element solver, was chosen to create low-rank solutions using a non-intrusive sparse sampling of the feature space. Results: In the application presented in the paper, the statistical shape model was trained on a database of 385 three dimensional liver shapes, extracted from medical images, in order to create a parametrized representation of the liver anatomy. This parametrization and an additional parameter describing the breathing motion in linear elasticity were then used as input in the reduced order model. Results show a consistent agreement with the high fidelity Finite Element models built from liver images that were excluded from the training dataset. However, we evidence in the discussion the difficulty of having compact shape parametrizations arising from the extreme variability of the shapes found in the dataset and we propose potential strategies to tackle this issue. Conclusions: A method to represent patient-specific real-time liver deformations during breathing is proposed in linear elasticity. Since the proposed method does not require any adaptation to the direct Finite Element solver used in the training phase, the procedure can be easily extended to more complex non-linear constitutive behaviors - such as hyperelasticity - and more general load cases. Therefore it can be integrated with little intrusiveness to generic simulation software including more sophisticated and realistic models

TRUSTED: The Paired 3D Transabdominal Ultrasound and CT Human Data for Kidney Segmentation and Registration Research

Inter-modal image registration (IMIR) and image segmentation with abdominal Ultrasound (US) data has many important clinical applications, including image-guided surgery, automatic organ measurement and robotic navigation. However, research is severely limited by the lack of public datasets. We propose TRUSTED (the Tridimensional Renal Ultra Sound TomodEnsitometrie Dataset), comprising paired transabdominal 3DUS and CT kidney images from 48 human patients (96 kidneys), including segmentation, and anatomical landmark annotations by two experienced radiographers. Inter-rater segmentation agreement was over 94 (Dice score), and gold-standard segmentations were generated using the STAPLE algorithm. Seven anatomical landmarks were annotated, important for IMIR systems development and evaluation. To validate the dataset's utility, 5 competitive Deep Learning models for automatic kidney segmentation were benchmarked, yielding average DICE scores from 83.2% to 89.1% for CT, and 61.9% to 79.4% for US images. Three IMIR methods were benchmarked, and Coherent Point Drift performed best with an average Target Registration Error of 4.53mm. The TRUSTED dataset may be used freely researchers to develop and validate new segmentation and IMIR methods.Comment: Alexandre Hostettler, and Toby Collins share last authorshi

Partial Hepatic Vein Occlusion and Venous Congestion in Liver Exploration Using a Hyperspectral Camera: A Proposal for Monitoring Intraoperative Liver Perfusion.

INTRODUCTION The changes occurring in the liver in cases of outflow deprivation have rarely been investigated, and no measurements of this phenomenon are available. This investigation explored outflow occlusion in a pig model using a hyperspectral camera. METHODS Six pigs were enrolled. The right hepatic vein was clamped for 30 min. The oxygen saturation (StO2%), deoxygenated hemoglobin level (de-Hb), near-infrared perfusion (NIR), and total hemoglobin index (THI) were investigated at different time points in four perfused lobes using a hyperspectral camera measuring light absorbance between 500 nm and 995 nm. Differences among lobes at different time points were estimated by mixed-effect linear regression. RESULTS StO2% decreased over time in the right lateral lobe (RLL, totally occluded) when compared to the left lateral (LLL, outflow preserved) and the right medial (RML, partially occluded) lobes (p < 0.05). De-Hb significantly increased after clamping in RLL when compared to RML and LLL (p < 0.05). RML was further analyzed considering the right portion (totally occluded) and the left portion of the lobe (with an autonomous draining vein). StO2% decreased and de-Hb increased more smoothly when compared to the totally occluded RLL (p < 0.05). CONCLUSIONS The variations of StO2% and deoxy-Hb could be considered good markers of venous liver congestion

Hyperspectral Imaging in Major Hepatectomies: Preliminary Results from the Ex-Machyna Trial.

Ischemia-reperfusion injury during major hepatic resections is associated with high rates of post-operative complications and liver failure. Real-time intra-operative detection of liver dysfunction could provide great insight into clinical outcomes. In the present study, we demonstrate the intra-operative application of a novel optical technology, hyperspectral imaging (HSI), to predict short-term post-operative outcomes after major hepatectomy. We considered fifteen consecutive patients undergoing major hepatic resection for malignant liver lesions from January 2020 to June 2021. HSI measures included tissue water index (TWI), organ hemoglobin index (OHI), tissue oxygenation (StO2%), and near infrared (NIR). Pre-operative, intra-operative, and post-operative serum and clinical outcomes were collected. NIR values were higher in unhealthy liver tissue (p = 0.003). StO2% negatively correlated with post-operative serum ALT values (r = -0.602), while ΔStO2% positively correlated with ALP (r = 0.594). TWI significantly correlated with post-operative reintervention and OHI with post-operative sepsis and liver failure. In conclusion, the HSI imaging system is accurate and precise in translating from pre-clinical to human studies in this first clinical trial. HSI indices are related to serum and outcome metrics. Further experimental and clinical studies are necessary to determine clinical value of this technology

The Liver Tumor Segmentation Benchmark (LiTS)

In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018. The image dataset is diverse and contains primary and secondary tumors with varied sizes and appearances with various lesion-to-background levels (hyper-/hypo-dense), created in collaboration with seven hospitals and research institutions. Seventy-five submitted liver and liver tumor segmentation algorithms were trained on a set of 131 computed tomography (CT) volumes and were tested on 70 unseen test images acquired from different patients. We found that not a single algorithm performed best for both liver and liver tumors in the three events. The best liver segmentation algorithm achieved a Dice score of 0.963, whereas, for tumor segmentation, the best algorithms achieved Dices scores of 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018). Retrospectively, we performed additional analysis on liver tumor detection and revealed that not all top-performing segmentation algorithms worked well for tumor detection. The best liver tumor detection method achieved a lesion-wise recall of 0.458 (ISBI 2017), 0.515 (MICCAI 2017), and 0.554 (MICCAI 2018), indicating the need for further research. LiTS remains an active benchmark and resource for research, e.g., contributing the liver-related segmentation tasks in http://medicaldecathlon.com/. In addition, both data and online evaluation are accessible via https://competitions.codalab.org/competitions/17094