Multimodal image fusion of anatomical structures for diagnosis, therapy planning and assistance

This paper provides an overview of work done in recent years by our research group to fuse multimodal images of the trunk of patients with Adolescent Idiopathic Scoliosis (AIS) treated at Sainte-Justine University Hospital Center (CHU). We first describe our surface acquisition system and introduce a set of clinical measurements (indices) based on the trunk's external shape, to quantify its degree of asymmetry. We then describe our 3D reconstruction system of the spine and rib cage from biplanar radiographs and present our methodology for multimodal fusion of MRI, X-ray and external surface images of the trunk We finally present a physical model of the human trunk including bone and soft tissue for the simulation of the surgical outcome on the external trunk shape in AIS.CIHR / IRS

Joint segmentation and classification of retinal arteries/veins from fundus images

Objective Automatic artery/vein (A/V) segmentation from fundus images is required to track blood vessel changes occurring with many pathologies including retinopathy and cardiovascular pathologies. One of the clinical measures that quantifies vessel changes is the arterio-venous ratio (AVR) which represents the ratio between artery and vein diameters. This measure significantly depends on the accuracy of vessel segmentation and classification into arteries and veins. This paper proposes a fast, novel method for semantic A/V segmentation combining deep learning and graph propagation. Methods A convolutional neural network (CNN) is proposed to jointly segment and classify vessels into arteries and veins. The initial CNN labeling is propagated through a graph representation of the retinal vasculature, whose nodes are defined as the vessel branches and edges are weighted by the cost of linking pairs of branches. To efficiently propagate the labels, the graph is simplified into its minimum spanning tree. Results The method achieves an accuracy of 94.8% for vessels segmentation. The A/V classification achieves a specificity of 92.9% with a sensitivity of 93.7% on the CT-DRIVE database compared to the state-of-the-art-specificity and sensitivity, both of 91.7%. Conclusion The results show that our method outperforms the leading previous works on a public dataset for A/V classification and is by far the fastest. Significance The proposed global AVR calculated on the whole fundus image using our automatic A/V segmentation method can better track vessel changes associated to diabetic retinopathy than the standard local AVR calculated only around the optic disc.Comment: Preprint accepted in Artificial Intelligence in Medicin

Non-invasive quantitative assessment of scoliosis spinal surgery outcome

Improving the appearance of the trunk is an important goal of scoliosis surgical treatment, mainly in patients' eyes. Unfortunately, existing methods for assessing postoperative trunk appearance are rather subjective as they rely on a qualitative evaluation of the trunk shape. In this paper, an objective method is proposed to quantify the changes in trunk shape after surgery. Using a non-invasive optical system, the whole trunk surface is acquired and reconstructed in 3D. Trunk shape is described by two functional measurements spanning the trunk length: the lateral deviation and the axial rotation. To measure the pre and postoperative differences, a correction rate is computed for both measurements. On a cohort of 36 scoliosis patients with the same spinal curve type who underwent the same surgical approach, surgery achieved a very good correction of the lateral trunk deviation (median correction of 76%) and a poor to moderate correction of the back axial rotation (median correction of 19%). These results demonstrate that after surgery, patients are still confronted with residual trunk deformity, mainly a persisting hump on the back. That can be explained by the fact that current scoliosis assessment and treatment planning are based solely on radiographic measures of the spinal deformity and do not take trunk deformity into consideration. It is believed that with our novel quantitative trunk shape descriptor, clinicians and surgeons can now objectively assess trunk deformity and postoperative shape and propose new treatment strategies that could better address patients' concern about their appearance. © (2013) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.CIHR / IRS

Personalized 3D reconstruction of the rib cage for clinical assessment of trunk deformities

Scoliosis is a 3D deformity of the spine and rib cage. Extensive validation of 3D reconstruction methods of the spine from biplanar radiography has already been published. In this article, we propose a novel method to reconstruct the rib cage, using the same biplanar views as for the 3D reconstruction of the spine, to allow clinical assessment of whole trunk deformities. This technique uses a semi-automatic segmentation of the ribs in the postero-anterior X-ray view and an interactive segmentation of partial rib edges in the lateral view. The rib midlines are automatically extracted in 2D and reconstructed in 3D using the epipolar geometry. For the ribs not visible in the lateral view, the method predicts their 3D shape. The accuracy of the proposed method has been assessed using data obtained from a synthetic bone model as a gold standard and has also been evaluated using data of real patients with scoliotic deformities. Results show that the reconstructed ribs enable a reliable evaluation of the rib axial rotation, which will allow a 3D clinical assessment of the spine and rib cage deformities.CIHR / IRS

Semiautomatic Detection of Scoliotic Rib Borders From Posteroanterior Chest Radiographs

3-D assessment of scoliotic deformities relies on an accurate 3-D reconstruction of bone structures from biplanar X-rays, which requires a precise detection and matching of anatomical structures in both views. In this paper, we propose a novel semiautomated technique for detecting complete scoliotic rib borders from PA-0° and PA-20° chest radiographs, by using an edge-following approach with multiple-path branching and oriented filtering. Edge-following processes are initiated from user starting points along upper and lower rib edges and the final rib border is obtained by finding the most parallel pair among detected edges. The method is based on a perceptual analysis leading to the assumption that no matter how bent a scoliotic rib is, it will always present relatively parallel upper and lower edges. The proposed method was tested on 44 chest radiographs of scoliotic patients and was validated by comparing pixels from all detected rib borders against their reference locations taken from the associated manually delineated rib borders. The overall 2-D detection accuracy was 2.64 ± 1.21 pixels. Comparing this accuracy level to reported results in the literature shows that the proposed method is very well suited for precisely detecting borders of scoliotic ribs from PA-0° and PA-20° chest radiographs.CIHR / IRS

Automatic Grading of Diabetic Retinopathy on a Public Database

With the growing diabetes epidemic, retina specialists have to examine a tremendous amount of fundus images for the detection and grading of diabetic retinopathy. In this study, we propose a first automatic grading system for diabetic retinopathy. First, a red lesion detection is performed to generate a lesion probability map. The latter is then represented by 35 features combining location, size and probability information, which are finally used for classification. A leave-one-out cross-validation using a random forest is conducted on a public database of 1200 images, to classify the images into 4 grades. The proposed system achieved a classification accuracy of 74.1% and a weighted kappa value of 0.731 indicating a significant agreement with the reference. These preliminary results prove that automatic DR grading is feasible, with a performance comparable to that of human experts

Multi-domain learning CNN model for microscopy image classification

For any type of microscopy image, getting a deep learning model to work well requires considerable effort to select a suitable architecture and time to train it. As there is a wide range of microscopes and experimental setups, designing a single model that can apply to multiple imaging domains, instead of having multiple per-domain models, becomes more essential. This task is challenging and somehow overlooked in the literature. In this paper, we present a multi-domain learning architecture for the classification of microscopy images that differ significantly in types and contents. Unlike previous methods that are computationally intensive, we have developed a compact model, called Mobincep, by combining the simple but effective techniques of depth-wise separable convolution and the inception module. We also introduce a new optimization technique to regulate the latent feature space during training to improve the network's performance. We evaluated our model on three different public datasets and compared its performance in single-domain and multiple-domain learning modes. The proposed classifier surpasses state-of-the-art results and is robust for limited labeled data. Moreover, it helps to eliminate the burden of designing a new network when switching to new experiments

Modified Large Margin Nearest Neighbor Metric Learning for Regression

The main objective of this letter is to formulate a new approach of learning a Mahalanobis distance metric for nearest neighbor regression from a training sample set. We propose a modified version of the large margin nearest neighbor metric learning method to deal with regression problems. As an application, the prediction of post-operative trunk 3-D shapes in scoliosis surgery using nearest neighbor regression is described. Accuracy of the proposed method is quantitatively evaluated through experiments on real medical data.IRSC / CIH

Statistical model based 3D shape prediction of postoperative trunks for non-invasive scoliosis surgery planning

One of the major concerns of scoliosis patients undergoing surgical treatment is the aesthetic aspect of the surgery outcome. It would be useful to predict the postoperative appearance of the patient trunk in the course of a surgery planning process in order to take into account the expectations of the patient. In this paper, we propose to use least squares support vector regression for the prediction of the postoperative trunk 3D shape after spine surgery for adolescent idiopathic scoliosis. Five dimensionality reduction techniques used in conjunction with the support vector machine are compared. The methods are evaluated in terms of their accuracy, based on the leave-one-out cross-validation performed on a database of 141 cases. The results indicate that the 3D shape predictions using a dimensionality reduction obtained by simultaneous decomposition of the predictors and response variables have the best accuracy.CIHR / IRS

Multilevel Analysis of Trunk Surface Measurements for Noninvasive Assessment of Scoliosis Deformities

Study Design. Reliability study. Objectives. To assess between-acquisition reliability of new multilevel trunk cross sections measurements, in order to define what is a real change when comparing 2 trunk surface acquisitions of a same patient, before and after surgery or throughout the clinical monitoring. Summary of Background Data. Several cross-sectional surface measurements have been proposed in the literature for noninvasive assessment of trunk deformity in patients with adolescent idiopathic scoliosis (AIS). However, only the maximum values along the trunk are evaluated and used for monitoring progression and assessing treatment outcome. Methods. Back surface rotation (BSR), trunk rotation (TR), and coronal and sagittal trunk deviation are computed on 300 cross sections of the trunk. Each set of 300 measures is represented as a single functional data, using a set of basis functions. To evaluate between-acquisition variability at all trunk levels, a test-retest reliability study is conducted on 35 patients with AIS. A functional correlation analysis is also carried out to evaluate any redundancy between the measurements. Results. Each set of 300 measures was successfully described using only 10 basis functions. The test-retest reliability of the functional measurements is good to very good all over the trunk, except above the shoulders level. The typical errors of measurement are between 1.20° and 2.2° for the rotational measures and between 2 and 6 mm for deviation measures. There is a very strong correlation between BSR and TR all over the trunk, a moderate correlation between coronal trunk deviation and both BSR and TR, and no correlation between sagittal trunk deviation and any other measurement. Conclusion. This novel representation of trunk surface measurements allows for a global assessment of trunk surface deformity. Multilevel trunk measurements provide a broader perspective of the trunk deformity and allow a reliable multilevel monitoring during clinical follow-up of patients with AIS and a reliable assessment of the esthetic outcome after surgery.CIHR /IRS