Silver Standard Masks for Data Augmentation Applied to Deep-Learning-Based Skull-Stripping

The bottleneck of convolutional neural networks (CNN) for medical imaging is the number of annotated data required for training. Manual segmentation is considered to be the "gold-standard". However, medical imaging datasets with expert manual segmentation are scarce as this step is time-consuming and expensive. We propose in this work the use of what we refer to as silver standard masks for data augmentation in deep-learning-based skull-stripping also known as brain extraction. We generated the silver standard masks using the consensus algorithm Simultaneous Truth and Performance Level Estimation (STAPLE). We evaluated CNN models generated by the silver and gold standard masks. Then, we validated the silver standard masks for CNNs training in one dataset, and showed its generalization to two other datasets. Our results indicated that models generated with silver standard masks are comparable to models generated with gold standard masks and have better generalizability. Moreover, our results also indicate that silver standard masks could be used to augment the input dataset at training stage, reducing the need for manual segmentation at this step

Aprendizado profundo para análise do cérebro em imagens de ressonância magnética

Orientadores: Roberto de Alencar Lotufo, Sebastien Ourselin e Leticia RittnerDissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia Elétrica e de Computação e University College LondonResumo: Redes neurais convolucionais (CNNs-Convolutional neural networks) são uma vertente do apredizado profundo que obtiveram muito sucesso quando aplicadas em várias análises em imagens de ressonância magnética (MR-magnetic resonance) do cérebro. As CNNs são métodos de aprendizagem de representação com várias camadas empilhadas compostas por uma operação de convolução seguida de uma ativação não linear e de camadas de agru- pamento. Nessas redes, cada camada gera uma representação mais alta e mais abstrata de uma determinada entrada, na qual os pesos das camadas convolucionais são aprendidos por um problema de otimização. Neste trabalho, tratamos dois problemas usando aborda- gens baseadas em aprendizagem profunda: remoção da calota craniana (SS) e tractografia. Primeiramente, propusemos um SS completo baseado em CNN treinado com o que nos referimos como máscaras de padrão de prata. A segmentação de tecido cerebral a partir de tecido não cerebral é um processo conhecido como extração da calota craniana ou re- moção de crânios. As máscaras de padrão de prata são geradas pela formação do consenso a partir de um conjunto de oito métodos de SS públicos, não baseados em aprendizagem profunda, usando o algoritmo Verdade Simultânea e Estimativa do Nível de Desempenho (STAPLE-Simultaneous Truth and Performance Level Estimation). Nossa abordagem al- cançou o desempenho do estado da arte, generalizou de forma otimizada, diminuiu a variabilidade inter / intra-avaliador e evitou a super-especialização da segmentação da CNN em relação a uma anotação manual específica. Em segundo lugar, investigamos uma solução de tractografia baseada em CNN para cirurgia de epilepsia. O principal objetivo desta análise foi estruturar uma linha de base para uma regressão baseada em aprendiza- gem profunda para prever as orientações da fibra da matéria branca. Tractografia é uma visualização das fibras ou tratos da substância branca; seu objetivo no planejamento pré- operatório é simplesmente identificar a posição de caminhos eloqüentes, como os tratos motor, sensorial e de linguagem, para reduzir o risco de danificar essas estruturas críticas. Realizamos uma análise em um único paciente e também uma análise entre 10 pacientes em uma abordagem de validação cruzada. Nossos resultados não foram ótimos, entretanto, as fibras preditas pelo algoritmo tenderam a ter um comprimento similar e convergiram para os locais médios do trato das fibras. Além disso, até onde sabemos, nosso método é a primeira abordagem que investiga CNNs para tractografia, e assim, nosso trabalho é uma base para este tópicoAbstract: Convolutional neural networks (CNNs) are one branch of deep learning that have per- formed successfully in many brain magnetic resonance (MR) imaging analysis. CNNs are representation-learning methods with stacked layers comprised of a convolution op- eration followed by a non-linear activation and pooling layers. In these networks, each layer outputs a higher and more abstract representation from a given input, in which the weights of the convolutional layers are learned by an optimization problem. In this work, we tackled two problems using deep-learning-based approaches: skull-stripping (SS) and tractography. We firstly proposed a full CNN-based SS trained with what we refer to as silver standard masks. Segmenting brain tissue from non-brain tissue is a process known as brain extraction or skull-stripping. Silver standard masks are generated by forming the consensus from a set of eight, public, non-deep-learning-based SS methods using the algo- rithm Simultaneous Truth and Performance Level Estimation (STAPLE). Our approach reached state-of-the-art performance, generalized optimally, decreased inter-/intra-rater variability, and avoided CNN segmentation overfitting towards one specific manual anno- tation. Secondly, we investigated a CNN-based tractography solution for epilepsy surgery. The main goal of this analysis was to structure a baseline for a deep-learning-based- regression to predict white matter fiber orientations. Tractography is a visualization of the white matter fibers or tracts; its goal in presurgical planing is simply to identify the position of eloquent pathways, such as the motor, sensory, and language tracts to reduce the risk of damaging these critical structures. We performed analysis cross-validation us- ing only in a single patient per time, and also, training with data from 10 patients for training the CNN. Our results were not optimal, however, the tracts tended to be of a similar length and converged to the mean fiber tract locations. Additionally, to the best of our knowledge, our method is the first approach that investigates CNNs for tractography, and thus, our work is a baseline for this topicMestradoEngenharia de ComputaçãoMestre em Engenharia Elétrica2016/18332-8, 2017/23747-5FAPES

An open, multi-vendor, multi-field-strength brain MR dataset and analysis of publicly available skull stripping methods agreement

This paper presents an open, multi-vendor, multi-field strength magnetic resonance (MR) T1-weighted volumetric brain imaging dataset, named Calgary-Campinas-359 (CC-359). The dataset is composed of images of older healthy adults (29-80 years) acquired on scanners from three vendors (Siemens, Philips and General Electric) at both 1.5 T and 3 T. CC-359 is comprised of 359 datasets, approximately 60 subjects per vendor and magnetic field strength. The dataset is approximately age and gender balanced, subject to the constraints of the available images. It provides consensus brain extraction masks for all volumes generated using supervised classification. Manual segmentation results for twelve randomly selected subjects performed by an expert are also provided. The CC-359 dataset allows investigation of 1) the influences of both vendor and magnetic field strength on quantitative analysis of brain MR; 2) parameter optimization for automatic segmentation methods; and potentially 3) machine learning classifiers with big data, specifically those based on deep learning methods, as these approaches require a large amount of data. To illustrate the utility of this dataset, we compared to the results of a supervised classifier, the results of eight publicly available skull stripping methods and one publicly available consensus algorithm. A linear mixed effects model analysis indicated that vendor (p - value < 0.001) and magnetic field strength (p - value < 0.001) have statistically significant impacts on skull stripping results170482494CONSELHO NACIONAL DE DESENVOLVIMENTO CIENTÍFICO E TECNOLÓGICO - CNPQCOORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR - CAPESFUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO DE SÃO PAULO - FAPESP311228/2014-3; 157534/2015-488881.062158/2014-012013/07559-3; 2013/23514-0; 2016/18332-

Medical Image Segmentation: Thresholding and Minimum Spanning Trees

I bildesegmentering deles et bilde i separate objekter eller regioner. Det er et essensielt skritt i bildebehandling for å definere interesseområder for videre behandling eller analyse. Oppdelingsprosessen reduserer kompleksiteten til et bilde for å forenkle analysen av attributtene oppnådd etter segmentering. Det forandrer representasjonen av informasjonen i det opprinnelige bildet og presenterer pikslene på en måte som er mer meningsfull og lettere å forstå. Bildesegmentering har forskjellige anvendelser. For medisinske bilder tar segmenteringsprosessen sikte på å trekke ut bildedatasettet for å identifisere områder av anatomien som er relevante for en bestemt studie eller diagnose av pasienten. For eksempel kan man lokalisere berørte eller anormale deler av kroppen. Segmentering av oppfølgingsdata og baseline lesjonssegmentering er også svært viktig for å vurdere behandlingsresponsen. Det er forskjellige metoder som blir brukt for bildesegmentering. De kan klassifiseres basert på hvordan de er formulert og hvordan segmenteringsprosessen utføres. Metodene inkluderer de som er baserte på terskelverdier, graf-baserte, kant-baserte, klynge-baserte, modell-baserte og hybride metoder, og metoder basert på maskinlæring og dyp læring. Andre metoder er baserte på å utvide, splitte og legge sammen regioner, å finne diskontinuiteter i randen, vannskille segmentering, aktive kontuter og graf-baserte metoder. I denne avhandlingen har vi utviklet metoder for å segmentere forskjellige typer medisinske bilder. Vi testet metodene på datasett for hvite blodceller (WBCs) og magnetiske resonansbilder (MRI). De utviklede metodene og analysen som er utført på bildedatasettet er presentert i tre artikler. I artikkel A (Paper A) foreslo vi en metode for segmentering av nukleuser og cytoplasma fra hvite blodceller. Metodene estimerer terskelen for segmentering av nukleuser automatisk basert på lokale minima. Metoden segmenterer WBC-ene før segmentering av cytoplasma avhengig av kompleksiteten til objektene i bildet. For bilder der WBC-ene er godt skilt fra røde blodlegemer (RBC), er WBC-ene segmentert ved å ta gjennomsnittet av $n$ bilder som allerede var filtrert med en terskelverdi. For bilder der RBC-er overlapper WBC-ene, er hele WBC-ene segmentert ved hjelp av enkle lineære iterative klynger (SLIC) og vannskillemetoder. Cytoplasmaet oppnås ved å trekke den segmenterte nukleusen fra den segmenterte WBC-en. Metoden testes på to forskjellige offentlig tilgjengelige datasett, og resultatene sammenlignes med toppmoderne metoder. I artikkel B (Paper B) foreslo vi en metode for segmentering av hjernesvulster basert på minste dekkende tre-konsepter (minimum spanning tree, MST). Metoden utfører interaktiv segmentering basert på MST. I denne artikkelen er bildet lastet inn i et interaktivt vindu for segmentering av svulsten. Fokusregion og bakgrunn skilles ved å klikke for å dele MST i to trær. Ett av disse trærne representerer fokusregionen og det andre representerer bakgrunnen. Den foreslåtte metoden ble testet ved å segmentere to forskjellige 2D-hjerne T1 vektede magnetisk resonans bildedatasett. Metoden er enkel å implementere og resultatene indikerer at den er nøyaktig og effektiv. I artikkel C (Paper C) foreslår vi en metode som behandler et 3D MRI-volum og deler det i hjernen, ikke-hjernevev og bakgrunnsegmenter. Det er en grafbasert metode som bruker MST til å skille 3D MRI inn i de tre regiontypene. Grafen lages av et forhåndsbehandlet 3D MRI-volum etterfulgt av konstrueringen av MST-en. Segmenteringsprosessen gir tre merkede, sammenkoblende komponenter som omformes tilbake til 3D MRI-form. Etikettene brukes til å segmentere hjernen, ikke-hjernevev og bakgrunn. Metoden ble testet på tre forskjellige offentlig tilgjengelige datasett og resultatene ble sammenlignet med ulike toppmoderne metoder.In image segmentation, an image is divided into separate objects or regions. It is an essential step in image processing to define areas of interest for further processing or analysis. The segmentation process reduces the complexity of an image to simplify the analysis of the attributes obtained after segmentation. It changes the representation of the information in the original image and presents the pixels in a way that is more meaningful and easier to understand. Image segmentation has various applications. For medical images, the segmentation process aims to extract the image data set to identify areas of the anatomy relevant to a particular study or diagnosis of the patient. For example, one can locate affected or abnormal parts of the body. Segmentation of follow-up data and baseline lesion segmentation is also very important to assess the treatment response. There are different methods used for image segmentation. They can be classified based on how they are formulated and how the segmentation process is performed. The methods include those based on threshold values, edge-based, cluster-based, model-based and hybrid methods, and methods based on machine learning and deep learning. Other methods are based on growing, splitting and merging regions, finding discontinuities in the edge, watershed segmentation, active contours and graph-based methods. In this thesis, we have developed methods for segmenting different types of medical images. We tested the methods on datasets for white blood cells (WBCs) and magnetic resonance images (MRI). The developed methods and the analysis performed on the image data set are presented in three articles. In Paper A we proposed a method for segmenting nuclei and cytoplasm from white blood cells. The method estimates the threshold for segmentation of nuclei automatically based on local minima. The method segments the WBCs before segmenting the cytoplasm depending on the complexity of the objects in the image. For images where the WBCs are well separated from red blood cells (RBCs), the WBCs are segmented by taking the average of $n$ images that were already filtered with a threshold value. For images where RBCs overlap the WBCs, the entire WBCs are segmented using simple linear iterative clustering (SLIC) and watershed methods. The cytoplasm is obtained by subtracting the segmented nucleus from the segmented WBC. The method is tested on two different publicly available datasets, and the results are compared with state of the art methods. In Paper B, we proposed a method for segmenting brain tumors based on minimum spanning tree (MST) concepts. The method performs interactive segmentation based on the MST. In this paper, the image is loaded in an interactive window for segmenting the tumor. The region of interest and the background are selected by clicking to split the MST into two trees. One of these trees represents the region of interest and the other represents the background. The proposed method was tested by segmenting two different 2D brain T1-weighted magnetic resonance image data sets. The method is simple to implement and the results indicate that it is accurate and efficient. In Paper C, we propose a method that processes a 3D MRI volume and partitions it into brain, non-brain tissues, and background segments. It is a graph-based method that uses MST to separate the 3D MRI into the brain, non-brain, and background regions. The graph is made from a preprocessed 3D MRI volume followed by constructing the MST. The segmentation process produces three labeled connected components which are reshaped back to the shape of the 3D MRI. The labels are used to segment the brain, non-brain tissues, and the background. The method was tested on three different publicly available data sets and the results were compared to different state of the art methods.Doktorgradsavhandlin

Abordagem CNN 2D estendida para o diagnóstico da doença de Alzheimer através de imagens de ressonância magnética estrutural

Orientadores: Leticia Rittner, Roberto de Alencar LotufoDissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia Elétrica e de ComputaçãoResumo: A doença de Alzheimer (AD - Alzheimer's disease) é um tipo de demência que afeta milhões de pessoas em todo o mundo. Até o momento, não há cura para a doença e seu diagnóstico precoce tem sido uma tarefa desafiadora. As técnicas atuais para o seu diagnóstico têm explorado as informações estruturais da Imagem por Ressonância Magnética (MRI - Magnetic Resonance Imaging) em imagens ponderadas em T1. Entre essas técnicas, a rede neural convolucional (CNN - Convolutional Neural Network) é a mais promissora e tem sido usada com sucesso em imagens médicas para uma variedade de aplicações devido à sua capacidade de extração de características. Antes do grande sucesso do aprendizado profundo e das CNNs, os trabalhos que objetivavam classificar os diferentes estágios de AD exploraram abordagens clássicas de aprendizado de máquina e uma meticulosa extração de características, principalmente para classificar testes binários. Recentemente, alguns autores combinaram técnicas de aprendizagem profunda e pequenos subconjuntos do conjunto de dados públicos da Iniciativa de Neuroimagem da Doença de Alzheimer (ADNI - Alzheimer's Disease Neuroimaging Initiative) para prever um estágio inicial da doença explorando abordagens 3D CNN geralmente combinadas com arquiteturas de auto-codificador convolucional 3D. Outros também exploraram uma abordagem de CNN 3D combinando-a ou não com uma etapa de pré-processamento para a extração de características. No entanto, a maioria desses trabalhos focam apenas na classificação binária, sem resultados para AD, comprometimento cognitivo leve (MCI - Mild Cognitive Impairment) e classificação de sujeitos normais (NC - Normal Control). Nosso principal objetivo foi explorar abordagens de CNN 2D para a tarefa de classificação das 3 classes usando imagens de MRI ponderadas em T1. Como objetivo secundário, preenchemos algumas lacunas encontradas na literatura ao investigar o uso de arquiteturas CNN 2D para o nosso problema, uma vez que a maioria dos trabalhos explorou o aprendizado de máquina clássico ou abordagens CNN 3D. Nossa abordagem CNN 2D estendida explora as informações volumétricas dos dados de ressonância magnética, mantendo baixo custo computacional associado a uma abordagem 2D, quando comparados às abordagens 3D. Além disso, nosso resultado supera as outras estratégias para a classificação das 3 classes e comparando o desempenho de nosso modelo com os métodos tradicionais de aprendizado de máquina e 3D CNN. Também investigamos o papel de diferentes técnicas amplamente utilizadas em aplicações CNN, por exemplo, pré-processamento de dados, aumento de dados, transferência de aprendizado e adaptação de domínio para um conjunto de dados brasileiroAbstract: Alzheimer's disease (AD) is a type of dementia that affects millions of people around the world. To date, there is no cure for Alzheimer's and its early-diagnosis has been a challenging task. The current techniques for Alzheimer's disease diagnosis have explored the structural information of Magnetic Resonance Imaging (MRI) in T1-weighted images. Among these techniques, deep convolutional neural network (CNN) is the most promising one and has been successfully used in medical images for a variety of applications due to its ability to perform features extraction. Before the great success of deep learning and CNNs, the works that aimed to classify the different stages of AD explored classic machine learning approaches and a meticulous feature engineering extraction, mostly to classify binary tasks. Recently, some authors have combined deep learning techniques and small subsets from the Alzheimer's Disease Neuroimaging Initiative (ADNI) public dataset, to predict an early-stage of AD exploring 3D CNN approaches usually combined with 3D convolutional autoencoder architectures. Others have also investigated a 3D CNN approach combining it or not with a pre-processing step for the extraction of features. However, the majority of these papers focus on binary classification only, with no results for Alzheimer's disease, Mild Cognitive Impairment (MCI), and Normal Control (NC) classification. Our primary goal was to explore 2D CNN approaches to tackle the 3-class classification using T1-weighted MRI. As a secondary goal, we filled some gaps we found in the literature by investigating the use of 2D CNN architectures to our problem, since most of the works either explored traditional machine learning or 3D CNN approaches. Our extended-2D CNN explores the MRI volumetric data information while maintaining the low computational costs associated with a 2D approach when compared to 3D-CNNs. Besides, our result overcomes the other strategies for the 3-class classification while analyzing the performance of our model with traditional machine-learning and 3D-CNN methods. We also investigated the role of different widely used techniques in CNN applications, for instance, data pre-processing, data augmentation, transfer-learning, and domain-adaptation to a Brazilian datasetMestradoEngenharia de ComputaçãoMestra em Engenharia Elétrica168468/2017-4  CNP

Similar neural pathways link psychological stress and brain-age in health and multiple sclerosis

Clinical and neuroscientific studies suggest a link between psychological stress and reduced brain health in health and neurological disease but it is unclear whether mediating pathways are similar. Consequently, we applied an arterial-spin-labeling MRI stress task in 42 healthy persons and 56 with multiple sclerosis, and investigated regional neural stress responses, associations between functional connectivity of stress-responsive regions and the brain-age prediction error, a highly sensitive machine learning brain health biomarker, and regional brain-age constituents in both groups. Stress responsivity did not differ between groups. Although elevated brain-age prediction errors indicated worse brain health in patients, anterior insula–occipital cortex (healthy persons: occipital pole; patients: fusiform gyrus) functional connectivity correlated with brain-age prediction errors in both groups. Finally, also gray matter contributed similarly to regional brain-age across groups. These findings might suggest a common stress–brain health pathway whose impact is amplified in multiple sclerosis by disease-specific vulnerability factors