TrakEM2 Software for Neural Circuit Reconstruction

A key challenge in neuroscience is the expeditious reconstruction of neuronal circuits. For model systems such as Drosophila and C. elegans, the limiting step is no longer the acquisition of imagery but the extraction of the circuit from images. For this purpose, we designed a software application, TrakEM2, that addresses the systematic reconstruction of neuronal circuits from large electron microscopical and optical image volumes. We address the challenges of image volume composition from individual, deformed images; of the reconstruction of neuronal arbors and annotation of synapses with fast manual and semi-automatic methods; and the management of large collections of both images and annotations. The output is a neural circuit of 3d arbors and synapses, encoded in NeuroML and other formats, ready for analysis

Guided Proofreading of Automatic Segmentations for Connectomics

Automatic cell image segmentation methods in connectomics produce merge and split errors, which require correction through proofreading. Previous research has identified the visual search for these errors as the bottleneck in interactive proofreading. To aid error correction, we develop two classifiers that automatically recommend candidate merges and splits to the user. These classifiers use a convolutional neural network (CNN) that has been trained with errors in automatic segmentations against expert-labeled ground truth. Our classifiers detect potentially-erroneous regions by considering a large context region around a segmentation boundary. Corrections can then be performed by a user with yes/no decisions, which reduces variation of information 7.5x faster than previous proofreading methods. We also present a fully-automatic mode that uses a probability threshold to make merge/split decisions. Extensive experiments using the automatic approach and comparing performance of novice and expert users demonstrate that our method performs favorably against state-of-the-art proofreading methods on different connectomics datasets.Comment: Supplemental material available at http://rhoana.org/guidedproofreading/supplemental.pd

Segmentation fusion for connectomics

We address the problem of automatic 3D segmentation of a stack of electron microscopy sections of brain tissue. Unlike previous efforts, where the reconstruction is usually done on a section-to-section basis, or by the agglomerative clustering of 2D segments, we leverage information from the entire volume to obtain a globally optimal 3D segmen-tation. To do this, we formulate the segmentation as the so-lution to a fusion problem. We first enumerate multiple pos-sible 2D segmentations for each section in the stack, and a set of 3D links that may connect segments across con-secutive sections. We then identify the fusion of segments and links that provide the most globally consistent segmen-tation of the stack. We show that this two-step approach of pre-enumeration and posterior fusion yields significant advantages and provides state-of-the-art reconstruction re-sults. Finally, as part of this method, we also introduce a robust rotationally-invariant set of features that we use to learn and enumerate the above 2D segmentations. Our fea-tures outperform previous connectomic-specific descriptors without relying on a large set of heuristics or manually de-signed filter banks. 1

FCNN-based axon segmentation for convection-enhanced delivery optimization

Purpose: Glioblastoma multiforme treatment is a challenging task in clinical oncology. Convection- enhanced delivery (CED) is showing encouraging but still suboptimal results due to drug leakages. Numerical models can predict drug distribution within the brain, but require retrieving brain physical properties, such as the axon diameter distribution (ADD), through axon architecture analysis. The goal of this work was to provide an automatic, accurate and fast method for axon segmentation in electronic microscopy images based on fully convolutional neural network (FCNN) as to allow automatic ADD computation. Methods: The segmentation was performed using a residual FCNN inspired by U-Net and Resnet. The FCNN training was performed exploiting mini-batch gradient descent and the Adam optimizer. The Dice coefficient was chosen as loss function. Results: The proposed segmentation method achieved results comparable with already existing methods for axon segmentation in terms of Information Theoretic Scoring (0.98 %) with a faster training (5 h on the deployed GPU) and without requiring heavy post-processing (testing time was 0.2 s with a non-optimized code). The ADDs computed from the segmented and ground-truth images were statistically equivalent. Conclusions: The algorithm proposed in this work allowed fast and accurate axon segmentation and ADD computation, showing promising performance for brain microstructure analysis for CED delivery optimization

Doctor of Philosophy in Computing

dissertationImage segmentation is the problem of partitioning an image into disjoint segments that are perceptually or semantically homogeneous. As one of the most fundamental computer vision problems, image segmentation is used as a primary step for high-level vision tasks, such as object recognition and image understanding, and has even wider applications in interdisciplinary areas, such as longitudinal brain image analysis. Hierarchical models have gained popularity as a key component in image segmentation frameworks. By imposing structures, a hierarchical model can efficiently utilize features from larger image regions and make optimal inference for final segmentation feasible. We develop a hierarchical merge tree (HMT) model for image segmentation. Motivated by the application in large-scale segmentation of neuronal structures in electron microscopy (EM) images, our model provides a compact representation of region merging hypotheses and utilizes higher order information for efficient segmentation inference. Taking advantage of supervised learning, our model is free from parameter tuning and outperforms previous state-of-the-art methods on both two-dimensional (2D) and three-dimensional EM image data sets without any change. We also extend HMT to the hierarchical merge forest (HMF) model. By identifying region correspondences, HMF utilizes inter-section information to correct intra-section errors and improves 2D EM segmentation accuracy. HMT is a generic segmentation model. We demonstrate this by applying it to natural image segmentation problems. We propose a constrained conditional model formulation with a globally optimal inference algorithm for HMT and an iterative merge tree sampling algorithm that significantly improves its performance. Experimental results show our approach achieves state-of-the-art accuracy for object-independent image segmentation. Finally, we propose a semi-supervised HMT (SSHMT) model to reduce the high demand for labeled data by supervised learning. We introduce a differentiable unsupervised loss term that enforces consistent boundary predictions and develop a Bayesian learning model that combines supervised and unsupervised information. We show that with a very small amount of labeled data, SSHMT consistently performs close to the supervised HMT with full labeled data sets and significantly outperforms HMT trained with the same labeled subsets

Augmenting Wiring Diagrams of Neural Circuits with Activity in Larval Drosophila

Neural circuit models explain an animal's behavior as evoked activity of different circuit elements of its nervous system. Synaptic wiring diagrams mapped from structural imaging of nervous systems guide modeling of neural circuits on the basis of connectivity. However, connectivity alone may not sufficiently constrain the set of plausible circuit hypotheses for empirical study. Combining structural imaging of synaptic connectivity with functional information from activity imaging can further constrain these hypotheses to create unequivocal neural circuit models. This thesis develops computational methods and tools to cross-reference structural and activity imaging of explant larval Drosophila central nervous systems at cellular resolution. Augmenting synaptic wiring diagrams with activity maps via these methods relates circuit structure and function at the neuronal level on a per-behavior basis. Neuronal activity of larval central nervous systems expressing pan-neuronal calcium indicators is imaged in a light sheet microscope, which are then structurally imaged with high throughput electron microscopy. Methods and tools are provided for the assembly of these image volumes, spatial registration between imaging modalities, automated detection of relevant tissue and cellular structures in each, extraction of activity time series, and morphological identification of neurons in structural imaging using reference wiring diagrams mapped from other larvae. Using these methods, existing wiring diagrams mapped from a reference first instar larva were identified with neurons in a larva augmented with activity information for a neural circuit involved in peristaltic motor control. This demonstrates the feasibility of the contributed methods to associate the wiring diagrams of arbitrary circuits of interest with activity timeseries across multiple individuals, behaviors, and behavioral bouts. To demonstrate capability to augment wiring diagrams with information besides activity, these methods are also applied to multiple larvae each expressing specific neurotransmitter labels rather than calcium indicators in the light sheet microscopy. This work scaffolds future modeling of circuits underlying behavior that can only be mechanistically understood in the context of multi-modal observation of synaptic connectivity, functional activity and molecular markers. The methods developed also enable comparative connectomics between multiple individuals, which is necessary to study inter-individual variability in circuits and to observe experimental intervention in the development, structure, and function of neural circuits.Howard Hughes Medical Institute Janelia Research Campu

Design CNN On Bone Spine Segmention TO Methodes Image Processing

This thesis proposes a deep learning approach to bone segmentation in abdominal CNN+PG. Segmentation is a common initial step in medical images analysis, often fundamental for computer-aided detection and diagnosis systems. The extraction of bones in PG is a challenging task, which if done manually by experts requires a time consuming process and that has not today a broadly recognized automatic solution. The method presented is based on a convolutional neural network, inspired by the U-Net and trained end-to-end, that performs a semantic segmentation of the data. The training dataset is made up of 21 abdominal PG+CNN, each one containing between 0 and 255 2D transversal images. Those images are in full resolution, 4*4*50 voxels, and each voxel is classified by the network into one of the following classes: background, femoral bones, hips, sacrum, sternum, spine and ribs. The output is therefore a bone mask where the bones are recognized and divided into six different classes. In the testing dataset, labeled by experts, the best model achieves a Dice coefficient as average of all bone classes of 0.8980. This work demonstrates, to the best of my knowledge for the first time, the feasibility of automatic bone segmentation and classification for PG using a convolutional neural network

Automated Detection and Segmentation of Synaptic Contacts in Nearly Isotropic Serial Electron Microscopy Images

We describe a protocol for fully automated detection and segmentation of asymmetric, presumed excitatory, synapses in serial electron microscopy images of the adult mammalian cerebral cortex, taken with the focused ion beam, scanning electron microscope (FIB/SEM). The procedure is based on interactive machine learning and only requires a few labeled synapses for training. The statistical learning is performed on geometrical features of 3D neighborhoods of each voxel and can fully exploit the high z-resolution of the data. On a quantitative validation dataset of 111 synapses in 409 images of 1948×1342 pixels with manual annotations by three independent experts the error rate of the algorithm was found to be comparable to that of the experts (0.92 recall at 0.89 precision). Our software offers a convenient interface for labeling the training data and the possibility to visualize and proofread the results in 3D. The source code, the test dataset and the ground truth annotation are freely available on the website http://www.ilastik.org/synapse-detection

Reconstruction of 3D Neuronal Structures from Densely Packed Electron Microscopy Data Stacks

The goal of fully decoding how the brain works requires a detailed wiring diagram of the brain network that reveals the complete connectivity matrix. Recent advances in high-throughput 3D electron microscopy (EM) image acquisition techniques have made it possible to obtain high-resolution 3D imaging data that allows researchers to follow axons and dendrites and to identify pre-synaptic and post-synaptic sites, enabling the reconstruction of detailed neural circuits of the nervous system at the level of synapses. However, these massive data sets pose unique challenges to structural reconstruction because the inevitable staining noise, incomplete boundaries, and inhomogeneous staining intensities increase difficulty of 3D reconstruction and visualization. In this dissertation, a new set of algorithms are provided for reconstruction of neuronal morphology from stacks of serial EM images. These algorithms include (1) segmentation algorithms for obtaining the full geometry of neural circuits, (2) interactive segmentation tools for manual correction of erroneous segmentations, and (3) a validation method for obtaining a topologically correct segmentation when a set of segmentation alternatives are available. Experimental results obtained by using EM images containing densely packed cells demonstrate that (1) the proposed segmentation methods can successfully reconstruct full anatomical structures from EM images, (2) the editing tools provide a way for the user to easily and quickly refine incorrect segmentations, (3) and the validation method is effective in combining multiple segmentation results. The algorithms presented in this dissertation are expected to contribute to the reconstruction of the connectome and to open new directions in the development of reconstruction methods