Diffusion Mechanism in Residual Neural Network: Theory and Applications

Diffusion, a fundamental internal mechanism emerging in many physical processes, describes the interaction among different objects. In many learning tasks with limited training samples, the diffusion connects the labeled and unlabeled data points and is a critical component for achieving high classification accuracy. Many existing deep learning approaches directly impose the fusion loss when training neural networks. In this work, inspired by the convection-diffusion ordinary differential equations (ODEs), we propose a novel diffusion residual network (Diff-ResNet), internally introduces diffusion into the architectures of neural networks. Under the structured data assumption, it is proved that the proposed diffusion block can increase the distance-diameter ratio that improves the separability of inter-class points and reduces the distance among local intra-class points. Moreover, this property can be easily adopted by the residual networks for constructing the separable hyperplanes. Extensive experiments of synthetic binary classification, semi-supervised graph node classification and few-shot image classification in various datasets validate the effectiveness of the proposed method

Boosting Standard Classification Architectures Through a Ranking Regularizer

We employ triplet loss as a feature embedding regularizer to boost classification performance. Standard architectures, like ResNet and Inception, are extended to support both losses with minimal hyper-parameter tuning. This promotes generality while fine-tuning pretrained networks. Triplet loss is a powerful surrogate for recently proposed embedding regularizers. Yet, it is avoided due to large batch-size requirement and high computational cost. Through our experiments, we re-assess these assumptions. During inference, our network supports both classification and embedding tasks without any computational overhead. Quantitative evaluation highlights a steady improvement on five fine-grained recognition datasets. Further evaluation on an imbalanced video dataset achieves significant improvement. Triplet loss brings feature embedding characteristics like nearest neighbor to classification models. Code available at \url{http://bit.ly/2LNYEqL}.Comment: WACV 2020 Camera ready + supplementary materia

Registration and analysis of dynamic magnetic resonance image series

Cystic fibrosis (CF) is an autosomal-recessive inherited metabolic disorder that affects all organs in the human body. Patients affected with CF suffer particularly from chronic inflammation and obstruction of the airways. Through early detection, continuous monitoring methods, and new treatments, the life expectancy of patients with CF has been increased drastically in the last decades. However, continuous monitoring of the disease progression is essential for a successful treatment. The current state-of-the-art method for lung disease detection and monitoring is computed tomography (CT) or X-ray. These techniques are ill-suited for the monitoring of disease progressions because of the ionizing radiation the patient is exposed during the examination. Through the development of new magnetic resonance imaging (MRI) sequences and evaluation methods, MRI is able to measure physiological changes in the lungs. The process to create physiological maps, i.e. ventilation and perfusion maps, of the lungs using MRI can be split up into three parts: MR-acquisition, image registration, and image analysis. In this work, we present different methods for the image registration part and the image analysis part. We developed a graph-based registration method for 2D dynamic MR image series of the lungs in order to overcome the problem of sliding motion at organ boundaries. Furthermore, we developed a human-inspired learning-based registration method. Here, the registration is defined as a sequence of local transformations. The sequence-based approach combines the advantage of dense transformation models, i.e. large space of transformations, and the advantage of interpolating transformation models, i.e. smooth local transformations. We also developed a general registration framework called Autograd Image Registration Laboratory (AIRLab), which performs automatic calculation of the gradients for the registration process. This allows rapid prototyping and an easy implementation of existing registration algorithms. For the image analysis part, we developed a deep-learning approach based on gated recurrent units that are able to calculate ventilation maps with less than a third of the number of images of the current method. Automatic defect detection in the estimated MRI ventilation and perfusion maps is essential for the clinical routine to automatically evaluate the treatment progression. We developed a weakly supervised method that is able to infer a pixel-wise defect segmentation by using only a continuous global label during training. In this case, we directly use the lung clearance index (LCI) as a global weak label, without any further manual annotations. The LCI is a global measure to describe ventilation inhomogeneities of the lungs and is obtained by a multiple breath washout test

The information regularization framework for semi-supervised learning

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2006.Includes bibliographical references (p. 147-154).In recent years, the study of classification shifted to algorithms for training the classifier from data that may be missing the class label. While traditional supervised classifiers already have the ability to cope with some incomplete data, the new type of classifiers do not view unlabeled data as an anomaly, and can learn from data sets in which the large majority of training points are unlabeled. Classification with labeled and unlabeled data, or semi-supervised classification, has important practical significance, as training sets with a mix of labeled an unlabeled data are commonplace. In many domains, such as categorization of web pages, it is easier to collect unlabeled data, than to annotate the training points with labels. This thesis is a study of the information regularization method for semi-supervised classification, a unified framework that encompasses many of the common approaches to semi-supervised learning, including parametric models of incomplete data, harmonic graph regularization, redundancy of sufficient features (co-training), and combinations of these principles in a single algorithm.(cont.) We discuss the framework in both parametric and non-parametric settings, as a transductive or inductive classifier, considered as a stand-alone classifier, or applied as post-processing to standard supervised classifiers. We study theoretical properties of the framework, and illustrate it on categorization of web pages, and named-entity recognition.by Adrian Corduneanu.Ph.D

Learning on Graphs with Partially Absorbing Random Walks: Theory and Practice

Learning on graphs has been studied for decades with abundant models proposed, yet many of their behaviors and relations remain unclear. This thesis fills this gap by introducing a novel second-order Markov chain, called partially absorbing random walks (ParWalk). Different from ordinary random walk, ParWalk is absorbed at the current state $i$ with probability $p_i$, and follows a random edge out with probability $1-p_i$. The partial absorption results in absorption probability between any two vertices, which turns out to encompass various popular models including PageRank, hitting times, label propagation, and regularized Laplacian kernels. The unified treatment reveals the distinguishing characteristics of these models arising from different contexts, and allows comparing them and transferring findings from one paradigm to another. The key for learning on graphs is capitalizing on the cluster structure of the underlying graph. The absorption probabilities of ParWalk, turn out to be highly effective in capturing the cluster structure. Given a query vertex $q$ in a cluster $\mathcal{S}$, we show that when the absorbing capacity ($p_i$) of each vertex on the graph is small, the probabilities of ParWalk to be absorbed at $q$ have small variations in region of high conductance (within clusters), but have large gaps in region of low conductance (between clusters). And the less absorbent the vertices of $\mathcal{S}$ are, the better the absorption probabilities can represent the local cluster $\mathcal{S}$. Our theory induces principles for designing reliable similarity measures and provides justification to a number of popular ones such as hitting times and the pseudo-inverse of graph Laplacian. Furthermore, it reveals their new important properties. For example, we are the first to show that hitting times is better in retrieving sparse clusters, while the pseudo-inverse of graph Laplacian is better for dense ones. The theoretical insights instilled from ParWalk guide us in developing robust algorithms for various applications including local clustering, semi-supervised learning, and ranking. For local clustering, we propose a new method for salient object segmentation. By taking a noisy saliency map as the probability distribution of query vertices, we compute the absorption probabilities of ParWalk to the queries, producing a high-quality refined saliency map where the objects can be easily segmented. For semi-supervised learning, we propose a new algorithm for label propagation. The algorithm is justified by our theoretical analysis and guaranteed to be superior than many existing ones. For ranking, we design a new similarity measure using ParWalk, which combines the strengths of both hitting times and the pseudo-inverse of graph Laplacian. The hybrid similarity measure can well adapt to complex data of diverse density, thus performs superiorly overall. For all these learning tasks, our methods achieve substantial improvements over the state-of-the-art on extensive benchmark datasets