109,543 research outputs found
Deep Component Analysis via Alternating Direction Neural Networks
Despite a lack of theoretical understanding, deep neural networks have
achieved unparalleled performance in a wide range of applications. On the other
hand, shallow representation learning with component analysis is associated
with rich intuition and theory, but smaller capacity often limits its
usefulness. To bridge this gap, we introduce Deep Component Analysis (DeepCA),
an expressive multilayer model formulation that enforces hierarchical structure
through constraints on latent variables in each layer. For inference, we
propose a differentiable optimization algorithm implemented using recurrent
Alternating Direction Neural Networks (ADNNs) that enable parameter learning
using standard backpropagation. By interpreting feed-forward networks as
single-iteration approximations of inference in our model, we provide both a
novel theoretical perspective for understanding them and a practical technique
for constraining predictions with prior knowledge. Experimentally, we
demonstrate performance improvements on a variety of tasks, including
single-image depth prediction with sparse output constraints
Multi-bin Trainable Linear Unit for Fast Image Restoration Networks
Tremendous advances in image restoration tasks such as denoising and
super-resolution have been achieved using neural networks. Such approaches
generally employ very deep architectures, large number of parameters, large
receptive fields and high nonlinear modeling capacity. In order to obtain
efficient and fast image restoration networks one should improve upon the above
mentioned requirements.
In this paper we propose a novel activation function, the multi-bin trainable
linear unit (MTLU), for increasing the nonlinear modeling capacity together
with lighter and shallower networks. We validate the proposed fast image
restoration networks for image denoising (FDnet) and super-resolution (FSRnet)
on standard benchmarks. We achieve large improvements in both memory and
runtime over current state-of-the-art for comparable or better PSNR accuracies
Deep Clustering With Intra-class Distance Constraint for Hyperspectral Images
The high dimensionality of hyperspectral images often results in the
degradation of clustering performance. Due to the powerful ability of deep
feature extraction and non-linear feature representation, the clustering
algorithm based on deep learning has become a hot research topic in the field
of hyperspectral remote sensing. However, most deep clustering algorithms for
hyperspectral images utilize deep neural networks as feature extractor without
considering prior knowledge constraints that are suitable for clustering. To
solve this problem, we propose an intra-class distance constrained deep
clustering algorithm for high-dimensional hyperspectral images. The proposed
algorithm constrains the feature mapping procedure of the auto-encoder network
by intra-class distance so that raw images are transformed from the original
high-dimensional space to the low-dimensional feature space that is more
conducive to clustering. Furthermore, the related learning process is treated
as a joint optimization problem of deep feature extraction and clustering.
Experimental results demonstrate the intense competitiveness of the proposed
algorithm in comparison with state-of-the-art clustering methods of
hyperspectral images
PACT: Parameterized Clipping Activation for Quantized Neural Networks
Deep learning algorithms achieve high classification accuracy at the expense
of significant computation cost. To address this cost, a number of quantization
schemes have been proposed - but most of these techniques focused on quantizing
weights, which are relatively smaller in size compared to activations. This
paper proposes a novel quantization scheme for activations during training -
that enables neural networks to work well with ultra low precision weights and
activations without any significant accuracy degradation. This technique,
PArameterized Clipping acTivation (PACT), uses an activation clipping parameter
that is optimized during training to find the right quantization
scale. PACT allows quantizing activations to arbitrary bit precisions, while
achieving much better accuracy relative to published state-of-the-art
quantization schemes. We show, for the first time, that both weights and
activations can be quantized to 4-bits of precision while still achieving
accuracy comparable to full precision networks across a range of popular models
and datasets. We also show that exploiting these reduced-precision
computational units in hardware can enable a super-linear improvement in
inferencing performance due to a significant reduction in the area of
accelerator compute engines coupled with the ability to retain the quantized
model and activation data in on-chip memories
Convex Shape Prior for Deep Neural Convolution Network based Eye Fundus Images Segmentation
Convex Shapes (CS) are common priors for optic disc and cup segmentation in
eye fundus images. It is important to design proper techniques to represent
convex shapes. So far, it is still a problem to guarantee that the output
objects from a Deep Neural Convolution Networks (DCNN) are convex shapes. In
this work, we propose a technique which can be easily integrated into the
commonly used DCNNs for image segmentation and guarantee that outputs are
convex shapes. This method is flexible and it can handle multiple objects and
allow some of the objects to be convex. Our method is based on the dual
representation of the sigmoid activation function in DCNNs. In the dual space,
the convex shape prior can be guaranteed by a simple quadratic constraint on a
binary representation of the shapes. Moreover, our method can also integrate
spatial regularization and some other shape prior using a soft thresholding
dynamics (STD) method. The regularization can make the boundary curves of the
segmentation objects to be simultaneously smooth and convex. We design a very
stable active set projection algorithm to numerically solve our model. This
algorithm can form a new plug-and-play DCNN layer called CS-STD whose outputs
must be a nearly binary segmentation of convex objects. In the CS-STD block,
the convexity information can be propagated to guide the DCNN in both forward
and backward propagation during training and prediction process. As an
application example, we apply the convexity prior layer to the retinal fundus
images segmentation by taking the popular DeepLabV3+ as a backbone network.
Experimental results on several public datasets show that our method is
efficient and outperforms the classical DCNN segmentation methods
Learning to Exploit the Prior Network Knowledge for Weakly-Supervised Semantic Segmentation
Training a Convolutional Neural Network (CNN) for semantic segmentation
typically requires to collect a large amount of accurate pixel-level
annotations, a hard and expensive task. In contrast, simple image tags are
easier to gather. With this paper we introduce a novel weakly-supervised
semantic segmentation model able to learn from image labels, and just image
labels. Our model uses the prior knowledge of a network trained for image
recognition, employing these image annotations as an attention mechanism to
identify semantic regions in the images. We then present a methodology that
builds accurate class-specific segmentation masks from these regions, where
neither external objectness nor saliency algorithms are required. We describe
how to incorporate this mask generation strategy into a fully end-to-end
trainable process where the network jointly learns to classify and segment
images. Our experiments on PASCAL VOC 2012 dataset show that exploiting these
generated class-specific masks in conjunction with our novel end-to-end
learning process outperforms several recent weakly-supervised semantic
segmentation methods that use image tags only, and even some models that
leverage additional supervision or training data
Per-Pixel Feedback for improving Semantic Segmentation
Semantic segmentation is the task of assigning a label to each pixel in the
image.In recent years, deep convolutional neural networks have been driving
advances in multiple tasks related to cognition. Although, DCNNs have resulted
in unprecedented visual recognition performances, they offer little
transparency. To understand how DCNN based models work at the task of semantic
segmentation, we try to analyze the DCNN models in semantic segmentation. We
try to find the importance of global image information for labeling pixels.
Based on the experiments on discriminative regions, and modeling of
fixations, we propose a set of new training loss functions for fine-tuning DCNN
based models. The proposed training regime has shown improvement in performance
of DeepLab Large FOV(VGG-16) Segmentation model for PASCAL VOC 2012 dataset.
However, further test remains to conclusively evaluate the benefits due to the
proposed loss functions across models, and data-sets.
Submitted in part fulfillment of the requirements for the degree of
Integrated Masters of Science in Applied Mathematics.
Update: Further Experiment showed minimal benefits.
Code Available [here](https://github.com/BardOfCodes/Seg-Unravel).Comment: 33 pages,18 figures,3 table
Discriminative Pattern Mining for Breast Cancer Histopathology Image Classification via Fully Convolutional Autoencoder
Accurate diagnosis of breast cancer in histopathology images is challenging
due to the heterogeneity of cancer cell growth as well as of a variety of
benign breast tissue proliferative lesions. In this paper, we propose a
practical and self-interpretable invasive cancer diagnosis solution. With
minimum annotation information, the proposed method mines contrast patterns
between normal and malignant images in unsupervised manner and generates a
probability map of abnormalities to verify its reasoning. Particularly, a fully
convolutional autoencoder is used to learn the dominant structural patterns
among normal image patches. Patches that do not share the characteristics of
this normal population are detected and analyzed by one-class support vector
machine and 1-layer neural network. We apply the proposed method to a public
breast cancer image set. Our results, in consultation with a senior
pathologist, demonstrate that the proposed method outperforms existing methods.
The obtained probability map could benefit the pathology practice by providing
visualized verification data and potentially leads to a better understanding of
data-driven diagnosis solutions
No Multiplication? No Floating Point? No Problem! Training Networks for Efficient Inference
For successful deployment of deep neural networks on
highly--resource-constrained devices (hearing aids, earbuds, wearables), we
must simplify the types of operations and the memory/power resources used
during inference. Completely avoiding inference-time floating-point operations
is one of the simplest ways to design networks for these highly-constrained
environments. By discretizing both our in-network non-linearities and our
network weights, we can move to simple, compact networks without floating point
operations, without multiplications, and avoid all non-linear function
computations. Our approach allows us to explore the spectrum of possible
networks, ranging from fully continuous versions down to networks with bi-level
weights and activations. Our results show that discretization can be done
without loss of performance and that we can train a network that will
successfully operate without floating-point, without multiplication, and with
less RAM on both regression tasks (auto encoding) and multi-class
classification tasks (ImageNet). The memory needed to deploy our discretized
networks is less than one third of the equivalent architecture that does use
floating-point operations
Quadratic Autoencoder (Q-AE) for Low-dose CT Denoising
Inspired by complexity and diversity of biological neurons, our group
proposed quadratic neurons by replacing the inner product in current artificial
neurons with a quadratic operation on input data, thereby enhancing the
capability of an individual neuron. Along this direction, we are motivated to
evaluate the power of quadratic neurons in popular network architectures,
simulating human-like learning in the form of quadratic-neuron-based deep
learning. Our prior theoretical studies have shown important merits of
quadratic neurons and networks in representation, efficiency, and
interpretability. In this paper, we use quadratic neurons to construct an
encoder-decoder structure, referred as the quadratic autoencoder, and apply it
to low-dose CT denoising. The experimental results on the Mayo low-dose CT
dataset demonstrate the utility of quadratic autoencoder in terms of image
denoising and model efficiency. To our best knowledge, this is the first time
that the deep learning approach is implemented with a new type of neurons and
demonstrates a significant potential in the medical imaging field
- …