41 research outputs found
Learning Task-Specific Generalized Convolutions in the Permutohedral Lattice
Dense prediction tasks typically employ encoder-decoder architectures, but
the prevalent convolutions in the decoder are not image-adaptive and can lead
to boundary artifacts. Different generalized convolution operations have been
introduced to counteract this. We go beyond these by leveraging guidance data
to redefine their inherent notion of proximity. Our proposed network layer
builds on the permutohedral lattice, which performs sparse convolutions in a
high-dimensional space allowing for powerful non-local operations despite small
filters. Multiple features with different characteristics span this
permutohedral space. In contrast to prior work, we learn these features in a
task-specific manner by generalizing the basic permutohedral operations to
learnt feature representations. As the resulting objective is complex, a
carefully designed framework and learning procedure are introduced, yielding
rich feature embeddings in practice. We demonstrate the general applicability
of our approach in different joint upsampling tasks. When adding our network
layer to state-of-the-art networks for optical flow and semantic segmentation,
boundary artifacts are removed and the accuracy is improved.Comment: To appear at GCPR 201
Permutohedral Lattice CNNs
This paper presents a convolutional layer that is able to process sparse
input features. As an example, for image recognition problems this allows an
efficient filtering of signals that do not lie on a dense grid (like pixel
position), but of more general features (such as color values). The presented
algorithm makes use of the permutohedral lattice data structure. The
permutohedral lattice was introduced to efficiently implement a bilateral
filter, a commonly used image processing operation. Its use allows for a
generalization of the convolution type found in current (spatial) convolutional
network architectures
Segmentation-Aware Convolutional Networks Using Local Attention Masks
We introduce an approach to integrate segmentation information within a
convolutional neural network (CNN). This counter-acts the tendency of CNNs to
smooth information across regions and increases their spatial precision. To
obtain segmentation information, we set up a CNN to provide an embedding space
where region co-membership can be estimated based on Euclidean distance. We use
these embeddings to compute a local attention mask relative to every neuron
position. We incorporate such masks in CNNs and replace the convolution
operation with a "segmentation-aware" variant that allows a neuron to
selectively attend to inputs coming from its own region. We call the resulting
network a segmentation-aware CNN because it adapts its filters at each image
point according to local segmentation cues. We demonstrate the merit of our
method on two widely different dense prediction tasks, that involve
classification (semantic segmentation) and regression (optical flow). Our
results show that in semantic segmentation we can match the performance of
DenseCRFs while being faster and simpler, and in optical flow we obtain clearly
sharper responses than networks that do not use local attention masks. In both
cases, segmentation-aware convolution yields systematic improvements over
strong baselines. Source code for this work is available online at
http://cs.cmu.edu/~aharley/segaware
Video Propagation Networks
We propose a technique that propagates information forward through video
data. The method is conceptually simple and can be applied to tasks that
require the propagation of structured information, such as semantic labels,
based on video content. We propose a 'Video Propagation Network' that processes
video frames in an adaptive manner. The model is applied online: it propagates
information forward without the need to access future frames. In particular we
combine two components, a temporal bilateral network for dense and video
adaptive filtering, followed by a spatial network to refine features and
increased flexibility. We present experiments on video object segmentation and
semantic video segmentation and show increased performance comparing to the
best previous task-specific methods, while having favorable runtime.
Additionally we demonstrate our approach on an example regression task of color
propagation in a grayscale video.Comment: Appearing in Computer Vision and Pattern Recognition, 2017 (CVPR'17
Learning Inference Models for Computer Vision
Computer vision can be understood as the ability to perform 'inference' on image data. Breakthroughs in computer vision technology are often marked by advances in inference techniques, as even the model design is often dictated by the complexity of inference in them. This thesis proposes learning based inference schemes and demonstrates applications in computer vision. We propose techniques for inference in both generative and discriminative computer vision models.
Despite their intuitive appeal, the use of generative models in vision is hampered by the difficulty of posterior inference, which is often too complex or too slow to be practical. We propose techniques for improving inference in two widely used techniques: Markov Chain Monte Carlo (MCMC) sampling and message-passing inference. Our inference strategy is to learn separate discriminative models that assist Bayesian inference in a generative model. Experiments on a range of generative vision models show that the proposed techniques accelerate the inference process and/or converge to better solutions.
A main complication in the design of discriminative models is the inclusion of prior knowledge in a principled way. For better inference in discriminative models, we propose techniques that modify the original model itself, as inference is simple evaluation of the model. We concentrate on convolutional neural network (CNN) models and propose a generalization of standard spatial convolutions, which are the basic building blocks of CNN architectures, to bilateral convolutions. First, we generalize the existing use of bilateral filters and then propose new neural network architectures with learnable bilateral filters, which we call `Bilateral Neural Networks'. We show how the bilateral filtering modules can be used for modifying existing CNN architectures for better image
segmentation and propose a neural network approach for temporal information propagation in videos. Experiments demonstrate the potential of the proposed bilateral networks on a wide range of vision tasks and datasets.
In summary, we propose learning based techniques for better inference in several computer vision models ranging from inverse graphics to freely parameterized neural networks. In generative vision models, our inference techniques alleviate some of the crucial hurdles in Bayesian posterior inference, paving new ways for the use of model based machine learning in vision. In discriminative CNN models, the proposed filter generalizations aid in the design of new neural network architectures that can handle sparse high-dimensional data as well as provide a way for incorporating prior knowledge into CNNs
Deep Learning Applications in Medical Image and Shape Analysis
Deep learning is one of the most rapidly growing fields in computer and data science in the past few years. It has been widely used for feature extraction and recognition in various applications. The training process as a black-box utilizes deep neural networks, whose parameters are adjusted by minimizing the difference between the predicted feedback and labeled data (so-called training dataset). The trained model is then applied to unknown inputs to predict the results that mimic human\u27s decision-making. This technology has found tremendous success in many fields involving data analysis such as images, shapes, texts, audio and video signals and so on. In medical applications, images have been regularly used by physicians for diagnosis of diseases, making treatment plans, and tracking progress of patient treatment. One of the most challenging and common problems in image processing is segmentation of features of interest, so-called feature extraction. To this end, we aim to develop a deep learning framework in the current thesis to extract regions of interest in wound images. In addition, we investigate deep learning approaches for segmentation of 3D surface shapes as a potential tool for surface analysis in our future work. Experiments are presented and discussed for both 2D image and 3D shape analysis using deep learning networks
Probabilistic Pixel-Adaptive Refinement Networks
Encoder-decoder networks have found widespread use in various dense
prediction tasks. However, the strong reduction of spatial resolution in the
encoder leads to a loss of location information as well as boundary artifacts.
To address this, image-adaptive post-processing methods have shown beneficial
by leveraging the high-resolution input image(s) as guidance data. We extend
such approaches by considering an important orthogonal source of information:
the network's confidence in its own predictions. We introduce probabilistic
pixel-adaptive convolutions (PPACs), which not only depend on image guidance
data for filtering, but also respect the reliability of per-pixel predictions.
As such, PPACs allow for image-adaptive smoothing and simultaneously
propagating pixels of high confidence into less reliable regions, while
respecting object boundaries. We demonstrate their utility in refinement
networks for optical flow and semantic segmentation, where PPACs lead to a
clear reduction in boundary artifacts. Moreover, our proposed refinement step
is able to substantially improve the accuracy on various widely used
benchmarks.Comment: To appear at CVPR 202