17 research outputs found
Image embedding and user multi-preference modeling for data collection sampling
This work proposes an end-to-end user-centric sampling method aimed at selecting the images from an image collection that are able to maximize the information perceived by a given user. As main contributions, we first introduce novel metrics that assess the amount of perceived information retained by the user when experiencing a set of images. Given the actual information present in a set of images, which is the volume spanned by the set in the corresponding latent space, we show how to take into account the user’s preferences in such a volume calculation to build a user-centric metric for the perceived information. Finally, we propose a sampling strategy seeking the minimum set of images that maximize the information perceived by a given user. Experiments using the coco dataset show the ability of the proposed approach to accurately integrate user preference while keeping a reasonable diversity in the sampled image set
HyperLearn: A Distributed Approach for Representation Learning in Datasets With Many Modalities
Multimodal datasets contain an enormous amount of relational information,
which grows exponentially with the introduction of new modalities. Learning
representations in such a scenario is inherently complex due to the presence of
multiple heterogeneous information channels. These channels can encode both (a)
inter-relations between the items of different modalities and (b)
intra-relations between the items of the same modality. Encoding multimedia
items into a continuous low-dimensional semantic space such that both types of
relations are captured and preserved is extremely challenging, especially if
the goal is a unified end-to-end learning framework. The two key challenges
that need to be addressed are: 1) the framework must be able to merge complex
intra and inter relations without losing any valuable information and 2) the
learning model should be invariant to the addition of new and potentially very
different modalities. In this paper, we propose a flexible framework which can
scale to data streams from many modalities. To that end we introduce a
hypergraph-based model for data representation and deploy Graph Convolutional
Networks to fuse relational information within and across modalities. Our
approach provides an efficient solution for distributing otherwise extremely
computationally expensive or even unfeasible training processes across
multiple-GPUs, without any sacrifices in accuracy. Moreover, adding new
modalities to our model requires only an additional GPU unit keeping the
computational time unchanged, which brings representation learning to truly
multimodal datasets. We demonstrate the feasibility of our approach in the
experiments on multimedia datasets featuring second, third and fourth order
relations
Blind Quality Assessment for Image Superresolution Using Deep Two-Stream Convolutional Networks
Numerous image superresolution (SR) algorithms have been proposed for
reconstructing high-resolution (HR) images from input images with lower spatial
resolutions. However, effectively evaluating the perceptual quality of SR
images remains a challenging research problem. In this paper, we propose a
no-reference/blind deep neural network-based SR image quality assessor
(DeepSRQ). To learn more discriminative feature representations of various
distorted SR images, the proposed DeepSRQ is a two-stream convolutional network
including two subcomponents for distorted structure and texture SR images.
Different from traditional image distortions, the artifacts of SR images cause
both image structure and texture quality degradation. Therefore, we choose the
two-stream scheme that captures different properties of SR inputs instead of
directly learning features from one image stream. Considering the human visual
system (HVS) characteristics, the structure stream focuses on extracting
features in structural degradations, while the texture stream focuses on the
change in textural distributions. In addition, to augment the training data and
ensure the category balance, we propose a stride-based adaptive cropping
approach for further improvement. Experimental results on three publicly
available SR image quality databases demonstrate the effectiveness and
generalization ability of our proposed DeepSRQ method compared with
state-of-the-art image quality assessment algorithms