528 research outputs found
Fine-grained Video Attractiveness Prediction Using Multimodal Deep Learning on a Large Real-world Dataset
Nowadays, billions of videos are online ready to be viewed and shared. Among
an enormous volume of videos, some popular ones are widely viewed by online
users while the majority attract little attention. Furthermore, within each
video, different segments may attract significantly different numbers of views.
This phenomenon leads to a challenging yet important problem, namely
fine-grained video attractiveness prediction. However, one major obstacle for
such a challenging problem is that no suitable benchmark dataset currently
exists. To this end, we construct the first fine-grained video attractiveness
dataset, which is collected from one of the most popular video websites in the
world. In total, the constructed FVAD consists of 1,019 drama episodes with
780.6 hours covering different categories and a wide variety of video contents.
Apart from the large amount of videos, hundreds of millions of user behaviors
during watching videos are also included, such as "view counts",
"fast-forward", "fast-rewind", and so on, where "view counts" reflects the
video attractiveness while other engagements capture the interactions between
the viewers and videos. First, we demonstrate that video attractiveness and
different engagements present different relationships. Second, FVAD provides us
an opportunity to study the fine-grained video attractiveness prediction
problem. We design different sequential models to perform video attractiveness
prediction by relying solely on video contents. The sequential models exploit
the multimodal relationships between visual and audio components of the video
contents at different levels. Experimental results demonstrate the
effectiveness of our proposed sequential models with different visual and audio
representations, the necessity of incorporating the two modalities, and the
complementary behaviors of the sequential prediction models at different
levels.Comment: Accepted by WWW 2018 The Big Web Trac
DeepStore: an interaction-aware Wide&Deep model for store site recommendation with attentional spatial embeddings
International audienceStore site recommendation is one of the essential business services in smart cities for brick-and-mortar enterprises. In recent years, the proliferation of multisource data in cities has fostered unprecedented opportunities to the data-driven store site recommendation, which aims at leveraging large-scale user-generated data to analyze and mine users' preferences for identifying the optimal location for a new store. However, most works in store site recommendation pay more attention to a single data source which lacks some significant data (e.g., consumption data and user profile data). In this paper, we aim to study the store site recommendation in a fine-grained manner. Specifically, we predict the consumption level of different users at the store based on multisource data, which can not only help the store placement but also benefit analyzing customer behavior in the store at different time periods. To solve this problem, we design a novel model based on the deep neural network, named DeepStore, which learns low-and high-order feature interactions explicitly and implicitly from dense and sparse features simultaneously. In particular, DeepStore incorporates three modules: 1) the cross network; 2) the deep network; and 3) the linear component. In addition, to learn the latent feature representation from multisource data, we propose two embedding methods for different types of data: 1) the filed embedding and 2) attention-based spatial embedding. Extensive experiments are conducted on a real-world dataset including store data, user data, and point-of-interest data, the results demonstrate that DeepStore outperforms the state-of-the-art models
Computational Aesthetics for Fashion
The online fashion industry is growing fast and with it, the need for advanced systems able to automatically solve different tasks in an accurate way. With the rapid advance of digital technologies, Deep Learning has played an important role in Computational Aesthetics, an interdisciplinary area that tries to bridge fine art, design, and computer science. Specifically, Computational Aesthetics aims to automatize human aesthetic judgments with computational methods. In this thesis, we focus on three applications of computer vision in fashion, and we discuss how Computational Aesthetics helps solve them accurately
Multimodal sentiment analysis in real-life videos
This thesis extends the emerging field of multimodal sentiment analysis of real-life videos, taking two components into consideration: the emotion and the emotion's target.
The emotion component of media is traditionally represented as a segment-based intensity model of emotion classes. This representation is replaced here by a value- and time-continuous view. Adjacent research fields, such as affective computing, have largely neglected the linguistic information available from automatic transcripts of audio-video material. As is demonstrated here, this text modality is well-suited for time- and value-continuous prediction. Moreover, source-specific problems, such as trustworthiness, have been largely unexplored so far.
This work examines perceived trustworthiness of the source, and its quantification, in user-generated video data and presents a possible modelling path. Furthermore, the transfer between the continuous and discrete emotion representations is explored in order to summarise the emotional context at a segment level.
The other component deals with the target of the emotion, for example, the topic the speaker is addressing. Emotion targets in a video dataset can, as is shown here, be coherently extracted based on automatic transcripts without limiting a priori parameters, such as the expected number of targets. Furthermore, alternatives to purely linguistic investigation in predicting targets, such as knowledge-bases and multimodal systems, are investigated.
A new dataset is designed for this investigation, and, in conjunction with proposed novel deep neural networks, extensive experiments are conducted to explore the components described above.
The developed systems show robust prediction results and demonstrate strengths of the respective modalities, feature sets, and modelling techniques. Finally, foundations are laid for cross-modal information prediction systems with applications to the correction of corrupted in-the-wild signals from real-life videos
Automated Deception Detection from Videos: Using End-to-End Learning Based High-Level Features and Classification Approaches
Deception detection is an interdisciplinary field attracting researchers from
psychology, criminology, computer science, and economics. We propose a
multimodal approach combining deep learning and discriminative models for
automated deception detection. Using video modalities, we employ convolutional
end-to-end learning to analyze gaze, head pose, and facial expressions,
achieving promising results compared to state-of-the-art methods. Due to
limited training data, we also utilize discriminative models for deception
detection. Although sequence-to-class approaches are explored, discriminative
models outperform them due to data scarcity. Our approach is evaluated on five
datasets, including a new Rolling-Dice Experiment motivated by economic
factors. Results indicate that facial expressions outperform gaze and head
pose, and combining modalities with feature selection enhances detection
performance. Differences in expressed features across datasets emphasize the
importance of scenario-specific training data and the influence of context on
deceptive behavior. Cross-dataset experiments reinforce these findings. Despite
the challenges posed by low-stake datasets, including the Rolling-Dice
Experiment, deception detection performance exceeds chance levels. Our proposed
multimodal approach and comprehensive evaluation shed light on the potential of
automating deception detection from video modalities, opening avenues for
future research.Comment: 29 pages, 17 figures (19 if counting subfigures
Recommended from our members
Inter-battery topic representation learning
In this paper, we present the Inter-Battery Topic Model (IBTM). Our
approach extends traditional topic models by learning a factorized latent variable representation. The structured representation leads to a model that marries benefits traditionally associated with a discriminative approach, such as feature selection, with those of a generative model, such as principled regularization and ability to handle missing data. The factorization is provided by representing data in terms of aligned pairs of observations as different views. This provides means for selecting a representation that separately models topics that exist in both views from the topics that are unique to a single view. This structured consolidation allows for efficient and robust inference and provides a compact and efficient representation. Learning is performed in a Bayesian fashion by maximizing a rigorous
bound on the log-likelihood. Firstly, we illustrate the benefits of the model on a synthetic dataset,. The model is then evaluated in both uni- and multi-modality settings on two different classification tasks with off-the-shelf convolutional neural network (CNN) features which generate state-of-the-art results with extremely compact representations
Learning Profitable NFT Image Diffusions via Multiple Visual-Policy Guided Reinforcement Learning
We study the task of generating profitable Non-Fungible Token (NFT) images
from user-input texts. Recent advances in diffusion models have shown great
potential for image generation. However, existing works can fall short in
generating visually-pleasing and highly-profitable NFT images, mainly due to
the lack of 1) plentiful and fine-grained visual attribute prompts for an NFT
image, and 2) effective optimization metrics for generating high-quality NFT
images. To solve these challenges, we propose a Diffusion-based generation
framework with Multiple Visual-Policies as rewards (i.e., Diffusion-MVP) for
NFT images. The proposed framework consists of a large language model (LLM), a
diffusion-based image generator, and a series of visual rewards by design.
First, the LLM enhances a basic human input (such as "panda") by generating
more comprehensive NFT-style prompts that include specific visual attributes,
such as "panda with Ninja style and green background." Second, the
diffusion-based image generator is fine-tuned using a large-scale NFT dataset
to capture fine-grained image styles and accessory compositions of popular NFT
elements. Third, we further propose to utilize multiple visual-policies as
optimization goals, including visual rarity levels, visual aesthetic scores,
and CLIP-based text-image relevances. This design ensures that our proposed
Diffusion-MVP is capable of minting NFT images with high visual quality and
market value. To facilitate this research, we have collected the largest
publicly available NFT image dataset to date, consisting of 1.5 million
high-quality images with corresponding texts and market values. Extensive
experiments including objective evaluations and user studies demonstrate that
our framework can generate NFT images showing more visually engaging elements
and higher market value, compared with SOTA approaches
- …