29 research outputs found
Hierarchical Photo-Scene Encoder for Album Storytelling
In this paper, we propose a novel model with a hierarchical photo-scene
encoder and a reconstructor for the task of album storytelling. The photo-scene
encoder contains two sub-encoders, namely the photo and scene encoders, which
are stacked together and behave hierarchically to fully exploit the structure
information of the photos within an album. Specifically, the photo encoder
generates semantic representation for each photo while exploiting temporal
relationships among them. The scene encoder, relying on the obtained photo
representations, is responsible for detecting the scene changes and generating
scene representations. Subsequently, the decoder dynamically and attentively
summarizes the encoded photo and scene representations to generate a sequence
of album representations, based on which a story consisting of multiple
coherent sentences is generated. In order to fully extract the useful semantic
information from an album, a reconstructor is employed to reproduce the
summarized album representations based on the hidden states of the decoder. The
proposed model can be trained in an end-to-end manner, which results in an
improved performance over the state-of-the-arts on the public visual
storytelling (VIST) dataset. Ablation studies further demonstrate the
effectiveness of the proposed hierarchical photo-scene encoder and
reconstructor.Comment: 8 pages, 4 figure
Deep Learning for Dense Interpretation of Video: Survey of Various Approach, Challenges, Datasets and Metrics
Video interpretation has garnered considerable attention in computer vision and natural language processing fields due to the rapid expansion of video data and the increasing demand for various applications such as intelligent video search, automated video subtitling, and assistance for visually impaired individuals. However, video interpretation presents greater challenges due to the inclusion of both temporal and spatial information within the video. While deep learning models for images, text, and audio have made significant progress, efforts have recently been focused on developing deep networks for video interpretation. A thorough evaluation of current research is necessary to provide insights for future endeavors, considering the myriad techniques, datasets, features, and evaluation criteria available in the video domain. This study offers a survey of recent advancements in deep learning for dense video interpretation, addressing various datasets and the challenges they present, as well as key features in video interpretation. Additionally, it provides a comprehensive overview of the latest deep learning models in video interpretation, which have been instrumental in activity identification and video description or captioning. The paper compares the performance of several deep learning models in this field based on specific metrics. Finally, the study summarizes future trends and directions in video interpretation
Accurate Temporal Action Proposal Generation with Relation-Aware Pyramid Network
Accurate temporal action proposals play an important role in detecting
actions from untrimmed videos. The existing approaches have difficulties in
capturing global contextual information and simultaneously localizing actions
with different durations. To this end, we propose a Relation-aware pyramid
Network (RapNet) to generate highly accurate temporal action proposals. In
RapNet, a novel relation-aware module is introduced to exploit bi-directional
long-range relations between local features for context distilling. This
embedded module enhances the RapNet in terms of its multi-granularity temporal
proposal generation ability, given predefined anchor boxes. We further
introduce a two-stage adjustment scheme to refine the proposal boundaries and
measure their confidence in containing an action with snippet-level actionness.
Extensive experiments on the challenging ActivityNet and THUMOS14 benchmarks
demonstrate our RapNet generates superior accurate proposals over the existing
state-of-the-art methods.Comment: accepted by AAAI-2