DramaQA: Character-Centered Video Story Understanding with Hierarchical QA

Despite recent progress on computer vision and natural language processing, developing video understanding intelligence is still hard to achieve due to the intrinsic difficulty of story in video. Moreover, there is not a theoretical metric for evaluating the degree of video understanding. In this paper, we propose a novel video question answering (Video QA) task, DramaQA, for a comprehensive understanding of the video story. The DramaQA focused on two perspectives: 1) hierarchical QAs as an evaluation metric based on the cognitive developmental stages of human intelligence. 2) character-centered video annotations to model local coherence of the story. Our dataset is built upon the TV drama "Another Miss Oh" and it contains 16,191 QA pairs from 23,928 various length video clips, with each QA pair belonging to one of four difficulty levels. We provide 217,308 annotated images with rich character-centered annotations, including visual bounding boxes, behaviors, and emotions of main characters, and coreference resolved scripts. Additionally, we provide analyses of the dataset as well as Dual Matching Multistream model which effectively learns character-centered representations of video to answer questions about the video. We are planning to release our dataset and model publicly for research purposes and expect that our work will provide a new perspective on video story understanding research.Comment: 21 pages, 10 figures, submitted to ECCV 202

Mitochondrial function contributes to oxysterol-induced osteogenic differentiation in mouse embryonic stem cells

AbstractOxysterols, oxidized derivatives of cholesterol, are biologically active molecules. Specific oxysterols have potent osteogenic properties that act on osteoprogenitor cells. However, the molecular mechanisms underlying these osteoinductive effects on embryonic stem cells (ESCs) are unknown. This study investigated the effect of an oxysterol combination of 22(S)-hydroxycholesterol and 20(S)-hydroxycholesterol (SS) on osteogenic differentiation of ESCs and the alterations to mitochondrial activity during differentiation. Osteogenic differentiation was assessed by alkaline phosphatase (ALP) activity, matrix mineralization, mRNA expression of osteogenic factors, runt-related transcription factor 2, osterix, and osteocalcin, and protein levels of collagen type IA (COLIA) and osteopontin (OPN). Treatment of cells with SS increased osteoinductive activity compared to the control group. Intracellular reactive oxygen species production, intracellular ATP content, mitochondrial membrane potential, mitochondrial mass, mitochondrial DNA copy number, and mRNA expression of peroxisome proliferator-activated receptor-γ coactivators 1α and β, transcription factors involved in mitochondrial biogenesis, were significantly increased during osteogenesis, indicating upregulation of mitochondrial activity. Oxysterol combinations also increased protein levels of mitochondrial respiratory complexes I–V. We also found that SS treatment increased hedgehog signaling target genes, Smo and Gli1 expression. Inhibition of Hh signaling by cyclopamine suppressed mitochondrial biogenesis and ESC osteogenesis. Subsequently, oxysterol-induced Wnt/β-catenin pathways were inhibited by repression of Hh signaling and mitochondrial biogenesis. Transfection of β-catenin specific siRNA decreased the protein levels of COLIA and OPN, as well as ALP activity. Collectively, these data suggest that lipid-based oxysterols enhance differentiation of ESCs toward the osteogenic lineage by regulating mitochondrial activity, canonical Hh/Gli, and Wnt/β-catenin signaling

Cut-Based Graph Learning Networks to Discover Compositional Structure of Sequential Video Data

Conventional sequential learning methods such as Recurrent Neural Networks (RNNs) focus on interactions between consecutive inputs, i.e. first-order Markovian dependency. However, most of sequential data, as seen with videos, have complex dependency structures that imply variable-length semantic flows and their compositions, and those are hard to be captured by conventional methods. Here, we propose Cut-Based Graph Learning Networks (CB-GLNs) for learning video data by discovering these complex structures of the video. The CB-GLNs represent video data as a graph, with nodes and edges corresponding to frames of the video and their dependencies respectively. The CB-GLNs find compositional dependencies of the data in multilevel graph forms via a parameterized kernel with graph-cut and a message passing framework. We evaluate the proposed method on the two different tasks for video understanding: Video theme classification (Youtube-8M dataset) and Video Question and Answering (TVQA dataset). The experimental results show that our model efficiently learns the semantic compositional structure of video data. Furthermore, our model achieves the highest performance in comparison to other baseline methods.Comment: 8 pages, 3 figures, Association for the Advancement of Artificial Intelligence (AAAI2020). arXiv admin note: substantial text overlap with arXiv:1907.0170