Structured Co-reference Graph Attention for Video-grounded Dialogue

A video-grounded dialogue system referred to as the Structured Co-reference Graph Attention (SCGA) is presented for decoding the answer sequence to a question regarding a given video while keeping track of the dialogue context. Although recent efforts have made great strides in improving the quality of the response, performance is still far from satisfactory. The two main challenging issues are as follows: (1) how to deduce co-reference among multiple modalities and (2) how to reason on the rich underlying semantic structure of video with complex spatial and temporal dynamics. To this end, SCGA is based on (1) Structured Co-reference Resolver that performs dereferencing via building a structured graph over multiple modalities, (2) Spatio-temporal Video Reasoner that captures local-to-global dynamics of video via gradually neighboring graph attention. SCGA makes use of pointer network to dynamically replicate parts of the question for decoding the answer sequence. The validity of the proposed SCGA is demonstrated on AVSD@DSTC7 and AVSD@DSTC8 datasets, a challenging video-grounded dialogue benchmarks, and TVQA dataset, a large-scale videoQA benchmark. Our empirical results show that SCGA outperforms other state-of-the-art dialogue systems on both benchmarks, while extensive ablation study and qualitative analysis reveal performance gain and improved interpretability.Comment: Accepted to AAAI202

Motion-Appearance Synergistic Networks for Video Question Answering

Video Question Answering is a task which requires an AI agent to answer questions grounded in video. This task entails three key challenges: (1)understand the intention of various questions, (2) capturing various elements of the input video (e.g., object, action, causality), and (3) cross-modal grounding between language and vision information. We propose MotionAppearance Synergistic Networks (MASN), which embed two crossmodal features grounded on motion and appearance information and selectively utilize them depending on the question’s intentions. MASN consists of a motion module, an appearance module, and a motion-appearance fusion module. The motion module computes the action-oriented cross-modal joint representations, while the appearance module focuses on the appearance aspect of the input video. Finally, the motion-appearance fusion module takes each output of the motion module and the appearance module as input, and performs question-guided fusion. As a result, MASN achieves new state-of-the-art performance on the TGIF-QA and MSVD-QA datasets. We also conduct qualitative analysis by visualizing the inference results of MASN.비디오 질의 응답은 AI 에이전트가 주어진 비디오를 기반으로 관련된 질문에 응답하는 문제이다. 비디오 질의 응답 문제를 해결하기 위해서는 세 가지 과제를 해결하여야 한다: (1) 다양한 질문의 의도를 이해하고, (2) 주어진 비디오의 다양한 요소(e.g. 물체, 행동, 인과관계)를 파악하여야 하며, (3) 언어와 시각 정보 두 modality 간의 상관관계를 기반으로 생성된 표상(cross-modal representation)을 통해 정답을 추론하여야 한다. 따라서 본 학위논문에서는 동작 정보 및 모양 정보에 기반한 두 가지 cross-modal representation 을 생성하고, 이를 질문의 의도에 따라 가중합하는 동작-모양 시너지 네트워크를 제안한다. 제안하는 모델은 세 가지의 모듈: 동작 모듈, 모양 모듈, 동작-모양 융합 모듈로 구성되어 있다. 동작 모듈에서는 질문과 행동 정보를 융합한 cross-modal representation 을 생성하며, 모양 모듈에서는 주어진 비디오의 모양 측면에 집중하여 표상을 생성한다. 최종적으로 동작-모양 융합 모듈에서 인코딩된 두 정보가 질문의 내용을 기반으로 융합된다. 실험 결과, 제안하는 모델은 대규모 비디오 질의 응답 데이터셋인 TGIF-QA 와 MSVD-QA 에 대해 최첨단의 성능을 보였다. 본 논문에서는 또한 제안하는 모델의 정성적 평가 결과에 대해서도 보여준다.제 1 장 서 론 1 제 1 절 연구의 배경 1 제 2 절 연구의 내용 2 제 2 장 배경 연구 5 제 1 절 시각 정보 기반 질의 응답 모델들 5 제 2 절 행동 분류 모델들 5 제 3 절 어텐션 메커니즘 6 제 3 장 동작-모양 시너지 네트워크 7 제 1 절 시각 및 언어 표상 7 제 2 절 동작 및 모양 모듈 9 제 3 절 동작-모양 융합 모듈 10 제 4 절 정답 추론 및 목적 함수 13 제 4 장 실험 및 결과 14 제 1 절 학습 데이터 14 제 2 절 학습 조건 15 제 3 절 최첨단 접근 방식과의 비교 15 제 4 절 모듈 별 기여도 평가 17 제 5 절 정성적 평가 19 제 5 장 결론 및 제언 21 참고문헌 22 Abstract 27석

Video question answering supported by a multi-task learning objective

Video Question Answering (VideoQA) concerns the realization of models able to analyze a video, and produce a meaningful answer to visual content-related questions. To encode the given question, word embedding techniques are used to compute a representation of the tokens suitable for neural networks. Yet almost all the works in the literature use the same technique, although recent advancements in NLP brought better solutions. This lack of analysis is a major shortcoming. To address it, in this paper we present a twofold contribution about this inquiry and its relation with question encoding. First of all, we integrate four of the most popular word embedding techniques in three recent VideoQA architectures, and investigate how they influence the performance on two public datasets: EgoVQA and PororoQA. Thanks to the learning process, we show that embeddings carry question type-dependent characteristics. Secondly, to leverage this result, we propose a simple yet effective multi-task learning protocol which uses an auxiliary task defined on the question types. By using the proposed learning strategy, significant improvements are observed in most of the combinations of network architecture and embedding under analysis

Reasoning with Heterogeneous Graph Alignment for Video Question Answering

The dominant video question answering methods are based on fine-grained representation or model-specific attention mechanism. They usually process video and question separately, then feed the representations of different modalities into following late fusion networks. Although these methods use information of one modality to boost the other, they neglect to integrate correlations of both inter- and intra-modality in an uniform module. We propose a deep heterogeneous graph alignment network over the video shots and question words. Furthermore, we explore the network architecture from four steps: representation, fusion, alignment, and reasoning. Within our network, the inter- and intra-modality information can be aligned and interacted simultaneously over the heterogeneous graph and used for cross-modal reasoning. We evaluate our method on three benchmark datasets and conduct extensive ablation study to the effectiveness of the network architecture. Experiments show the network to be superior in quality