An Empirical Evaluation of Visual Question Answering for Novel Objects

We study the problem of answering questions about images in the harder setting, where the test questions and corresponding images contain novel objects, which were not queried about in the training data. Such setting is inevitable in real world-owing to the heavy tailed distribution of the visual categories, there would be some objects which would not be annotated in the train set. We show that the performance of two popular existing methods drop significantly (up to 28%) when evaluated on novel objects cf. known objects. We propose methods which use large existing external corpora of (i) unlabeled text, i.e. books, and (ii) images tagged with classes, to achieve novel object based visual question answering. We do systematic empirical studies, for both an oracle case where the novel objects are known textually, as well as a fully automatic case without any explicit knowledge of the novel objects, but with the minimal assumption that the novel objects are semantically related to the existing objects in training. The proposed methods for novel object based visual question answering are modular and can potentially be used with many visual question answering architectures. We show consistent improvements with the two popular architectures and give qualitative analysis of the cases where the model does well and of those where it fails to bring improvements.Comment: 11 pages, 4 figures, accepted in CVPR 2017 (poster

질의응답 시스템을 위한 텍스트 랭킹 심층 신경망

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2020. 8. 정교민.The question answering (QA) system has attracted huge interests due to its applicability in real-world applications. This dissertation proposes novel ranking algorithms for the QA system based on deep neural networks. We first tackle the long-text QA that requires the model to understand the excessively large sequence of text inputs. To solve this problem, we propose a hierarchical recurrent dual encoder that encodes texts from word-level to paragraph-level. We further propose a latent topic clustering method that utilizes semantic information in the target corpus, and thus it increases the performance of the QA system. Secondly, we investigate the short-text QA, where the information in text pairs are limited. To overcome the insufficiency, we combine a pretrained language model and an enhanced latent clustering method to the QA model. This novel architecture enables the model to utilizes additional information, resulting in achieving state-of-the-art performance for the standard answer-selection tasks (i.e., WikiQA, TREC-QA). Finally, we investigate detecting supporting sentences for complex QA system. As opposed to the previous studies, the model needs to understand the relationship between sentences to answer the question. Inspired by the hierarchical nature of the text, we propose a graph neural network-based model that iteratively propagates necessary information between text nodes and achieve the best performance among existing methods.본 학위 논문은 딥 뉴럴 네트워크 기반 질의응답 시스템에 관한 모델을 제안한다. 먼저 긴 문장에 대한 질의응답을 하기 위해서 계층 구조의 재귀신경망 모델을 제안하였다. 이를 통해 모델이 주어진 문장을 짧은 시퀀스 단위로 효율적으로 다룰 수 있게 하여 큰 성능 향상을 얻었다. 또한 학습 과정에서 데이터 안에 내포된 토픽을 자동 분류하는 모델을 제안하고, 이를 기존 질의응답 모델에 병합하여 추가 성능 개선을 이루었다. 이어지는 연구로 짧은 문장에 대한 질의응답 모델을 제안하였다. 문장의 길이가 짧아질수록 문장 안에서 얻을 수 있는 정보의 양도 줄어들게 된다. 우리는 이러한 문제를 해결하기 위해, 사전 학습된 언어 모델과 새로운 토픽 클러스터링 기법을 적용하였다. 제안한 모델은 종래 짧은 문장 질의응답 연구 중 가장 좋은 성능을 획득하였다. 마지막으로 여러 문장 사이의 관계를 이용하여 답변을 찾아야 하는 질의응답 연구를 진행하였다. 우리는 문서 내 각 문장을 그래프로 도식화한 후 이를 학습할 수 있는 그래프 뉴럴 네트워크를 제안하였다. 제안한 모델은 각 문장의 관계성을 성공적으로 계산하였고, 이를 통해 복잡도가 높은 질의응답 시스템에서 기존에 제안된 모델들과 비교하여 가장 좋은 성능을 획득하였다.1 Introduction 1 2 Background 8 2.1 Textual Data Representation 8 2.2 Encoding Sequential Information in Text 12 3 Question-Answer Pair Ranking for Long Text 16 3.1 Related Work 18 3.2 Method 19 3.2.1 Baseline Approach 19 3.2.2 Proposed Approaches (HRDE+LTC) 22 3.3 Experimental Setup and Dataset 26 3.3.1 Dataset 26 3.3.2 Consumer Product Question Answering Corpus 30 3.3.3 Implementation Details 32 3.4 Empirical Results 34 3.4.1 Comparison with other methods 35 3.4.2 Degradation Comparison for Longer Texts 37 3.4.3 Effects of the LTC Numbers 38 3.4.4 Comprehensive Analysis of LTC 38 3.5 Further Investigation on Ranking Lengthy Document 40 3.5.1 Problem and Dataset 41 3.5.2 Methods 45 3.5.3 Experimental Results 51 3.6 Conclusion 55 4 Answer-Selection for Short Sentence 56 4.1 Related Work 57 4.2 Method 59 4.2.1 Baseline approach 59 4.2.2 Proposed Approaches (Comp-Clip+LM+LC+TL) 62 4.3 Experimental Setup and Dataset 66 4.3.1 Dataset 66 4.3.2 Implementation Details 68 4.4 Empirical Results 69 4.4.1 Comparison with Other Methods 69 4.4.2 Impact of Latent Clustering 72 4.5 Conclusion 72 5 Supporting Sentence Detection for Question Answering 73 5.1 Related Work 75 5.2 Method 76 5.2.1 Baseline approaches 76 5.2.2 Proposed Approach (Propagate-Selector) 78 5.3 Experimental Setup and Dataset 82 5.3.1 Dataset 82 5.3.2 Implementation Details 83 5.4 Empirical Results 85 5.4.1 Comparisons with Other Methods 85 5.4.2 Hop Analysis 86 5.4.3 Impact of Various Graph Topologies 88 5.4.4 Impact of Node Representation 91 5.5 Discussion 92 5.6 Conclusion 93 6 Conclusion 94Docto

Finding Answers from the Word of God: Domain Adaptation for Neural Networks in Biblical Question Answering

Question answering (QA) has significantly benefitted from deep learning techniques in recent years. However, domain-specific QA remains a challenge due to the significant amount of data required to train a neural network. This paper studies the answer sentence selection task in the Bible domain and answer questions by selecting relevant verses from the Bible. For this purpose, we create a new dataset BibleQA based on bible trivia questions and propose three neural network models for our task. We pre-train our models on a large-scale QA dataset, SQuAD, and investigate the effect of transferring weights on model accuracy. Furthermore, we also measure the model accuracies with different answer context lengths and different Bible translations. We affirm that transfer learning has a noticeable improvement in the model accuracy. We achieve relatively good results with shorter context lengths, whereas longer context lengths decreased model accuracy. We also find that using a more modern Bible translation in the dataset has a positive effect on the task.Comment: The paper has been accepted at IJCNN 201