A Dataset and Baselines for Visual Question Answering on Art

Answering questions related to art pieces (paintings) is a difficult task, as it implies the understanding of not only the visual information that is shown in the picture, but also the contextual knowledge that is acquired through the study of the history of art. In this work, we introduce our first attempt towards building a new dataset, coined AQUA (Art QUestion Answering). The question-answer (QA) pairs are automatically generated using state-of-the-art question generation methods based on paintings and comments provided in an existing art understanding dataset. The QA pairs are cleansed by crowdsourcing workers with respect to their grammatical correctness, answerability, and answers' correctness. Our dataset inherently consists of visual (painting-based) and knowledge (comment-based) questions. We also present a two-branch model as baseline, where the visual and knowledge questions are handled independently. We extensively compare our baseline model against the state-of-the-art models for question answering, and we provide a comprehensive study about the challenges and potential future directions for visual question answering on art

Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication

Visual storytelling aims to generate a narrative paragraph from a sequence of images automatically. Existing approaches construct text description independently for each image and roughly concatenate them as a story, which leads to the problem of generating semantically incoherent content. In this paper, we propose a new way for visual storytelling by introducing a topic description task to detect the global semantic context of an image stream. A story is then constructed with the guidance of the topic description. In order to combine the two generation tasks, we propose a multi-agent communication framework that regards the topic description generator and the story generator as two agents and learn them simultaneously via iterative updating mechanism. We validate our approach on VIST dataset, where quantitative results, ablations, and human evaluation demonstrate our method's good ability in generating stories with higher quality compared to state-of-the-art methods.Comment: Accepted to COLING 202

PolyAQG Framework: Auto-generating assessment questions

Designing and setting assessment questions for examinations is always a necessary task for educators. In this article, we identify the research gaps in (semi-) automatically generating questions by evaluating all the available approaches developed thus far. We then propose a framework that puts together previous approaches and suggests ways to fill in their gaps. One hundred and thirteen pieces of literature relevant to question generation approaches have been reviewed and compared. For each of the approaches, the uniqueness of the techniques is explained. The PolyAQG Framework is presented with an explanation of how it would contribute to the solution of the problem, by improving the variety of the questions, increasing the total number of possible choices of question selections, as well as providing a better quality of questions. Apart from the framework, another novelty in this work is the innovative way a domain ontology is used to generate a wider variety of questions

PolyAQG Framework: Auto-generating assessment questions

Designing and setting assessment questions for examinations is always a necessary task for educators. In this article, we identify the research gaps in (semi-) automatically generating questions by evaluating all the available approaches developed thus far. We then propose a framework that puts together previous approaches and suggests ways to fill in their gaps. One hundred and thirteen pieces of literature relevant to question generation approaches have been reviewed and compared. For each of the approaches, the uniqueness of the techniques is explained. The PolyAQG Framework is presented with an explanation of how it would contribute to the solution of the problem, by improving the variety of the questions, increasing the total number of possible choices of question selections, as well as providing a better quality of questions. Apart from the framework, another novelty in this work is the innovative way a domain ontology is used to generate a wider variety of questions