Detecting and Grounding Important Characters in Visual Stories

Characters are essential to the plot of any story. Establishing the characters before writing a story can improve the clarity of the plot and the overall flow of the narrative. However, previous work on visual storytelling tends to focus on detecting objects in images and discovering relationships between them. In this approach, characters are not distinguished from other objects when they are fed into the generation pipeline. The result is a coherent sequence of events rather than a character-centric story. In order to address this limitation, we introduce the VIST-Character dataset, which provides rich character-centric annotations, including visual and textual co-reference chains and importance ratings for characters. Based on this dataset, we propose two new tasks: important character detection and character grounding in visual stories. For both tasks, we develop simple, unsupervised models based on distributional similarity and pre-trained vision-and-language models. Our new dataset, together with these models, can serve as the foundation for subsequent work on analysing and generating stories from a character-centric perspective.Comment: AAAI 202

Visual Storytelling with Question-Answer Plans

Visual storytelling aims to generate compelling narratives from image sequences. Existing models often focus on enhancing the representation of the image sequence, e.g., with external knowledge sources or advanced graph structures. Despite recent progress, the stories are often repetitive, illogical, and lacking in detail. To mitigate these issues, we present a novel framework which integrates visual representations with pretrained language models and planning. Our model translates the image sequence into a visual prefix, a sequence of continuous embeddings which language models can interpret. It also leverages a sequence of question-answer pairs as a blueprint plan for selecting salient visual concepts and determining how they should be assembled into a narrative. Automatic and human evaluation on the VIST benchmark (Huang et al., 2016) demonstrates that blueprint-based models generate stories that are more coherent, interesting, and natural compared to competitive baselines and state-of-the-art systems.Comment: EMNLP 2023 Finding

DNA-Packing Portal and Capsid-Associated Tegument Complexes in the Tumor Herpesvirus KSHV.

Assembly of Kaposi's sarcoma-associated herpesvirus (KSHV) begins at a bacteriophage-like portal complex that nucleates formation of an icosahedral capsid with capsid-associated tegument complexes (CATCs) and facilitates translocation of an ∼150-kb dsDNA genome, followed by acquisition of a pleomorphic tegument and envelope. Because of deviation from icosahedral symmetry, KSHV portal and tegument structures have largely been obscured in previous studies. Using symmetry-relaxed cryo-EM, we determined the in situ structure of the KSHV portal and its interactions with surrounding capsid proteins, CATCs, and the terminal end of KSHV's dsDNA genome. Our atomic models of the portal and capsid/CATC, together with visualization of CATCs' variable occupancy and alternate orientation of CATC-interacting vertex triplexes, suggest a mechanism whereby the portal orchestrates procapsid formation and asymmetric long-range determination of CATC attachment during DNA packaging prior to pleomorphic tegumentation/envelopment. Structure-based mutageneses confirm that a triplex deep binding groove for CATCs is a hotspot that holds promise for antiviral development