Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation

We propose a hierarchically structured reinforcement learning approach to address the challenges of planning for generating coherent multi-sentence stories for the visual storytelling task. Within our framework, the task of generating a story given a sequence of images is divided across a two-level hierarchical decoder. The high-level decoder constructs a plan by generating a semantic concept (i.e., topic) for each image in sequence. The low-level decoder generates a sentence for each image using a semantic compositional network, which effectively grounds the sentence generation conditioned on the topic. The two decoders are jointly trained end-to-end using reinforcement learning. We evaluate our model on the visual storytelling (VIST) dataset. Empirical results from both automatic and human evaluations demonstrate that the proposed hierarchically structured reinforced training achieves significantly better performance compared to a strong flat deep reinforcement learning baseline.Comment: Accepted to AAAI 201

Plan-And-Write: Towards Better Automatic Storytelling

Automatic storytelling is challenging since it requires generating long, coherent natural language to describes a sensible sequence of events. Despite considerable efforts on automatic story generation in the past, prior work either is restricted in plot planning, or can only generate stories in a narrow domain. In this paper, we explore open-domain story generation that writes stories given a title (topic) as input. We propose a plan-and-write hierarchical generation framework that first plans a storyline, and then generates a story based on the storyline. We compare two planning strategies. The dynamic schema interweaves story planning and its surface realization in text, while the static schema plans out the entire storyline before generating stories. Experiments show that with explicit storyline planning, the generated stories are more diverse, coherent, and on topic than those generated without creating a full plan, according to both automatic and human evaluations.Comment: Accepted by AAAI 201

Implementation of an evaluation platform for Alzheimer patients based on Egocentric Sequences Description

Treballs Finals de Grau d'Enginyeria Informàtica, Facultat de Matemàtiques, Universitat de Barcelona, Any: 2017, Director: Marc Bolaños Solà i Petia RadevaNumerous international population-based studies have been conducted to document the frequency of MCI, estimating its prevalence to be between 15% and 20% in persons 60 years and older, making it a common condition encountered by clinicians[17]. This number is predicted to increase to 75.6 million in 2030, and 135.5 million in 2050[14], leading to deep social and economical costs. The most common dementia type is Alzheimer (between 50% and 70% of the cases) and its early detection can greatly affect the recovery of the patient. That is why it is important to have tools for its early diagnosis and follow-up. Serious games, with an increasing popularity, are a good way to MCI as an early stage of Alzheimer and improve the memory capacities of the patients. These video games focusing on different stages of the illness can help doctors to document and check the progress of the illness. This work aims on developing a software for patients with MCI, which is the lack of memory and other human characteristics like reasoning and language. These individuals usually progress to Alzheimer disease, but if detected early, in some cases they can also remain stable or even recover with time. To help them to exercise their memory, we propose that our program uses their own experiences caught by a wearable camera. This software will provide images of the patient’s life in order to do exercises that will evaluate their ability to remember and reason about the scenes they visualize. With these tests, the doctors will be able to see the evolution of the patients, and help them to diagnose and track the illness. In this project, we additionally work on an application, for the first time, of Deep Neural Networks for the automatic generation of descriptions of egocentric sequences. This will serve as the first step to automate the evaluation process by automatically comparing the subjective descriptions provided by the patients to the objective ones generated by our system

Contextualizing Artificial Intelligence: The History, Values, and Epistemology of Technology in the Philosophy of Science

Artificial intelligence (AI) and other advanced technologies pose new questions for philosophers of science regarding epistemology, science and values, and the history of science. I will address these issues across three essays in this dissertation. The first essay concerns epistemic problems that emerge with existing accounts of scientific explanation when they are applied to deep neural networks (DNNs). Causal explanations in particular, which appear at first to be well suited to the task of explaining DNNs, fail to provide any such explanation. The second essay will explore bias in systems of automated decision-making, and the role of various conceptions of objectivity in either reinforcing or mitigating bias. I focus on conceptions of objectivity common in social epistemology and the feminist philosophy of science. The third essay probes the history of the development of 20th century telecommunications technology and the relationship between formal and informal systems of scientific knowledge production. Inquiring into the role that early phone and computer hackers played in the scientific developments of those technologies, I untangle the messy web of relationships between various groups that had a lasting impact on this history while engaging in a conceptual analysis of hacking and hackers