Attend to You: Personalized Image Captioning with Context Sequence Memory Networks

We address personalization issues of image captioning, which have not been discussed yet in previous research. For a query image, we aim to generate a descriptive sentence, accounting for prior knowledge such as the user's active vocabularies in previous documents. As applications of personalized image captioning, we tackle two post automation tasks: hashtag prediction and post generation, on our newly collected Instagram dataset, consisting of 1.1M posts from 6.3K users. We propose a novel captioning model named Context Sequence Memory Network (CSMN). Its unique updates over previous memory network models include (i) exploiting memory as a repository for multiple types of context information, (ii) appending previously generated words into memory to capture long-term information without suffering from the vanishing gradient problem, and (iii) adopting CNN memory structure to jointly represent nearby ordered memory slots for better context understanding. With quantitative evaluation and user studies via Amazon Mechanical Turk, we show the effectiveness of the three novel features of CSMN and its performance enhancement for personalized image captioning over state-of-the-art captioning models.Comment: Accepted paper at CVPR 201

Why Neurons Have Thousands of Synapses, A Theory of Sequence Memory in Neocortex

Neocortical neurons have thousands of excitatory synapses. It is a mystery how neurons integrate the input from so many synapses and what kind of large-scale network behavior this enables. It has been previously proposed that non-linear properties of dendrites enable neurons to recognize multiple patterns. In this paper we extend this idea by showing that a neuron with several thousand synapses arranged along active dendrites can learn to accurately and robustly recognize hundreds of unique patterns of cellular activity, even in the presence of large amounts of noise and pattern variation. We then propose a neuron model where some of the patterns recognized by a neuron lead to action potentials and define the classic receptive field of the neuron, whereas the majority of the patterns recognized by a neuron act as predictions by slightly depolarizing the neuron without immediately generating an action potential. We then present a network model based on neurons with these properties and show that the network learns a robust model of time-based sequences. Given the similarity of excitatory neurons throughout the neocortex and the importance of sequence memory in inference and behavior, we propose that this form of sequence memory is a universal property of neocortical tissue. We further propose that cellular layers in the neocortex implement variations of the same sequence memory algorithm to achieve different aspects of inference and behavior. The neuron and network models we introduce are robust over a wide range of parameters as long as the network uses a sparse distributed code of cellular activations. The sequence capacity of the network scales linearly with the number of synapses on each neuron. Thus neurons need thousands of synapses to learn the many temporal patterns in sensory stimuli and motor sequences.Comment: Submitted for publicatio

Prefrontal Pathways Provide Top-Down Control of Memory for Sequences of Events

We remember our lives as sequences of events, but it is unclear how these memories are controlled during retrieval. In rats, the medial prefrontal cortex (mPFC) is positioned to influence sequence memory through extensive top-down inputs to regions heavily interconnected with the hippocampus, notably the nucleus reuniens of the thalamus (RE) and perirhinal cortex (PER). Here, we used an hM4Di synaptic-silencing approach to test our hypothesis that specific mPFC→RE and mPFC→PER projections regulate sequence memory retrieval. First, we found non-overlapping populations of mPFC cells project to RE and PER. Second, suppressing mPFC activity impaired sequence memory. Third, inhibiting mPFC→RE and mPFC→PER pathways effectively abolished sequence memory. Finally, a sequential lag analysis showed that the mPFC→RE pathway contributes to a working memory retrieval strategy, whereas the mPFC→PER pathway supports a temporal context memory retrieval strategy. These findings demonstrate that mPFC→RE and mPFC→PER pathways serve as top-down mechanisms that control distinct sequence memory retrieval strategies

Metric State Space Reinforcement Learning for a Vision-Capable Mobile Robot

We address the problem of autonomously learning controllers for vision-capable mobile robots. We extend McCallum's (1995) Nearest-Sequence Memory algorithm to allow for general metrics over state-action trajectories. We demonstrate the feasibility of our approach by successfully running our algorithm on a real mobile robot. The algorithm is novel and unique in that it (a) explores the environment and learns directly on a mobile robot without using a hand-made computer model as an intermediate step, (b) does not require manual discretization of the sensor input space, (c) works in piecewise continuous perceptual spaces, and (d) copes with partial observability. Together this allows learning from much less experience compared to previous methods.Comment: 14 pages, 8 figure

Effects of early musical experience on auditory sequence memory

The present study investigated a possible link between musical training and immediate memory span by testing experienced musicians and three groups of musically inexperienced subjects (gymnasts, Psychology 101 students, and video game players) on sequence memory and word familiarity tasks. By including skilled gymnasts who began studying their craft by age six, video game players, and Psychology 101 students as comparison groups, we attempted to control for some of the ways skilled musicians may differ from participants drawn from the general population in terms of gross motor skills and intensive experience in a highly skilled domain from an early age. We found that musicians displayed longer immediate memory spans than the comparison groups on auditory presentation conditions of the sequence reproductive span task. No differences were observed between the four groups on the visual conditions of the sequence memory task. These results provide additional converging support to recent findings showing that early musical experience and activity-dependent learning may selectively affect verbal rehearsal processes and the allocation of attention in sequence memory tasks

Flexible Bayesian Dynamic Modeling of Correlation and Covariance Matrices

Modeling correlation (and covariance) matrices can be challenging due to the positive-definiteness constraint and potential high-dimensionality. Our approach is to decompose the covariance matrix into the correlation and variance matrices and propose a novel Bayesian framework based on modeling the correlations as products of unit vectors. By specifying a wide range of distributions on a sphere (e.g. the squared-Dirichlet distribution), the proposed approach induces flexible prior distributions for covariance matrices (that go beyond the commonly used inverse-Wishart prior). For modeling real-life spatio-temporal processes with complex dependence structures, we extend our method to dynamic cases and introduce unit-vector Gaussian process priors in order to capture the evolution of correlation among components of a multivariate time series. To handle the intractability of the resulting posterior, we introduce the adaptive $\Delta$-Spherical Hamiltonian Monte Carlo. We demonstrate the validity and flexibility of our proposed framework in a simulation study of periodic processes and an analysis of rat's local field potential activity in a complex sequence memory task.Comment: 49 pages, 15 figure

The consolidation of implicit sequence memory in obstructive sleep apnea

Obstructive Sleep Apnea (OSA) Syndrome is a relatively frequent sleep disorder characterized by disrupted sleep patterns. It is a well-established fact that sleep has beneficial effect on memory consolidation by enhancing neural plasticity. Implicit sequence learning is a prominent component of skill learning. However, the formation and consolidation of this fundamental learning mechanism remains poorly understood in OSA. In the present study we examined the consolidation of different aspects of implicit sequence learning in patients with OSA. We used the Alternating Serial Reaction Time task to measure general skill learning and sequence-specific learning. There were two sessions: a learning phase and a testing phase, separated by a 10-hour offline period with sleep. Our data showed differences in offline changes of general skill learning between the OSA and control group. The control group demonstrated offline improvement from evening to morning, while the OSA group did not. In contrast, we did not observe differences between the groups in offline changes in sequence-specific learning. Our findings suggest that disrupted sleep in OSA differently affects neural circuits involved in the consolidation of sequence learning