Temporal album

Transient synchronization has been used as a mechanism of recognizing auditory patterns using integrate-and-fire neural networks. We first extend the mechanism to vision tasks and investigate the role of spike dependent learning. We show that such a temporal Hebbian learning rule significantly improves accuracy of detection. We demonstrate how multiple patterns can be identified by a single pattern selective neuron and how a temporal album can be constructed. This principle may lead to multidimensional memories, where the capacity per neuron is considerably increased with accurate detection of spike synchronization

Hierarchical Photo-Scene Encoder for Album Storytelling

In this paper, we propose a novel model with a hierarchical photo-scene encoder and a reconstructor for the task of album storytelling. The photo-scene encoder contains two sub-encoders, namely the photo and scene encoders, which are stacked together and behave hierarchically to fully exploit the structure information of the photos within an album. Specifically, the photo encoder generates semantic representation for each photo while exploiting temporal relationships among them. The scene encoder, relying on the obtained photo representations, is responsible for detecting the scene changes and generating scene representations. Subsequently, the decoder dynamically and attentively summarizes the encoded photo and scene representations to generate a sequence of album representations, based on which a story consisting of multiple coherent sentences is generated. In order to fully extract the useful semantic information from an album, a reconstructor is employed to reproduce the summarized album representations based on the hidden states of the decoder. The proposed model can be trained in an end-to-end manner, which results in an improved performance over the state-of-the-arts on the public visual storytelling (VIST) dataset. Ablation studies further demonstrate the effectiveness of the proposed hierarchical photo-scene encoder and reconstructor.Comment: 8 pages, 4 figure

Recognizing and Curating Photo Albums via Event-Specific Image Importance

Automatic organization of personal photos is a problem with many real world ap- plications, and can be divided into two main tasks: recognizing the event type of the photo collection, and selecting interesting images from the collection. In this paper, we attempt to simultaneously solve both tasks: album-wise event recognition and image- wise importance prediction. We collected an album dataset with both event type labels and image importance labels, refined from an existing CUFED dataset. We propose a hybrid system consisting of three parts: A siamese network-based event-specific image importance prediction, a Convolutional Neural Network (CNN) that recognizes the event type, and a Long Short-Term Memory (LSTM)-based sequence level event recognizer. We propose an iterative updating procedure for event type and image importance score prediction. We experimentally verified that image importance score prediction and event type recognition can each help the performance of the other.Comment: Accepted as oral in BMVC 201

Hierarchically-Attentive RNN for Album Summarization and Storytelling

We address the problem of end-to-end visual storytelling. Given a photo album, our model first selects the most representative (summary) photos, and then composes a natural language story for the album. For this task, we make use of the Visual Storytelling dataset and a model composed of three hierarchically-attentive Recurrent Neural Nets (RNNs) to: encode the album photos, select representative (summary) photos, and compose the story. Automatic and human evaluations show our model achieves better performance on selection, generation, and retrieval than baselines.Comment: To appear at EMNLP-2017 (7 pages

Deixis Analysis in the Song Lyrics of Ed Sheeran’s `Divide` Album

This research aims to analyze five types of deixis by using theory of Alan Cruse (2000), interpret the reference meaning of deixis and find out the most dominant type of deixis that found in the song lyrics of Ed Sheeran’s album. The researcher elects Divide album as the object of the analysis because it is one of the best-selling album in the world. it consists of such deictic words that has reference meanings. Therefore, the song lyrics can be analyzed using pragmatic approach, specifically about deixis. This study was conducted by using descriptive qualitative method. The data which is used is six songs of Ed Sheeran’s album and then they are classified into the types of deixis based on their own criteria. The findings showed that the types of deixis like person deixis, spatial deixis, temporal deixis, social deixis and discourse deixis are used in the song lyrics of Ed Sheeran’s Album. Based on discussion and finding can be concluded that all deixis are found in all songs, personal deixis is the most being found (46 Deictic words or 28%) the word “I. Me, My, You and Your dominate all songs. in the second position Temporal Deixis is the most being found (43 deictic words or 26%) the word now dominate in all songs. the third position is Spatial deixis (41 Deictic words or 25% ) in the fourth position is Discourse Deixis (20 deictic words or 12%) and in the last position is Social Deixis (15 Deictic words or 9%)

Recurrent Poisson Factorization for Temporal Recommendation

Poisson factorization is a probabilistic model of users and items for recommendation systems, where the so-called implicit consumer data is modeled by a factorized Poisson distribution. There are many variants of Poisson factorization methods who show state-of-the-art performance on real-world recommendation tasks. However, most of them do not explicitly take into account the temporal behavior and the recurrent activities of users which is essential to recommend the right item to the right user at the right time. In this paper, we introduce Recurrent Poisson Factorization (RPF) framework that generalizes the classical PF methods by utilizing a Poisson process for modeling the implicit feedback. RPF treats time as a natural constituent of the model and brings to the table a rich family of time-sensitive factorization models. To elaborate, we instantiate several variants of RPF who are capable of handling dynamic user preferences and item specification (DRPF), modeling the social-aspect of product adoption (SRPF), and capturing the consumption heterogeneity among users and items (HRPF). We also develop a variational algorithm for approximate posterior inference that scales up to massive data sets. Furthermore, we demonstrate RPF's superior performance over many state-of-the-art methods on synthetic dataset, and large scale real-world datasets on music streaming logs, and user-item interactions in M-Commerce platforms.Comment: Submitted to KDD 2017 | Halifax, Nova Scotia - Canada - sigkdd, Codes are available at https://github.com/AHosseini/RP

Hop and HipHop : Multitier Web Orchestration

Rich applications merge classical computing, client-server concurrency, web-based interfaces, and the complex time- and event-based reactive programming found in embedded systems. To handle them, we extend the Hop web programming platform by HipHop, a domain-specific language dedicated to event-based process orchestration. Borrowing the synchronous reactive model of Esterel, HipHop is based on synchronous concurrency and preemption primitives that are known to be key components for the modular design of complex reactive behaviors. HipHop departs from Esterel by its ability to handle the dynamicity of Web applications, thanks to the reflexivity of Hop. Using a music player example, we show how to modularly build a non-trivial Hop application using HipHop orchestration code.Comment: International Conference on Distributed Computing and Internet Technology (2014

Large-Scale User Modeling with Recurrent Neural Networks for Music Discovery on Multiple Time Scales

The amount of content on online music streaming platforms is immense, and most users only access a tiny fraction of this content. Recommender systems are the application of choice to open up the collection to these users. Collaborative filtering has the disadvantage that it relies on explicit ratings, which are often unavailable, and generally disregards the temporal nature of music consumption. On the other hand, item co-occurrence algorithms, such as the recently introduced word2vec-based recommenders, are typically left without an effective user representation. In this paper, we present a new approach to model users through recurrent neural networks by sequentially processing consumed items, represented by any type of embeddings and other context features. This way we obtain semantically rich user representations, which capture a user's musical taste over time. Our experimental analysis on large-scale user data shows that our model can be used to predict future songs a user will likely listen to, both in the short and long term.Comment: Author pre-print version, 20 pages, 6 figures, 4 table