Evaluation of recommender systems in streaming environments

Evaluation of recommender systems is typically done with finite datasets. This means that conventional evaluation methodologies are only applicable in offline experiments, where data and models are stationary. However, in real world systems, user feedback is continuously generated, at unpredictable rates. Given this setting, one important issue is how to evaluate algorithms in such a streaming data environment. In this paper we propose a prequential evaluation protocol for recommender systems, suitable for streaming data environments, but also applicable in stationary settings. Using this protocol we are able to monitor the evolution of algorithms' accuracy over time. Furthermore, we are able to perform reliable comparative assessments of algorithms by computing significance tests over a sliding window. We argue that besides being suitable for streaming data, prequential evaluation allows the detection of phenomena that would otherwise remain unnoticed in the evaluation of both offline and online recommender systems.Comment: Workshop on 'Recommender Systems Evaluation: Dimensions and Design' (REDD 2014), held in conjunction with RecSys 2014. October 10, 2014, Silicon Valley, United State

A study on contextual influences on automatic playlist continuation

Recommender systems still mainly base their reasoning on pairwise interactions or information on individual entities, like item attributes or ratings, without properly evaluating the multiple dimensions of the recommendation problem. However, in many cases, like in music, items are rarely consumed in isolation, thus users rather need a set of items, selected to work well together, serving a specific purpose, while having some cognitive properties as a whole, related to their perception of quality and satisfaction, under given circumstances. In this paper, we introduce the term of playlist concept in order to capture the implicit characteristics of joint music item selections, related to their context, scope and general perception by the users. Although playlist consumptions may be associated with contextual attributes, these may be of various types, differently influencing users' preferences, based on their character and emotional state, therefore differently reflected on their final selections. We highlight on the use of this term in HybA, our hybrid recommender system, to identify clusters of similar playlists able to capture inherit characteristics and semantic properties, not explicitly described in them. The experimental results presented, show that this conceptual clustering results in playlist continuations of improved quality, compared to using explicit contextual parameters, or the commonly used collaborative filtering technique.Peer ReviewedPostprint (published version

Sistema de recomendação Web usando agentes

O crescimento da Web trouxe vários problemas aos utilizadores. A grande quantidade de informação existente hoje em dia em alguns sítios Web torna a procura de informação útil muito difícil. Os objetivos dos proprietários dos sítios Web e dos utilizadores nem sempre coincidem. O conhecimento dos padrões de visitas dos utilizadores é crucial para que os proprietários possam transformar e adaptar o sítio Web. Este é o princípio do sítio Web adaptativo: o sítio Web adapta-se de forma a melhorar a experiência do utilizador. Alguns algoritmos foram propostos para adaptar um sítio da Web. Neste artigo, descrevemos uma proposta de um sistema de recomendação Web baseado em agentes que combina dois algoritmos: regras de associação e filtragem colaborativa. Ambos os algoritmos são incrementais e funcionam com dados binários. Os resultados mostram que, em algumas situações, a abordagem multiagente melhora a capacidade preditiva quando comparada com os agentes individuais.The growth of the Web has brought several problems for users. Today the vast amount of information on some web sites makes useful information finding very difficult. The objectives of the owners of the web sites and users do not always coincide. The knowledge of patterns of user visits is crucial to the owners to transform and adapt their web site. This is the adaptive website principle: the website adapts to improve the user experience. Some algorithms have been proposed to tailor a website. In this paper, we describe a proposal for a web recommendation system based on agents that combines two algorithms: association rules and collaborative filtering. Both algorithms are incremental and work with binary data. The results show that, in some situations, the multi-agent approach overcomes the predictive capacity of individual agents

LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond

Distributional semantics based on neural approaches is a cornerstone of Natural Language Processing, with surprising connections to human meaning representation as well. Recent Transformer-based Language Models have proven capable of producing contextual word representations that reliably convey sense-specific information, simply as a product of self-supervision. Prior work has shown that these contextual representations can be used to accurately represent large sense inventories as sense embeddings, to the extent that a distance-based solution to Word Sense Disambiguation (WSD) tasks outperforms models trained specifically for the task. Still, there remains much to understand on how to use these Neural Language Models (NLMs) to produce sense embeddings that can better harness each NLM's meaning representation abilities. In this work we introduce a more principled approach to leverage information from all layers of NLMs, informed by a probing analysis on 14 NLM variants. We also emphasize the versatility of these sense embeddings in contrast to task-specific models, applying them on several sense-related tasks, besides WSD, while demonstrating improved performance using our proposed approach over prior work focused on sense embeddings. Finally, we discuss unexpected findings regarding layer and model performance variations, and potential applications for downstream tasks.Comment: Accepted to Artificial Intelligence Journal (AIJ

A hybrid recommender system for improving automatic playlist continuation

Although widely used, the majority of current music recommender systems still focus on recommendations’ accuracy, userpreferences and isolated item characteristics, without evaluating other important factors, like the joint item selections and the recommendation moment. However, when it comes to playlist recommendations, additional dimensions, as well as the notion of user experience and perception, should be taken into account to improve recommendations’ quality. In this work, HybA, a hybrid recommender system for automatic playlist continuation, that combines Latent Dirichlet Allocation and Case-Based Reasoning, is proposed. This system aims to address “similar concepts” rather than similar users. More than generating a playlist based on user requirements, like automatic playlist generation methods, HybA identifies the semantic characteristics of a started playlist and reuses the most similar past ones, to recommend relevant playlist continuations. In addition, support to beyond accuracy dimensions, like increased coherence or diverse items’ discovery, is provided. To overcome the semantic gap between music descriptions and user preferences, identify playlist structures and capture songs’ similarity, a graph model is used. Experiments on real datasets have shown that the proposed algorithm is able to outperform other state of the art techniques, in terms of accuracy, while balancing between diversity and coherence.This work has been partially supported by the Catalan Agency for Management of University and Research Grants (AGAUR) (2017 SGR 574), by the European Regional Development Fund (ERDF), through the Incentive System to Research and Technological development, within the Portugal2020 Competitiveness and Internationalization Operational Program –COMPETE 2020– (POCI-01-0145-FEDER006961), and by the Portuguese Foundation for Science and Technology (FCT) (UID/EEA/50014/2013).Peer ReviewedPostprint (author's final draft

A Biomedical Entity Extraction Pipeline for Oncology Health Records in Portuguese

Textual health records of cancer patients are usually protracted and highly unstructured, making it very time-consuming for health professionals to get a complete overview of the patient's therapeutic course. As such limitations can lead to suboptimal and/or inefficient treatment procedures, healthcare providers would greatly benefit from a system that effectively summarizes the information of those records. With the advent of deep neural models, this objective has been partially attained for English clinical texts, however, the research community still lacks an effective solution for languages with limited resources. In this paper, we present the approach we developed to extract procedures, drugs, and diseases from oncology health records written in European Portuguese. This project was conducted in collaboration with the Portuguese Institute for Oncology which, besides holding over $10$ years of duly protected medical records, also provided oncologist expertise throughout the development of the project. Since there is no annotated corpus for biomedical entity extraction in Portuguese, we also present the strategy we followed in annotating the corpus for the development of the models. The final models, which combined a neural architecture with entity linking, achieved $F_1$ scores of $88.6$, $95.0$, and $55.8$ per cent in the mention extraction of procedures, drugs, and diseases, respectively

GTE-Cluster: A Temporal Search Interface for Implicit Temporal Queries

In this work, we aim to define the temporal intents of implicit tempora

Preference rules for label ranking: Mining patterns in multi-target relations

In this paper, we investigate two variants of association rules for preference data, Label Ranking Association Rules and Pairwise Association Rules. Label Ranking Association Rules (LRAR) are the equivalent of Class Association Rules (CAR) for the Label Ranking task. In CAR, the consequent is a single class, to which the example is expected to belong to. In LRAR, the consequent is a ranking of the labels. The generation of LRAR requires special support and confidence measures to assess the similarity of rankings. In this work, we carry out a sensitivity analysis of these similarity-based measures. We want to understand which datasets benefit more from such measures and which parameters have more influence in the accuracy of the model. Furthermore, we propose an alternative type of rules, the Pairwise Association Rules (PAR), which are defined as association rules with a set of pairwise preferences in the consequent. While PAR can be used both as descriptive and predictive models, they are essentially descriptive models. Experimental results show the potential of both approaches.This research has received funding from the ECSEL Joint Undertaking, the framework programme for research and innovation horizon 2020 (2014-2020) under grant agreement number 662189-MANTIS-2014-1, and by National Funds through the FCT — Fundação para a Ciência e a Tecnologia (Portuguese Foundation for Science and Technology) as part of project UID/EEA/50014/2013

A data warehouse to support web site automation

Background: \ud Due to the constant demand for new information and timely updates of services and content in order to satisfy the user’s needs, web site automation has emerged as a solution to automate several personalization and management activities of a web site. One goal of automation is the reduction of the editor’s effort and consequently of the costs for the owner. The other goal is that the site can more timely adapt to the behavior of the user, improving the browsing experience and helping the user in achieving his/her own goals. \ud \ud Methods: \ud A database to store rich web data is an essential component for web site automation. In this paper, we propose a data warehouse that is developed to be a repository of information to support different web site automation and monitoring activities. We implemented our data warehouse and used it as a repository of information in three different case studies related to the areas of e-commerce, e-learning, and e-news. \ud \ud Result: \ud The case studies showed that our data warehouse is appropriate for web site automation in different contexts. \ud \ud Conclusion: \ud In all cases, the use of the data warehouse was quite simple and with a good response time, mainly because of the simplicity of its structure.FCT - Science and Technology Foundation (SFRH/BD/22516/2005)project Site-O-Matic (POSC/EIA/58367/2004)São Paulo Research Foundation (FAPESP) (grants 2011/19850-9, 2012/13830-9