Reproducibility of experiments in recommender systems evaluation

© IFIP International Federation for Information Processing 2018 Published by Springer International Publishing AG 2018. All Rights Reserved. Recommender systems evaluation is usually based on predictive accuracy metrics with better scores meaning recommendations of higher quality. However, the comparison of results is becoming increasingly difficult, since there are different recommendation frameworks and different settings in the design and implementation of the experiments. Furthermore, there might be minor differences on algorithm implementation among the different frameworks. In this paper, we compare well known recommendation algorithms, using the same dataset, metrics and overall settings, the results of which point to result differences across frameworks with the exact same settings. Hence, we propose the use of standards that should be followed as guidelines to ensure the replication of experiments and the reproducibility of the results

How to Perform Reproducible Experiments in the ELLIOT Recommendation Framework: Data Processing, Model Selection, and Performance Evaluation

Recommender Systems have shown to be an efective way to alleviate the over-choice problem and provide accurate and tailored recommendations. However, the impressive number of proposed recommendation algorithms, splitting strategies, evaluation protocols, metrics, and tasks, has made rigorous experimental evaluation particularly challenging. ELLIOT is a comprehensive recommendation framework that aims to run and reproduce an entire experimental pipeline by processing a simple confguration fle. The framework loads, flters, and splits the data considering a vast set of strategies. Then, it optimizes hyperparameters for several recommendation algorithms, selects the best models, compares them with the baselines, computes metrics spanning from accuracy to beyond-accuracy, bias, and fairness, and conducts statistical analysis. The aim is to provide researchers a tool to ease all the experimental evaluation phases (and make them reproducible), from data reading to results collection. ELLIOT is freely available on GitHub at https://github.com/sisinflab/ellio

An explanation-based approach for experiment reproducibility in recommender systems

© 2019, Springer-Verlag London Ltd., part of Springer Nature. The offline evaluation of recommender systems is typically based on accuracy metrics such as the Mean Absolute Error and the Root Mean Squared Error for error rating prediction and Precision and Recall for measuring the quality of the top-N recommendations. However, it is difficult to reproduce the results since there are various libraries that can be used for running experiments and also within the same library there are many different settings that if not taken into consideration when replicating the results might vary. In this paper, we show that within the use of the same library an explanation-based approach can be used to assist in the reproducibility of experiments. Our proposed approach has been experimentally evaluated using a wide range of recommendation algorithms ranging from collaborative filtering to complicated fuzzy recommendation approaches that can solve the filter bubble problem, a real dataset, and the results show that it is both practical and effective

Identification of advanced data analysis in marketing: A systematic literature review

Aim/purpose – Marketing is an important area of activity for the vast majority of enterprises. Many of them try using marketing data analysis. Both the literature and the practice of many enterprises describe the use of advanced data analysis. However, interpretations of this concept differ. The aim of this paper is to identify the interpretation of advanced data analysis in marketing, in support of decision-making processes applied in the retail trading sector. Design/methodology/approach – The study was conducted using a systematic literature review, suggested by B. Kitchenham (2004), extended by C. Wohlin & R. Prikladniki (2013). This method was modified and expanded through the division of the whole study into two phases. Each phase is intended to facilitate obtaining answers to different important research questions. The first phase constitutes an exploratory study, whose results allow the detailed analysis of the literature in the second phase of the study. Findings – The results of this study of the relevant literature indicate that scholarly pub-lications do not use the phrase ‘advanced data analysis’, and its context is described with the term ‘data analysis’. Another term used broadly within the sphere of data analysis is ‘big data’. The concept of ‘data analysis’ in marketing is focused around the term ‘big data analytics’ and terms linked to the word ‘customer’, such as ‘customer-centric’, ‘customer engagement’, ‘customer experience’, ‘customer targeting service’, and ‘customers classification’. The study of the literature undertaken indicates that marketing employs data analysis in such areas as customer needs identification and market segmentation. Research implications/limitations – The study of the literature review was carried out using selected four databases containing publications, i.e. Web of Science, IEEE, Springer and ACM for the period 2008 to 2018. The research described in the article can be continued in two ways. First, by analysing the literature presented in this paper on advanced data analysis in marketing using the method called snowball sampling. Secondly, the results obtained from the first stage of the study can be used to conduct the study with other databases. Originality/value/contribution – The main contribution of this work is the proposal of modifying the systematic literature review method, which was expanded through the introduction of two phases. This division of two stages is important for conducting studies of literature when there are no clear, established definitions for the concepts being employed. The result of the study is also a set of ordered terms and their meanings that clearly define advanced data analysis in marketing

Comparative Recommender System Evaluation: Benchmarking Recommendation Frameworks

ABSTRACT Recommender systems research is often based on comparisons of predictive accuracy: the better the evaluation scores, the better the recommender. However, it is difficult to compare results from different recommender systems due to the many options in design and implementation of an evaluation strategy. Additionally, algorithmic implementations can diverge from the standard formulation due to manual tuning and modifications that work better in some situations. In this work we compare common recommendation algorithms as implemented in three popular recommendation frameworks. To provide a fair comparison, we have complete control of the evaluation dimensions being benchmarked: dataset, data splitting, evaluation strategies, and metrics. We also include results using the internal evaluation mechanisms of these frameworks. Our analysis points to large differences in recommendation accuracy across frameworks and strategies, i.e. the same baselines may perform orders of magnitude better or worse across frameworks. Our results show the necessity of clear guidelines when reporting evaluation of recommender systems to ensure reproducibility and comparison of results

“WARES”, a Web Analytics Recommender System

Il est difficile d'imaginer des entreprises modernes sans analyse, c'est une tendance dans les entreprises modernes, même les petites entreprises et les entrepreneurs individuels commencent à utiliser des outils d'analyse d'une manière ou d'une autre pour leur entreprise. Pas étonnant qu'il existe un grand nombre d'outils différents pour les différents domaines, ils varient dans le but de simples statistiques d'amis et de visites pour votre page Facebook à grands et sophistiqués dans le cas des systèmes conçus pour les grandes entreprises, ils pourraient être shareware ou payés. Parfois, vous devez passer une formation spéciale, être un spécialiste certifiés, ou même avoir un diplôme afin d'être en mesure d'utiliser l'outil d'analyse. D'autres outils offrent une interface d’utilisateur simple, avec des tableaux de bord, pour satisfaire leur compréhension d’information pour tous ceux qui les ont vus pour la première fois. Ce travail sera consacré aux outils d'analyse Web. Quoi qu'il en soit pour tous ceux qui pensent à utiliser l'analyse pour ses propres besoins se pose une question: "quel outil doit je utiliser, qui convient à mes besoins, et comment payer moins et obtenir un gain maximum". Dans ce travail je vais essayer de donner une réponse sur cette question en proposant le système de recommandation pour les outils analytiques web –WARES, qui aideront l'utilisateur avec cette tâche "simple". Le système WARES utilise l'approche hybride, mais surtout, utilise des techniques basées sur le contenu pour faire des suggestions. Le système utilise certains ratings initiaux faites par utilisateur, comme entrée, pour résoudre le problème du “démarrage à froid”, offrant la meilleure solution possible en fonction des besoins des utilisateurs. Le besoin de consultations coûteuses avec des experts ou de passer beaucoup d'heures sur Internet, en essayant de trouver le bon outil. Le système lui–même devrait effectuer une recherche en ligne en utilisant certaines données préalablement mises en cache dans la base de données hors ligne, représentée comme une ontologie d'outils analytiques web existants extraits lors de la recherche en ligne précédente.It is hard to imagine modern business without analytics; it is a trend in modern business, even small companies and individual entrepreneurs start using analytics tools, in one way or another, for their business. Not surprising that there exist many different tools for different domains, they vary in purpose from simple friends and visits statistic for your Facebook page, to big and sophisticated systems designed for the big corporations, they could be free or paid. Sometimes you need to pass special training, be a certified specialist, or even have a degree to be able to use analytics tool, other tools offers simple user interface with dashboards for easy understanding and availability for everyone who saw them for the first time. Anyway, for everyone who is thinking about using analytics for his/her own needs stands a question: “what tool should I use, which one suits my needs and how to pay less and get maximum gain”. In this work, I will try to give an answer to this question by proposing a recommender tool, which will help the user with this “simple task”. This paper is devoted to the creation of WARES, as reduction from Web Analytics REcommender System. Proposed recommender system uses hybrid approach, but mostly, utilize content–based techniques for making suggestions, while using some user’s ratings as an input for “cold start” search. System produces recommendations depending on user’s needs, also allowing quick adjustments in selection without need of expensive consultations with experts or spending lots of hours for Internet search, trying to find out the right tool. The system itself should perform as an online search using some pre–cached data in offline database, represented as an ontology of existing web analytics tools, extracted during the previous online search

AI Watch: Assessing Technology Readiness Levels for Artificial Intelligence

Artificial Intelligence (AI) offers the potential to transform our lives in radical ways. However, the main unanswered questions about this foreseen transformation are when and how this is going to happen. Not only do we lack the tools to determine what achievements will be attained in the near future, but we even underestimate what various technologies in AI are capable of today. Many so-called breakthroughs in AI are simply associated with highly-cited research papers or good performance on some particular benchmarks. Certainly, the translation from papers and benchmark performance to products is faster in AI than in other non-digital sectors. However, it is still the case that research breakthroughs do not directly translate to a technology that is ready to use in real-world environments. This document describes an exemplar-based methodology to categorise and assess several AI research and development technologies, by mapping them into Technology Readiness Levels (TRL) (e.g., maturity and availability levels). We first interpret the nine TRLs in the context of AI and identify different categories in AI to which they can be assigned. We then introduce new bidimensional plots, called readiness-vs-generality charts, where we see that higher TRLs are achievable for low-generality technologies focusing on narrow or specific abilities, while low TRLs are still out of reach for more general capabilities. We include numerous examples of AI technologies in a variety of fields, and show their readiness-vs-generality charts, serving as exemplars. Finally, we use the dynamics of several AI technology exemplars at different generality layers and moments of time to forecast some short-term and mid-term trends for AI.JRC.B.6-Digital Econom