Replicable Evaluation of Recommender Systems

This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in RecSys '15 Proceedings of the 9th ACM Conference on Recommender Systems, http://dx.doi.org/10.1145/2792838.2792841.Recommender systems research is by and large based on comparisons of recommendation algorithms’ predictive accuracies: the better the evaluation metrics (higher accuracy scores or lower predictive errors), the better the recommendation algorithm. Comparing the evaluation results of two recommendation approaches is however a difficult process as there are very many factors to be considered in the implementation of an algorithm, its evaluation, and how datasets are processed and prepared. This tutorial shows how to present evaluation results in a clear and concise manner, while ensuring that the results are comparable, replicable and unbiased. These insights are not limited to recommender systems research alone, but are also valid for experiments with other types of personalized interactions and contextual information access.Supported in part by the Ministerio de Educación y Ciencia (TIN2013-47090-C3-2)

Diseño de una propuesta metodológica de educación ambiental para fortalecer las Comunidades Negras en Jamundí

El presente documento es el diseño de una propuesta metodológica de educación ambiental para fortalecer las Comunidades Negras asentadas en el Corregimiento de Potrerito, ubicado en Jamundí- Valle del Cauca, quienes cuentan con una gran riqueza ambiental y cultural, pero están perdiendo sus conocimientos tradicionales asociados a la biodiversidad. Tiene como objetivo principal: Diseñar de una propuesta metodológica de educación ambiental para fortalecer las Comunidades Negras en procesos bioculturales en el Corregimiento de Potrerito de Jamundí Valle del Cauca, a su vez, cuenta con 3 objetivos específicos: 1) Determinar de qué manera se realizará la caracterización para conocer cuál es el estado socioambiental del Corregimiento Potrerito de Jamundí Valle del Cauca, 2) Establecer cómo se van a conocer cuáles son los conocimientos tradicionales y activos bioculturales para incluirlos en la propuesta de educación ambiental que fortalezca el desarrollo sostenible de las comunidades negras del Corregimiento de Potrerito Jamundí-Valle, 3) Conocer cómo estructuraran los contenidos pedagógicos de la propuesta de educación ambiental que fortalecerá a las comunidades en sus conocimientos tradicionales asociados a los procesos bioculturales en Jamundí Valle del Cauca. En términos metodológicos se realizará con el método investigación- acción. Haciendo uso de estrategias como el diagnostico, las entrevistas, encuestas, talleres; y con instrumentos como diario de campo, fotografías, entre otras.This pedagogical proposal of environmental education seeks to strengthen the black communities settled in the Corregimiento de Potrerito, located in in Jamundí- Valle del Cauca, who have a great environmental and cultural wealth, but are losing their traditional knowledge associated with biodiversity. Its main objective is: To design a pedagogical proposal of environmental education that strengthens black communities in their traditional knowledge associated with biocultural and ethnodevelopment processes in the Corregimiento de Potrerito en Jamundí Valle del Cauca, in turn, has 3 specific objectives: 1. Carry out a socio-environmental characterization to know what is the state in the Potrerito Corregimiento of Jamundí Valle del Cauca. 2. Identify traditional knowledge and biocultural assets conducive to strengthening the sustainable development of the black communities of the Corregimiento de Potrerito-Jamundí-Valle. 3. Structure with the participation of black communities and thematic experts the strategies, objectives, contents and activities that will be part of the pedagogical proposal that will strengthen the communities in their traditional knowledge associated with biocultural processes from an ethno educational approach in Jamundí Valle del Cauca. In methodological terms it will be carried out with the research, action method. Making use of strategies such as social mapping, interviews, surveys, workshops; and with instruments such as field diary, photographs, among others

Improving accountability in recommender systems research through reproducibility

Reproducibility is a key requirement for scientific progress. It allows the reproduction of the works of others, and, as a consequence, to fully trust the reported claims and results. In this work, we argue that, by facilitating reproducibility of recommender systems experimentation, we indirectly address the issues of accountability and transparency in recommender systems research from the perspectives of practitioners, designers, and engineers aiming to assess the capabilities of published research works. These issues have become increasingly prevalent in recent literature. Reasons for this include societal movements around intelligent systems and artificial intelligence striving toward fair and objective use of human behavioral data (as in Machine Learning, Information Retrieval, or Human–Computer Interaction). Society has grown to expect explanations and transparency standards regarding the underlying algorithms making automated decisions for and around us. This work surveys existing definitions of these concepts and proposes a coherent terminology for recommender systems research, with the goal to connect reproducibility to accountability. We achieve this by introducing several guidelines and steps that lead to reproducible and, hence, accountable experimental workflows and research. We additionally analyze several instantiations of recommender system implementations available in the literature and discuss the extent to which they fit in the introduced framework. With this work, we aim to shed light on this important problem and facilitate progress in the field by increasing the accountability of researchThis work has been funded by the Ministerio de Ciencia, Innovación y Universidades (reference: PID2019-108965GB-I00

Applying reranking strategies to route recommendation using sequence-aware evaluation

Venue recommendation approaches have become particularly useful nowadays due to the increasing number of users registered in location-based social networks (LBSNs), applications where it is possible to share the venues someone has visited and establish connections with other users in the system. Besides, the venue recommendation problem has certain characteristics that differ from traditional recommendation, and it can also benefit from other contextual aspects to not only recommend independent venues, but complete routes or venue sequences of related locations. Hence, in this paper, we investigate the problem of route recommendation under the perspective of generating a sequence of meaningful locations for the users, by analyzing both their personal interests and the intrinsic relationships between the venues. We divide this problem into three stages, proposing general solutions to each case: First, we state a general methodology to derive user routes from LBSNs datasets that can be applied in as many scenarios as possible; second, we define a reranking framework that generate sequences of items from recommendation lists using different techniques; and third, we propose an evaluation metric that captures both accuracy and sequentiality at the same time. We report our experiments on several LBSNs datasets and by means of different recommendation quality metrics and algorithms. As a result, we have found that classical recommender systems are comparable to specifically tailored algorithms for this task, although exploiting the temporal dimension, in general, helps on improving the performance of these techniques; additionally, the proposed reranking strategies show promising results in terms of finding a trade-off between relevance, sequentiality, and distance, essential dimensions in both venue and route recommendation tasksThis work has been funded by the Ministerio de Ciencia, Innovación y Universidades (reference: TIN2016-80630-P) and by the European Social Fund (ESF), within the 2017 call for predoctoral contract

Report from Dagstuhl Seminar 23031: Frontiers of Information Access Experimentation for Research and Education

This report documents the program and the outcomes of Dagstuhl Seminar 23031 ``Frontiers of Information Access Experimentation for Research and Education'', which brought together 37 participants from 12 countries. The seminar addressed technology-enhanced information access (information retrieval, recommender systems, natural language processing) and specifically focused on developing more responsible experimental practices leading to more valid results, both for research as well as for scientific education. The seminar brought together experts from various sub-fields of information access, namely IR, RS, NLP, information science, and human-computer interaction to create a joint understanding of the problems and challenges presented by next generation information access systems, from both the research and the experimentation point of views, to discuss existing solutions and impediments, and to propose next steps to be pursued in the area in order to improve not also our research methods and findings but also the education of the new generation of researchers and developers. The seminar featured a series of long and short talks delivered by participants, who helped in setting a common ground and in letting emerge topics of interest to be explored as the main output of the seminar. This led to the definition of five groups which investigated challenges, opportunities, and next steps in the following areas: reality check, i.e. conducting real-world studies, human-machine-collaborative relevance judgment frameworks, overcoming methodological challenges in information retrieval and recommender systems through awareness and education, results-blind reviewing, and guidance for authors.Comment: Dagstuhl Seminar 23031, report

Controlling the Fairness / Accuracy Tradeoff in Recommender Systems

Recommender systems are one of the most pervasive applications of machine learning. They play a pivotal role in helping users find items tailored to their taste. Although these systems intend to assist people in their information needs, they can cause implicit or explicit discrimination against individuals or groups. There are several ways that different biases can creep into recommender systems. Reflection of societal and historical prejudices in datasets and during the data collection process, lack of sufficient data on minority groups, lack of suitable evaluation methods and model designs to detect these biases and lessen the unfairness caused by them are among the many reasons for unfairness in these systems. A system needs to defend against the biases in recommendation output to prevent harm and unfairness. However, integrating the goal of fairness with accuracy in recommender systems is challenging, primarily because of this goal's significant trade-offs with accuracy. Accuracy in recommender systems is the ability of that system to predict users' needs and interests accurately. On the other hand, fairness is a complicated concept with a variety of definitions. To use fairness as an objective, we need to define it based on the application area and the context of a problem. Additionally, we need to specify the fairness concerns of the different stakeholders involved in the recommender systems and the fairness priorities of a system. Any of these aspects might disagree with the goal of accuracy. For example, if fairness for content providers is more exposure to users, increasing it might cause a reduction in accuracy. Therefore, controlling the trade-off between accuracy and fairness becomes essential. Throughout this dissertation, several recommendation models and re-ranking approaches are presented that aim to address this problem using in- and post- processing methods. These approaches show promising results, but it is worth mentioning that they have intrinsic limitations and, therefore, shouldn't be considered ultimate solutions