Offline Evaluation of Reward-Optimizing Recommender Systems: The Case of Simulation

Both in academic and industry-based research, online evaluation methods are seen as the golden standard for interactive applications like recommendation systems. Naturally, the reason for this is that we can directly measure utility metrics that rely on interventions, being the recommendations that are being shown to users. Nevertheless, online evaluation methods are costly for a number of reasons, and a clear need remains for reliable offline evaluation procedures. In industry, offline metrics are often used as a first-line evaluation to generate promising candidate models to evaluate online. In academic work, limited access to online systems makes offline metrics the de facto approach to validating novel methods. Two classes of offline metrics exist: proxy-based methods, and counterfactual methods. The first class is often poorly correlated with the online metrics we care about, and the latter class only provides theoretical guarantees under assumptions that cannot be fulfilled in real-world environments. Here, we make the case that simulation-based comparisons provide ways forward beyond offline metrics, and argue that they are a preferable means of evaluation.Comment: Accepted at the ACM RecSys 2021 Workshop on Simulation Methods for Recommender System

New Measures for Offline Evaluation of Learning Path Recommenders

International audienceRecommending students useful and effective learning paths is highly valuable to improve their learning experience. The evaluation of the effectiveness of this recommendation is a challenging task that can be performed online or offline. Online evaluation is highly popular but it relies on actual path recommendations to students, which may have dramatic implications. Offline evaluation relies on static datasets of students' learning activities and simulates paths recommendations. Although easier to run, it is difficult to accurately evaluate offline the effectiveness of a learning path recommendation. To tackle this issue, this work proposes simple offline evaluation measures. We show that they actually allow to characterise and differentiate the algorithms

COVID-19 and its implications on students’ learning behaviour

Due to the outbreak of COVID-19, the educational institutions in India suddenly switched to online mode of teaching. This new learning mode gives the flexibility to connect at any time from any place. This sudden shift has impacted the learning behaviour of students to a large extent, which is studied and analysed in this paper for a computer programming course. An online questionnaire is prepared and circulated among the students for which 158 responses were received. Based on the responses, it is found out from the analysis that 75.32% of students favour learning this course in offline mode while 48.1% favour the virtual mode. The maximum support for the classroom teaching is evident from the data which shows that 98.73% students find the teacher’s competency good in offline mode, 98.1% find teacher’s content delivery effective in offline mode and 79.75% are of the view that possibility of frequent interaction is more in offline mode, whereas 87.97%, 85.44% and 42.41% of students are congenial with the online mode in terms of same parameters. Also, 69.62% of students are comfortable with offline mode, while 55.06% with online mode. For evaluation mode and pattern of question paper, 78.5% favour online mode of evaluation with a mix of multiple-choice questions and coding questions. It is irrespective of their preference to offline mode for teaching-learning. In the end, some recommendations are proposed based on the analysis to improve the teaching-learning methodology during the time of crisis

Analyzing the students\u27 learning behaviour for a technical course during COVID-19

The purpose of this paper is to analyse the learning behaviour of students towards a technical course in the two learning modes, remote learning (online) and in-person learning (offline). Due to the outbreak of Covid-19 pandemic in India, the educational fraternity has successfully reached out to the students using the various virtual tools available. Although, the offline mode of teaching-learning i.e. the actual classroom interaction is quite important as far as a technical course is concerned, but during these tough times the online platforms like Zoom, Webex meetings, Google meet have made the teaching-learning feasible remotely at any time from any place. This paper compares the learning behaviour of students in the two modes, emergency virtual mode and offline mode. A total of 213 Bachelors of Engineering (BE) students studying a technical course, Modern and Computational Physics, participated in the survey and their responses based on a questionnaire were recorded. The questionnaire considered all aspects related to the delivery of contents, the evaluation method, the preferred way of clarifying students’ doubts, course difficulty level and duration of the course. The analysis suggests that 72.3% of students are in favour of learning this course using offline mode, while 27.7 % of students are comfortable with virtual online mode. Furthermore, the present study reveals that 95.7%, 95.3%, and 75.1 % of students are congenial with the offline mode in terms of teacher competency, content delivery, and interaction possibility respectively, whereas 85%, 76.9% and 48.4% of students are congenial with the remote online mode in terms of same parameters. The higher magnitude of average mean value for offline mode (3.99) anticipates its dominance over online mode (3.18). More than 50% students favoured online mode along with multiple choice question papers for the evaluation process irrespective of their preference to offline mode for teaching-learning. Based on the present analysis, some recommendations are proposed as the future strategies to improve the performance of teaching-learning activities during the times of crisis

Random performance differences between online recommender system algorithms

In the evaluation of recommender systems, the quality of recommendations made by a newly proposed algorithm is compared to the state-of-the-art, using a given quality measure and dataset. Validity of the evaluation depends on the assumption that the evaluation does not exhibit artefacts resulting from the process of collecting the dataset. The main difference between online and offline evaluation is that in the online setting, the user’s response to a recommendation is only observed once. We used the NewsREEL challenge to gain a deeper understanding of the implications of this difference for making comparisons between different recommender systems. The experiments aim to quantify the expected degree of variation in performance that cannot be attributed to differences between systems. We classify and discuss the non-algorithmic causes of performance differences observed

The Benefits and Costs of Online Privacy Legislation

Many people are concerned that information about their private life is more readily available and more easily captured on the Internet as compared to offline technologies. Specific concerns include unwanted email, credit card fraud, identity theft, and harassment. This paper analyzes key issues surrounding the protection of online privacy. It makes three important contributions: First, it provides the most comprehensive assessment to date of the estimated benefits and costs of regulating online privacy. Second, it provides the most comprehensive evaluation of legislation and legislative proposals in the U.S. aimed at protecting online privacy. Finally, it offers some policy prescriptions for the regulation of online privacy and suggests areas for future research. After analyzing the current debate on online privacy and assessing the potential costs and benefits of proposed regulations, our specific recommendations concerning the government's involvement in protecting online privacy include the following: The government should fund research that evaluates the effectiveness of existing privacy legislation before considering new regulations. The government should not generally regulate matters of privacy differently based on whether an issue arises online or offline. The government should not require a Web site to provide notification of its privacy policy because the vast majority of commercial U.S.-based Web sites already do so. The government should distinguish between how it regulates the use and dissemination of highly sensitive information, such as certain health records or Social Security numbers, versus more general information, such as consumer name and purchasing habits. The government should not require companies to provide consumers broad access to the personal information that is collected online for marketing purposes because the benefits do not appear to be significant and the costs could be quite high. The government should make it easier for the public to obtain information on online privacy and the tools available for consumers to protect their own privacy. The message of this paper is not that online privacy should be unregulated, but rather that policy makers should think through their options carefully, weighing the likely costs and benefits of each proposal.

Efficient Large-Scale Visual Representation Learning

In this article, we present our approach to single-modality visual representation learning. Understanding visual representations of product content is vital for recommendations, search, and advertising applications in e-commerce. We detail and contrast techniques used to fine-tune large-scale visual representation learning models in an efficient manner under low-resource settings, including several pretrained backbone architectures, both in the convolutional neural network as well as the vision transformer family. We highlight the challenges for e-commerce applications at-scale and highlight the efforts to more efficiently train, evaluate, and serve visual representations. We present ablation studies evaluating the representation offline performance for several downstream tasks, including our visually similar ad recommendations. To this end, we present a novel text-to-image generative offline evaluation method for visually similar recommendation systems. Finally, we include online results from deployed machine learning systems in production at Etsy