A Top-N Recommender System Evaluation Protocol Inspired by Deployed Systems

The evaluation of recommender systems is crucial for their development. In today's recommendation landscape there are many standardized recommendation algorithms and approaches, however, there exists no standardized method for experimental setup of evaluation -- not even for widely used measures such as precision and root-mean-squared error. This creates a setting where comparison of recommendation results using the same datasets becomes problematic. In this paper, we propose an evaluation protocol specifically developed with the recommendation use-case in mind, i.e. the recommendation of one or several items to an end user. The protocol attempts to closely mimic a scenario of a deployed (production) recommendation system, taking specific user aspects into consideration and allowing a comparison of small and large scale recommendat

Supporting Situation Awareness and Decision Making in Weather Forecasting

Weather forecasting is full of uncertainty, and as in domains such as air traffic control or medical decision making, decision support systems can affect a forecaster’s ability to make accurate and timely judgments. Well-designed decision aids can help forecasters build situation awareness (SA), a construct regarded as a component of decision making. SA involves the ability to perceive elements within a system, comprehend their significance, and project their meaning into the future in order to make a decision. However, how SA is affected by uncertainty within a system has received little attention. This tension between managing uncertainty, situation assessment, and the impact that technology has on the two, is the focus of this dissertation. To address this tension, this dissertation is centered on the evaluation of a set of coupled models that integrate rainfall observations and hydrologic simulations, coined “the FLASH system” (Flooded Locations and Simulated Hydrographs project). Prediction of flash flooding is unique from forecasting other weather-related threats due to its multi-disciplinary nature. In the United States, some weather forecasters have limited hydrologic forecasting experience. Unlike FLASH, current flash flood forecasting tools are based upon rainfall rates, and with the recent expansion into coupled rainfall and hydrologic models, forecasters have to learn quickly how to incorporate these new data sources into their work. New models may help forecasters to increase their prediction skill, but no matter how far the technology advances, forecasters must be able to accept and integrate the new tools into their work in order to gain any benefit. A focus on human factors principles in the design stage can help to ensure that by the time the product is transitioned into operational use, the decision support system addresses users’ needs while minimizing task time, workload, and attention constraints. This dissertation discusses three qualitative and quantitative studies designed to explore the relationship between flash flood forecasting, decision aid design, and SA. The first study assessed the effects of visual data aggregation methods on perception and comprehension of a flash flood threat. Next, a mixed methods approach described how forecasters acquire SA and mitigate situational uncertainty during real-time forecasting operations. Lastly, the third study used eye tracking assessment to identify the effects of an automated forecasting decision support tool on SA and information scanning behavior. Findings revealed that uncertainty management in forecasting involves individual, team, and organizational processes. We make several recommendations for future decision support systems to promote SA and performance in the weather forecasting domain

Advances in next-track music recommendation

Technological advances in the music industry have dramatically changed how people access and listen to music. Today, online music stores and streaming services offer easy and immediate means to buy or listen to a huge number of songs. One traditional way to find interesting items in such cases when a vast amount of choices are available is to ask others for recommendations. Music providers utilize correspondingly music recommender systems as a software solution to the problem of music overload to provide a better user experience for their customers. At the same time, an enhanced user experience can lead to higher customer retention and higher business value for music providers. Different types of music recommendations can be found on today's music platforms, such as Spotify or Deezer. Providing a list of currently trending music, finding similar tracks to the user's favorite ones, helping users discover new artists, or recommending curated playlists for a certain mood (e.g., romantic) or activity (e.g., driving) are examples of common music recommendation scenarios. "Next-track music recommendation" is a specific form of music recommendation that relies mainly on the user's recently played tracks to create a list of tracks to be played next. Next-track music recommendations are used, for instance, to support users during playlist creation or to provide personalized radio stations. A particular challenge in this context is that the recommended tracks should not only match the general taste of the listener but should also match the characteristics of the most recently played tracks. This thesis by publication focuses on the next-track music recommendation problem and explores some challenges and questions that have not been addressed in previous research. In the first part of this thesis, various next-track music recommendation algorithms as well as approaches to evaluate them from the research literature are reviewed. The recommendation techniques are categorized into the four groups of content-based filtering, collaborative filtering, co-occurrence-based, and sequence-aware algorithms. Moreover, a number of challenges, such as personalizing next-track music recommendations and generating recommendations that are coherent with the user's listening history are discussed. Furthermore, some common approaches in the literature to determine relevant quality criteria for next-track music recommendations and to evaluate the quality of such recommendations are presented. The second part of the thesis contains a selection of the author's publications on next- track music recommendation as follows. 1. The results of comprehensive analyses of the musical characteristics of manually created playlists for music recommendation; 2. the results of a multi-dimensional comparison of different academic and commercial next-track recommending techniques; 3. the results of a multi-faceted comparison of different session-based recommenders, among others, for the next-track music recommendation problem with respect to their accuracy, popularity bias, catalog coverage as well as computational complexity; 4. a two-phase approach to recommend accurate next-track recommendations that also match the characteristics of the most recent listening history; 5. a personalization approach based on multi-dimensional user models that are extracted from the users' long-term preferences; 6. a user study with the aim of determining the quality perception of next-track music recommendations generated by different algorithms

Advances in session-based and session-aware recommendation

As of today, personalized item suggestions provided by an automated recommender system have become a crucial part of many online services, e.g., online shops or media streaming applications, and extensive evidence exists that such systems increase both the user experience as well as the revenue of the providers. In academia, the recommendation problem is often framed as finding suitable items that a user is not yet aware of based on his long-term preference profile. In the real world, however, this problem formulation has a number of problems. Long-term profiles, e.g., are not available for new or anonymous users and recommendations can then only be based on the few most recent interactions in an ongoing usage session. Various approaches to this highly relevant setting of session-based recommendation that recently emerged in the research community were proposed over the recent years. However, in terms of the evaluation procedure, no common standard has been established so far. In this thesis, the author, therefore, proposes a publicly available framework for reproducible research and, furthermore, fairly compares many approaches, of which some were proposed by himself. Extensive experiments and a user study surprisingly showed that comparably simple nearest-neighbor techniques usually outperform recent deep learning models across many domains, datasets, and metrics. Even if long-term preferences are available for the users, recent works indicated that it might still be beneficial to consider the ongoing session, e.g., because a user started the session with a specific intent in mind. The author of this thesis, thus, conducted a systematic statistical analysis to assess what helps recommendations in being effective in such a session-aware scenario. This analysis is based on log data from a fashion retailer and insights were, furthermore, operationalized into novel session-aware recommendation approaches. Matching items of the customer’s ongoing session, reminding him of previously inspected clothes, recommending discounted items, and considering recent trends in the community showed to be particularly effective strategies, not only for item-item recommendation but also in the related scenario of search personalization

Supporting complex workflows for data-intensive discovery reliably and efficiently

Scientific workflows have emerged as well-established pillars of large-scale computational science and appeared as torchbearers to formalize and structure a massive amount of complex heterogeneous data and accelerate scientific progress. Scientists of diverse domains can analyze their data by constructing scientific workflows as a useful paradigm to manage complex scientific computations. A workflow can analyze terabyte-scale datasets, contain numerous individual tasks, and coordinate between heterogeneous tasks with the help of scientific workflow management systems (SWfMSs). However, even for expert users, workflow creation is a complex task due to the dramatic growth of tools and data heterogeneity. Scientists are now more willing to publicly share scientific datasets and analysis pipelines in the interest of open science. As sharing of research data and resources increases in scientific communities, scientists can reuse existing workflows shared in several workflow repositories. Unfortunately, several challenges can prevent scientists from reusing those workflows, which hurts the purpose of the community-oriented knowledge base. In this thesis, we first identify the repositories that scientists use to share and reuse scientific workflows. Among several repositories, we find Galaxy repositories have numerous workflows, and Galaxy is the mostly used SWfMS. After selecting the Galaxy repositories, we attempt to explore the workflows and encounter several challenges in reusing them. We classify the reusability status (reusable/nonreusable). Based on the effort level, we further categorize the reusable workflows (reusable without modification, easily reusable, moderately difficult to reuse, and difficult to reuse). Upon failure, we record the associated challenges that prevent reusability. We also list the actions upon success. The challenges preventing reusability include tool upgrading, tool support unavailability, design flaws, incomplete workflows, failure to load a workflow, etc. We need to perform several actions to overcome the challenges. The actions include identifying proper input datasets, updating/upgrading tools, finding alternative tools support for obsolete tools, debugging to find the issue creating tools and connections and solving them, modifying tools connections, etc. Such challenges and our action list offer guidelines to future workflow composers to create better workflows with enhanced reusability. A SWfMS stores provenance data at different phases of a workflow life cycle, which can help workflow construction. This provenance data allows reproducibility and knowledge reuse in the scientific community. But, this provenance information is usually many times larger than the workflow and input data, and managing provenance data is growing in complexity with large-scale applications. In our second study, we document the challenges of provenance management and reuse in e-science, focusing primarily on scientific workflow approaches by exploring different SWfMSs and provenance management systems. We also investigate the ways to overcome the challenges. Creating a workflow is difficult but essential for data-intensive complex analysis, and the existing workflows have several challenges to be reused, so in our third study, we build a recommendation system to recommend tool(s) using machine learning approaches to help scientists create optimal, error-free, and efficient workflows by using existing reusable workflows in Galaxy workflow repositories. The findings from our studies and proposed techniques have the potential to simplify the data-intensive analysis, ensuring reliability and efficiency

Sistemas de recomendación basados en técnicas de predicción de enlaces para jueces en línea

La oferta de todo tipo de productos o experiencias que se puede encontrar en Internet hoy en día es inmensa y difícil de valorar para los usuarios que quieren buscar un producto que se adapte a sus necesidades. Debido a este problema, surgen los sistemas de recomendación, que ayudan a los usuarios a encontrar productos que sean de su interés facilitando sus tareas de búsqueda. Los sistemas de recomendación están implantados en muchísimas plataformas de consumo, pero no en otro tipo de plataformas donde su uso también sería interesante y necesario. Una de estas plataformas son los jueces en línea, donde los sistemas de recomendación podrían ayudar a los usuarios en la selección de los problemas a resolver que les resulten más interesantes. En este Trabajo Fin de Máster se proponen diferentes métodos de recomendación para implantar en jueces en línea que están basados en grafos de interacciones y que hacen uso de técnicas de predicción de enlaces con el fin de generar recomendaciones. En este trabajo se ha realizado una evaluación de los métodos de recomendación propuestos a través de la generación de experimentos realizados sobre el juez en línea de Acepta el Reto con el objetivo de determinar qué métodos resultan más prometedore

Harnessing the power of the general public for crowdsourced business intelligence: a survey

International audienceCrowdsourced business intelligence (CrowdBI), which leverages the crowdsourced user-generated data to extract useful knowledge about business and create marketing intelligence to excel in the business environment, has become a surging research topic in recent years. Compared with the traditional business intelligence that is based on the firm-owned data and survey data, CrowdBI faces numerous unique issues, such as customer behavior analysis, brand tracking, and product improvement, demand forecasting and trend analysis, competitive intelligence, business popularity analysis and site recommendation, and urban commercial analysis. This paper first characterizes the concept model and unique features and presents a generic framework for CrowdBI. It also investigates novel application areas as well as the key challenges and techniques of CrowdBI. Furthermore, we make discussions about the future research directions of CrowdBI