Layered evaluation of interactive adaptive systems : framework and formative methods

Peer reviewedPostprin

From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise through Online Reviews

Recommending products to consumers means not only understanding their tastes, but also understanding their level of experience. For example, it would be a mistake to recommend the iconic film Seven Samurai simply because a user enjoys other action movies; rather, we might conclude that they will eventually enjoy it -- once they are ready. The same is true for beers, wines, gourmet foods -- or any products where users have acquired tastes: the `best' products may not be the most `accessible'. Thus our goal in this paper is to recommend products that a user will enjoy now, while acknowledging that their tastes may have changed over time, and may change again in the future. We model how tastes change due to the very act of consuming more products -- in other words, as users become more experienced. We develop a latent factor recommendation system that explicitly accounts for each user's level of experience. We find that such a model not only leads to better recommendations, but also allows us to study the role of user experience and expertise on a novel dataset of fifteen million beer, wine, food, and movie reviews.Comment: 11 pages, 7 figure

Pragmatic Ontology Evolution: Reconciling User Requirements and Application Performance

Increasingly, organizations are adopting ontologies to describe their large catalogues of items. These ontologies need to evolve regularly in response to changes in the domain and the emergence of new requirements. An important step of this process is the selection of candidate concepts to include in the new version of the ontology. This operation needs to take into account a variety of factors and in particular reconcile user requirements and application performance. Current ontology evolution methods focus either on ranking concepts according to their relevance or on preserving compatibility with existing applications. However, they do not take in consideration the impact of the ontology evolution process on the performance of computational tasks – e.g., in this work we focus on instance tagging, similarity computation, generation of recommendations, and data clustering. In this paper, we propose the Pragmatic Ontology Evolution (POE) framework, a novel approach for selecting from a group of candidates a set of concepts able to produce a new version of a given ontology that i) is consistent with the a set of user requirements (e.g., max number of concepts in the ontology), ii) is parametrised with respect to a number of dimensions (e.g., topological considerations), and iii) effectively supports relevant computational tasks. Our approach also supports users in navigating the space of possible solutions by showing how certain choices, such as limiting the number of concepts or privileging trendy concepts rather than historical ones, would reflect on the application performance. An evaluation of POE on the real-world scenario of the evolving Springer Nature taxonomy for editorial classification yielded excellent results, demonstrating a significant improvement over alternative approaches

Hybrid group recommendations for a travel service

Recommendation techniques have proven their usefulness as a tool to cope with the information overload problem in many classical domains such as movies, books, and music. Additional challenges for recommender systems emerge in the domain of tourism such as acquiring metadata and feedback, the sparsity of the rating matrix, user constraints, and the fact that traveling is often a group activity. This paper proposes a recommender system that offers personalized recommendations for travel destinations to individuals and groups. These recommendations are based on the users' rating profile, personal interests, and specific demands for their next destination. The recommendation algorithm is a hybrid approach combining a content-based, collaborative filtering, and knowledge-based solution. For groups of users, such as families or friends, individual recommendations are aggregated into group recommendations, with an additional opportunity for users to give feedback on these group recommendations. A group of test users evaluated the recommender system using a prototype web application. The results prove the usefulness of individual and group recommendations and show that users prefer the hybrid algorithm over each individual technique. This paper demonstrates the added value of various recommendation algorithms in terms of different quality aspects, compared to an unpersonalized list of the most-popular destinations

Human-Machine Collaborative Optimization via Apprenticeship Scheduling

Coordinating agents to complete a set of tasks with intercoupled temporal and resource constraints is computationally challenging, yet human domain experts can solve these difficult scheduling problems using paradigms learned through years of apprenticeship. A process for manually codifying this domain knowledge within a computational framework is necessary to scale beyond the ``single-expert, single-trainee" apprenticeship model. However, human domain experts often have difficulty describing their decision-making processes, causing the codification of this knowledge to become laborious. We propose a new approach for capturing domain-expert heuristics through a pairwise ranking formulation. Our approach is model-free and does not require enumerating or iterating through a large state space. We empirically demonstrate that this approach accurately learns multifaceted heuristics on a synthetic data set incorporating job-shop scheduling and vehicle routing problems, as well as on two real-world data sets consisting of demonstrations of experts solving a weapon-to-target assignment problem and a hospital resource allocation problem. We also demonstrate that policies learned from human scheduling demonstration via apprenticeship learning can substantially improve the efficiency of a branch-and-bound search for an optimal schedule. We employ this human-machine collaborative optimization technique on a variant of the weapon-to-target assignment problem. We demonstrate that this technique generates solutions substantially superior to those produced by human domain experts at a rate up to 9.5 times faster than an optimization approach and can be applied to optimally solve problems twice as complex as those solved by a human demonstrator.Comment: Portions of this paper were published in the Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) in 2016 and in the Proceedings of Robotics: Science and Systems (RSS) in 2016. The paper consists of 50 pages with 11 figures and 4 table

Opening the Black Box: Explaining the Process of Basing a Health Recommender System on the I-Change Behavioral Change Model

Recommender systems are gaining traction in healthcare because they can tailor recommendations based on users' feedback concerning their appreciation of previous health-related messages. However, recommender systems are often not grounded in behavioral change theories, which may further increase the effectiveness of their recommendations. This paper's objective is to describe principles for designing and developing a health recommender system grounded in the I-Change behavioral change model that shall be implemented through a mobile app for a smoking cessation support clinical trial. We built upon an existing smoking cessation health recommender system that delivered motivational messages through a mobile app. A group of experts assessed how the system may be improved to address the behavioral change determinants of the I-Change behavioral change model. The resulting system features a hybrid recommender algorithm for computer tailoring smoking cessation messages. A total of 331 different motivational messages were designed using 10 health communication methods. The algorithm was designed to match 58 message characteristics to each user pro le by following the principles of the I-Change model and maintaining the bene ts of the recommender system algorithms. The mobile app resulted in a streamlined version that aimed to improve the user experience, and this system's design bridges the gap between health recommender systems and the use of behavioral change theories. This article presents a novel approach integrating recommender system technology, health behavior technology, and computer-tailored technology. Future researchers will be able to build upon the principles applied in this case study.European Union's Horizon 2020 Research and Innovation Programme under Grant 68112

Recomendation systems and crowdsourcing: a good wedding for enabling innovation? Results from technology affordances and costraints theory

Recommendation Systems have come a long way since their first appearance in the e-commerce platforms.Since then, evolved Recommendation Systems have been successfully integrated in social networks. Now its time to test their usability and replicate their success in exciting new areas of web -enabled phenomena. One of these is crowdsourcing. Research in the IS field is investigating the need, benefits and challenges of linking the two phenomena. At the moment, empirical works have only highlighted the need to implement these techniques for tasks assignment in crowdsourcing distributed work platforms and the derived benefits for contributors and firms. We review the variety of the tasks that can be crowdsourced through these platforms and theoretically evaluate the efficiency of using RS to recommend a task in creative crowdsourcing platforms. Adopting a Technology Affordances and Constraints Theory, an emerging perspective in the Information Systems (IS) literature to understand technology use and consequences, we anticipate the tensions that this implementation can generate

User interface patterns in recommendation-empowered content intensive multimedia applications

Design Patterns (DPs) are acknowledged as powerful conceptual tools to improve design quality and to reduce time and cost of the development process by effect of the reuse of “good” design solutions. In many fields (e.g., software engineering, web engineering, interface design) patterns are widely used by practitioners and are also investigated from a research perspective. Still, they have been seldom explored in the arena of Recommender Systems (RSs). RSs provide suggestions (“recommendations”) for items that are likely to be appropriate for the user profile, and are increasingly adopted in content-intensive multimedia applications to complement traditional forms of search in large information spaces. This paper explores RSs through the lens of User Interface (UI) Design Patterns. We have performed a systematic analysis of 54 recommendation-empowered content-intensive multimedia applications, in order to: (i) discover the occurrences of existing domain independent UI patterns; (ii) identify frequently adopted UI solutions that are not modelled by existing patterns, and define a set of new UI patterns, some of which are specific of the interfaces for recommendation features while others can be useful also in a broader context. The results of our inspection have been discussed with and evaluated by a team of experts, leading to a consolidated set of 14 new patterns that are reported in the paper. Reusing pattern-based design solutions instead of building new solutions from scratch enables novice and expert designers to build good UIs for Recommendation-empowered content intensive multimedia applications more effectively, and ultimately can improve the UX experience in this class of systems. From a broader perspective, our work can stimulate future research bridging Recommender Systems, Web Engineering and Interface Design by means of Design Patterns, and highlights new research directions also discussed in the paper