User-Centered Evaluation of Adaptive and Adaptable Systems

Adaptive and adaptable systems provide tailored output to various users in various contexts. While adaptive systems base their output on implicit inferences, adaptable systems use explicitly provided information. Since the presentation or output of these systems is adapted, standard user-centered evaluation methods do not produce results that can be easily generalized. This calls for a reflection on the appropriateness of standard evaluation methods for user-centered evaluations of these systems. We have conducted a literature review to create an overview of the methods that have been used. When reviewing the empirical evaluation studies we have, among other things, focused on the variables measured and the implementation of results in the (re)design process. The goal of our review has been to compose a framework for user-centered evaluation. In the next phase of the project, we intend to test some of the most valid and feasible methods with an adaptive or adaptable system

Context, intelligence and interactions for personalized systems

This special issue on Context, Intelligence and Interactions for Personalized Systems provides a snapshot of the latest research activities, results, and technologies and application developments focusing on the smart personalised systems in Ambient Intelligence and Humanized Computing. It is intended for researchers and practitioners from artificial intelligence (AI) with expertise in formal modeling, representation and inference on situations, activities and goals; researchers from ubiquitous computing and embedded systems with expertise in context-aware computing; and application developers or users with expertise and experience in user requirements, system implementation and evaluation. The special issue also serves to motivate application scenarios from various domains including smart homes and cities, localisation tracking, image analysis and environmental monitoring. For solution developers and providers of specific application domains, this special issue will provide an opportunity to convey needs and requirements, as well as obtain first-hand information on the latest technologies, prototypes, and application exemplars

Assessing the Effectiveness and Usability of Personalized Internet Search through a Longitudinal Evaluation

This paper discusses a longitudinal user evaluation of Prospector, a personalized Internet meta-search engine capable of personalized re-ranking of search results. Twenty-one participants used Prospector as their primary search engine for 12 days, agreed to have their interaction with the system logged, and completed three questionnaires. The data logs show that the personalization provided by Prospector is successful: participants preferred re-ranked results that appeared higher up. However, the questionnaire results indicated that people would prefer to use Google instead (their search engine of choice). Users would, nevertheless, consider employing a personalized search engine to perform searches with terms that require disambiguation and/or contextualization. We conclude the paper with a discussion on the merit of combining system- and user-centered evaluation for the case of personalized systems

Causal Estimation of User Learning in Personalized Systems

In online platforms, the impact of a treatment on an observed outcome may change over time as 1) users learn about the intervention, and 2) the system personalization, such as individualized recommendations, change over time. We introduce a non-parametric causal model of user actions in a personalized system. We show that the Cookie-Cookie-Day (CCD) experiment, designed for the measurement of the user learning effect, is biased when there is personalization. We derive new experimental designs that intervene in the personalization system to generate the variation necessary to separately identify the causal effect mediated through user learning and personalization. Making parametric assumptions allows for the estimation of long-term causal effects based on medium-term experiments. In simulations, we show that our new designs successfully recover the dynamic causal effects of interest.Comment: EC 202

Performance evaluation of ductless personalized ventilation in comparison with desk fans using numerical simulations

The performance of ductless personalized ventilation (DPV) was compared to the performance of a typical desk fan since they are both stand-alone systems that allow the users to personalize their indoor environment. The two systems were evaluated using a validated computational fluid dynamics (CFD) model of an office room occupied by two users. To investigate the impact of DPV and the fan on the inhaled air quality, two types of contamination sources were modelled in the domain: an active source and a passive source. Additionally, the influence of the compared systems on thermal comfort was assessed using the coupling of CFD with the comfort model developed by the University of California, Berkeley (UCB model). Results indicated that DPV performed generally better than the desk fan. It provided better thermal comfort and showed a superior performance in removing the exhaled contaminants. However, the desk fan performed better in removing the contaminants emitted from a passive source near the floor level. This indicates that the performance of DPV and desk fans depends highly on the location of the contamination source. Moreover, the simulations showed that both systems increased the spread of exhaled contamination when used by the source occupant

User-controllable personalization: A case study with SetFusion

In this research we investigated the role of user controllability on personalized systems by implementing and studying a novel interactive recommender interface, SetFusion. We examined whether allowing the user to control the process of fusing or integrating different algorithms (i.e., different sources of relevance) resulted in increased engagement and a better user experience. The essential contribution of this research stems from the results of a user study (N=40) of controllability in a scenario where users could fuse different recommendation approaches, with the possibility of inspecting and filtering the items recommended. First, we introduce an interactive Venn diagram visualization, which combined with sliders, can provide an efficient visual paradigm for information filtering. Second, we provide a three-fold evaluation of the user experience: objective metrics, subjective user perception, and behavioral measures. Through the analysis of these metrics, we confirmed results from recent studies, such as the effect of trusting propensity on accepting the recommendations and also unveiled the importance of features such as being a native speaker. Our results present several implications for the design and implementation of user-controllable personalized systems

An architecture for personalized systems based on web mining agents

[EN]The development of the present web systems is becoming a complex activity due to the need to integrate the last technologies in order to make more efficient and competitive applications. Endowing systems with personalized recommendation procedures contributes to achieve these objectives. In this paper, a web mining method for personalization is proposed. It uses the information already available from other users to discover patterns that are used later for making recommendations. The work deals with the problem of introducing new information items and new users who do not have a profile. We propose an architectural design of intelligent data mining agents for the system implementation

Designing an Adaptive Web Navigation Interface for Users with Variable Pointing Performance

Many online services and products require users to point and interact with user interface elements. For individuals who experience variable pointing ability due to physical impairments, environmental issues or age, using an input device (e.g., a computer mouse) to select elements on a website can be difficult. Adaptive user interfaces dynamically change their functionality in response to user behavior. They can support individuals with variable pointing abilities by 1) adapting dynamically to make element selection easier when a user is experiencing pointing difficulties, and 2) informing users about these pointing errors. While adaptive interfaces are increasingly prevalent on the Web, little is known about the preferences and expectations of users with variable pointing abilities and how to design systems that dynamically support them given these preferences. We conducted an investigation with 27 individuals who intermittently experience pointing problems to inform the design of an adaptive interface for web navigation. We used a functional high-fidelity prototype as a probe to gather information about user preferences and expectations. Our participants expected the system to recognize and integrate their preferences for how pointing tasks were carried out, preferred to receive information about system functionality and wanted to be in control of the interaction. We used findings from the study to inform the design of an adaptive Web navigation interface, PINATA that tracks user pointing performance over time and provides dynamic notifications and assistance tailored to their specifications. Our work contributes to a better understanding of users' preferences and expectations of the design of an adaptive pointing system

Extracting Relevance and Affect Information from Physiological Text Annotation

We present physiological text annotation, which refers to the practice of associating physiological responses to text content in order to infer characteristics of the user information needs and affective responses. Text annotation is a laborious task, and implicit feedback has been studied as a way to collect annotations without requiring any explicit action from the user. Previous work has explored behavioral signals, such as clicks or dwell time to automatically infer annotations, and physiological signals have mostly been explored for image or video content. We report on two experiments in which physiological text annotation is studied first to 1) indicate perceived relevance and then to 2) indicate affective responses of the users. The first experiment tackles the user’s perception of relevance of an information item, which is fundamental towards revealing the user’s information needs. The second experiment is then aimed at revealing the user’s affective responses towards a -relevant- text document. Results show that physiological user signals are associated with relevance and affect. In particular, electrodermal activity (EDA) was found to be different when users read relevant content than when they read irrelevant content and was found to be lower when reading texts with negative emotional content than when reading texts with neutral content. Together, the experiments show that physiological text annotation can provide valuable implicit inputs for personalized systems. We discuss how our findings help design personalized systems that can annotate digital content using human physiology without the need for any explicit user interaction