Layered evaluation of interactive adaptive systems : framework and formative methods

Peer reviewedPostprin

Assessing the Effectiveness and Usability of Personalized Internet Search through a Longitudinal Evaluation

This paper discusses a longitudinal user evaluation of Prospector, a personalized Internet meta-search engine capable of personalized re-ranking of search results. Twenty-one participants used Prospector as their primary search engine for 12 days, agreed to have their interaction with the system logged, and completed three questionnaires. The data logs show that the personalization provided by Prospector is successful: participants preferred re-ranked results that appeared higher up. However, the questionnaire results indicated that people would prefer to use Google instead (their search engine of choice). Users would, nevertheless, consider employing a personalized search engine to perform searches with terms that require disambiguation and/or contextualization. We conclude the paper with a discussion on the merit of combining system- and user-centered evaluation for the case of personalized systems

Using thematic ontologies for user- and group- based adaptive personalization in web searching

This paper presents Prospector, an adaptive meta-search layer, which performs personalized re-ordering of search results. Prospector combines elements from two approaches to adaptive search support: (a) collaborative web searching; and, (b) personalized searching using semantic metadata. The paper focuses on the way semantic metadata and the users’ search behavior are utilized for user- and group- modeling, as well as on how these models are used to re-rank results returned for individual queries. The paper also outlines past evaluation activities related to Prospector, and discusses potential applications of the approach for the adaptive retrieval of multimedia documents

Transferring an Outcome-Oriented Learning Architecture to an IT Learning Game

Schmitz, B., Klemke, R., Totschnig, M., Czauderna, A., & Specht, M. (2011). Transferring an Outcome-Oriented Learning Architecture to an IT Learning Game. In C. D. Kloos, D. Gillet, R. M. Crespo Carcía, F. Wild, & M. Wolpers (Eds.), Towards Ubiquitous Learning: 6th European Conference on Technology Anhanced Learning, EC-TEL 2011 (pp. 483-488). September, 20-23, 2011, Palermo, Italy. LNCS 6964; Heidelberg, Berlin: Springer.Today’s technology enhanced learning scenarios focus on outcome-oriented delivery of learning processes, contents, and services. Also, learners increasingly demand for innovative and motivating learning scenarios that match their habits of using media. The European project ICOPER researches outcome-oriented learning infrastructures for higher education contexts. The German BMBF-project SpITKom aims at transferring such approaches to basic qualification. Based on a Browser Game, it uses ICOPER’s technical infrastructure which combines learning object metadata repositories, learning outcome repositories, learning design repositories and learner profile repositories. This paper initially depicts the technical infrastructure of an outcome-oriented learning scenario that was developed in the course of ICOPER and then outlines its transformation to the game-based learning approach as realized in the course of SpITKom.ICOPER, SPITKO

Evaluating the effectiveness of explanations for recommender systems : Methodological issues and empirical studies on the impact of personalization

Peer reviewedPostprin

What am I not seeing? An Interactive Approach to Social Content Discovery in Microblogs

In this paper, we focus on the informational and user experience benefits of user-driven topic exploration in microblog communities, such as Twitter, in an inspectable, controllable and personalized manner. To this end, we introduce ``HopTopics'' -- a novel interactive tool for exploring content that is popular just beyond a user's typical information horizon in a microblog, as defined by the network of individuals that they are connected to. We present results of a user study (N=122) to evaluate HopTopics with varying complexity against a typical microblog feed in both personalized and non-personalized conditions. Results show that the HopTopics system, leveraging content from both the direct and extended network of a user, succeeds in giving users a better sense of control and transparency. Moreover, participants had a poor mental model for the degree of novel content discovered when presented with non-personalized data in the Inspectable interface

Effects of Individual Differences in Working Memory on Plan Presentational Choices.

This paper addresses research questions that are central to the area of visualization interfaces for decision support: (RQ1) whether individual user differences in working memory should be considered when choosing how to present visualizations; (RQ2) how to present the visualization to support effective decision making and processing; and (RQ3) how to evaluate the effectiveness of presentational choices. These questions are addressed in the context of presenting plans, or sequences of actions, to users. The experiments are conducted in several domains, and the findings are relevant to applications such as semi-autonomous systems in logistics. That is, scenarios that require the attention of humans who are likely to be interrupted, and require good performance but are not time critical. Following a literature review of different types of individual differences in users that have been found to affect the effectiveness of presentational choices, we consider specifically the influence of individuals' working memory (RQ1). The review also considers metrics used to evaluate presentational choices, and types of presentational choices considered. As for presentational choices (RQ2), we consider a number of variants including interactivity, aggregation, layout, and emphasis. Finally, to evaluate the effectiveness of plan presentational choices (RQ3) we adopt a layered-evaluation approach and measure performance in a dual task paradigm, involving both task interleaving and evaluation of situational awareness. This novel methodology for evaluating visualizations is employed in a series of experiments investigating presentational choices for a plan. A key finding is that emphasizing steps (by highlighting borders) can improve effectiveness on a primary task, but only when controlling for individual variation in working memory