The Relationship between IR Effectiveness Measures and User Satisfaction

This paper presents an experimental study of users assessing the quality of Google web search results. In particular we look at how users' satisfaction correlates with the effectiveness of Google as quantified by IR measures such as precision and the suite of Cumulative Gain measures (CG, DCG, NDCG). Results indicate strong correlation between users' satisfaction, CG and precision, moderate correlation with DCG, with perhaps surprisingly negligible correlation with NDCG. The reasons for the low correlation with NDCG are examined

Users' effectiveness and satisfaction for image retrieval

This paper presents results from an initial user study exploring the relationship between system effectiveness as quantified by traditional measures such as precision and recall, and users’ effectiveness and satisfaction of the results. The tasks involve finding images for recall-based tasks. It was concluded that no direct relationship between system effectiveness and users’ performance could be proven (as shown by previous research). People learn to adapt to a system regardless of its effectiveness. This study recommends that a combination of attributes (e.g. system effectiveness, user performance and satisfaction) is a more effective way to evaluate interactive retrieval systems. Results of this study also reveal that users are more concerned with accuracy than coverage of the search results

The WEB Book experiments in electronic textbook design

This paper describes a series of three evaluations of electronic textbooks on the Web, which focused on assessing how appearance and design can affect users' sense of engagement and directness with the material. The EBONI Project's methodology for evaluating electronic textbooks is outlined and each experiment is described, together with an analysis of results. Finally, some recommendations for successful design are suggested, based on an analysis of all experimental data. These recommendations underline the main findings of the evaluations: that users want some features of paper books to be preserved in the electronic medium, while also preferring electronic text to be written in a scannable style

Critical review of the e-loyalty literature: a purchase-centred framework

Over the last few years, the concept of online loyalty has been examined extensively in the literature, and it remains a topic of constant inquiry for both academics and marketing managers. The tremendous development of the Internet for both marketing and e-commerce settings, in conjunction with the growing desire of consumers to purchase online, has promoted two main outcomes: (a) increasing numbers of Business-to-Customer companies running businesses online and (b) the development of a variety of different e-loyalty research models. However, current research lacks a systematic review of the literature that provides a general conceptual framework on e-loyalty, which would help managers to understand their customers better, to take advantage of industry-related factors, and to improve their service quality. The present study is an attempt to critically synthesize results from multiple empirical studies on e-loyalty. Our findings illustrate that 62 instruments for measuring e-loyalty are currently in use, influenced predominantly by Zeithaml et al. (J Marketing. 1996;60(2):31-46) and Oliver (1997; Satisfaction: a behavioral perspective on the consumer. New York: McGraw Hill). Additionally, we propose a new general conceptual framework, which leads to antecedents dividing e-loyalty on the basis of the action of purchase into pre-purchase, during-purchase and after-purchase factors. To conclude, a number of managerial implementations are suggested in order to help marketing managers increase their customers’ e-loyalty by making crucial changes in each purchase stage

Exploratory study on performance measures as indicators of IS effectiveness

The Information Systems Directorate at JSC/NASA has undertaken the reevaluation of its performance measures process and measures. Under the direction of a quality approach it is essential to identify an external perspective of how well an organization is performing. This study was conducted with two major objectives: (1) survey and summarize the academic literature on performance measures as indicators of information systems (IS) effectiveness; and (2) survey organizations for their experience in measuring for IS effectiveness. Four approaches to measuring the effectiveness of IS performance were identified: (1) listen to the customer for the things they need; (2) align with corporate goals; (3) benchmark against well-respected organizations; and (4) ask yourself what critical factors lead to success. The list of known methods for soliciting customer feedback are as follows: (1) executive visit; (2) survey, interview, and focus group; (3) complaints and compliments; and (4) service level agreements. A common set of characteristics that satisfy customers was identified from the literature. The list includes elements such as the following: accuracy, timeliness, relevance, understandability, reliability, and completeness. Future research in this topic area should prove beneficial to determine the metrics for external validity