Evaluating the retrieval effectiveness of Web search engines using a representative query sample

Search engine retrieval effectiveness studies are usually small-scale, using only limited query samples. Furthermore, queries are selected by the researchers. We address these issues by taking a random representative sample of 1,000 informational and 1,000 navigational queries from a major German search engine and comparing Google's and Bing's results based on this sample. Jurors were found through crowdsourcing, data was collected using specialised software, the Relevance Assessment Tool (RAT). We found that while Google outperforms Bing in both query types, the difference in the performance for informational queries was rather low. However, for navigational queries, Google found the correct answer in 95.3 per cent of cases whereas Bing only found the correct answer 76.6 per cent of the time. We conclude that search engine performance on navigational queries is of great importance, as users in this case can clearly identify queries that have returned correct results. So, performance on this query type may contribute to explaining user satisfaction with search engines

Auditing Search Engines for Differential Satisfaction Across Demographics

Many online services, such as search engines, social media platforms, and digital marketplaces, are advertised as being available to any user, regardless of their age, gender, or other demographic factors. However, there are growing concerns that these services may systematically underserve some groups of users. In this paper, we present a framework for internally auditing such services for differences in user satisfaction across demographic groups, using search engines as a case study. We first explain the pitfalls of na\"ively comparing the behavioral metrics that are commonly used to evaluate search engines. We then propose three methods for measuring latent differences in user satisfaction from observed differences in evaluation metrics. To develop these methods, we drew on ideas from the causal inference literature and the multilevel modeling literature. Our framework is broadly applicable to other online services, and provides general insight into interpreting their evaluation metrics.Comment: 8 pages Accepted at WWW 201

Evaluation of a personalized digital library based on cognitive styles: Adaptivity vs. adaptability

Personalization can be addressed by adaptability and adaptivity, which have different advantages and disadvantages. This study investigates how digital library users react to these two techniques. More specifically, we develop a personalized digital library to suit the needs of different cognitive styles based on the findings of our previous work (Frias-Martinez, et al., in press). The personalized digital library includes two versions: adaptive version and adaptable version. The results showed that users not only performed better in the adaptive version, but also they perceived more positively to the adaptive version. In addition, cognitive styles have great effects on users’ responses to adaptability and adaptivity. These results provide guidance for designers to select suitable techniques to develop personalized digital libraries

What is usability in the context of the digital library and how can it be measured?

This paper reviews how usability has been defined in the context of the digital library, what methods have been applied and their applicability, and proposes an evaluation model and a suite of instruments for evaluating usability for academic digital libraries. The model examines effectiveness, efficiency, satisfaction, and learnability. It is found that there exists an interlocking relationship among effectiveness, efficiency, and satisfaction. It also examines how learnability interacts with these three attributes

The WEB Book experiments in electronic textbook design

This paper describes a series of three evaluations of electronic textbooks on the Web, which focused on assessing how appearance and design can affect users' sense of engagement and directness with the material. The EBONI Project's methodology for evaluating electronic textbooks is outlined and each experiment is described, together with an analysis of results. Finally, some recommendations for successful design are suggested, based on an analysis of all experimental data. These recommendations underline the main findings of the evaluations: that users want some features of paper books to be preserved in the electronic medium, while also preferring electronic text to be written in a scannable style

Workshop on web information seeking and interaction

The World Wide Web has provided access to a diverse range of information sources and systems. People engaging with this rich network of information may need to interact with different technologies, interfaces, and information providers in the course of a single search task. These systems may offer different interaction affordances and require users to adapt their informationseeking strategies. Not only is this challenging for users, but it also presents challenges for the designers of interactive systems, who need to make their own system useful and usable to broad user groups. The popularity of Web browsing and Web search engines has given rise to distinct forms of information-seeking behaviour, and new interaction styles, but we do not yet fully understand these or their implications for the development of new systems