Purpose: With this work we intend to evaluate several generalist and health-specific search engines on retrieval of health information by consumers. We compare the retrieval effectiveness of these engines in different types of clinical queries, medical specialties and condition's severity. Finally, we compare the use of evaluation metrics for binary relevance scales and for graded ones. Design/methodology/approach: We conducted a user study in which users evaluated the relevance of documents retrieved by 4 search engines in 2 different health information needs. Users could choose between generalist (Bing, Google, Sapo and Yahoo!) and health-specific search engines (MedlinePlus, SapoSaúde and WebMD). We then analyse the differences between search engines and groups of information needs with six different measures: graded average precision (gap), average precision (ap), gap@5, gap@10, ap@5 and ap@10. Findings: Results show that generalist web search engines surpass the precision of health-specific engines. Google has the better performance, mainly on the top-10 results. We found that information needs associated with severe conditions are associated with higher precision just like overview and psychiatry questions. Originality/value: Our study is one of the first studies to use a recently proposed measure to evaluate the effectiveness of retrieval systems with graded relevance scales. It includes tasks of several medical specialties, types of clinical questions and different levels of severity what, to the best of our knowledge, has not been done before. Also, it is a study in which users have a large involvement in the experiment. Results are useful to understand how search engines differ in their responses to health information needs, to inform about what types of online health information are more common on the Web and to infer ways to improve this type of search. Keywords: Evaluation, Health information retrieval, User study, Graded-relevance, Web search engines Paper type: Research paper INTRODUCTION Patients, their family and friends, commonly designated by health consumers, are increasingly using the Web to search for health information. The last Pew Internet report on health information According to This study evaluates the performance of 4 generalist search engines (Google, Bing, Yahoo! and Sapo) and 3 health-specific search engines (MedlinePlus, WebMD and SapoSaúde). The evaluation is based on the data collected in a user study with undergraduate students and work tasks defined according to the framework proposed by Borlun
