Testing the stability of “wisdom of crowds” judgments of search results over time and their similarity with the search engine rankings

PURPOSE: One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgments of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the “wisdom of crowds”. The aim of this study is to examine whether aggregated judgments will be more stable and thus more reliable over time than individual user judgments. DESIGN/METHODS: In this study two simple measures are proposed to calculate the aggregated judgments of search results and compare their reliability and stability to individual user judgments. In addition, the aggregated “wisdom of crowds” judgments were used as a means to compare the differences between human assessments of search results and search engine’s rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgments were considered in this study: 1) relevance on a 4-point scale, and 2) ranking on a 10-point scale without ties. FINDINGS: It was found that aggregated judgments are much more stable than individual user judgments, yet they are quite different from search engine rankings. Practical implications: The proposed “wisdom of crowds” based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalization and adapting search engine’s ranking over time to changes in users preferences. ORIGINALITY/VALUE: This is a first study that applies the notion of “wisdom of crowds” to examine the under-explored phenomenon in the literature of “change in time” in user evaluation of relevance

Crossing the academic ocean? Judit Bar-Ilan's oeuvre on search engines studies

[EN] The main objective of this work is to analyse the contributions of Judit Bar-Ilan to the search engines studies. To do this, two complementary approaches have been carried out. First, a systematic literature review of 47 publications authored and co-authored by Judit and devoted to this topic. Second, an interdisciplinarity analysis based on the cited references (publications cited by Judit) and citing documents (publications that cite Judit's work) through Scopus. The systematic literature review unravels an immense amount of search engines studied (43) and indicators measured (especially technical precision, overlap and fluctuation over time). In addition to this, an evolution over the years is detected from descriptive statistical studies towards empirical user studies, with a mixture of quantitative and qualitative methods. Otherwise, the interdisciplinary analysis evidences that a significant portion of Judit's oeuvre was intellectually founded on the computer sciences, achieving a significant, but not exclusively, impact on library and information sciences.Orduña-Malea, E. (2020). Crossing the academic ocean? Judit Bar-Ilan's oeuvre on search engines studies. Scientometrics. 123(3):1317-1340. https://doi.org/10.1007/s11192-020-03450-4S131713401233Bar-Ilan, J. (1998a). On the overlap, the precision and estimated recall of search engines. A case study of the query “Erdos”. Scientometrics,42(2), 207–228. https://doi.org/10.1007/bf02458356.Bar-Ilan, J. (1998b). The mathematician, Paul Erdos (1913–1996) in the eyes of the Internet. Scientometrics,43(2), 257–267. https://doi.org/10.1007/bf02458410.Bar-Ilan, J. (2000). The web as an information source on informetrics? A content analysis. Journal of the American Society for Information Science and Technology,51(5), 432–443. https://doi.org/10.1002/(sici)1097-4571(2000)51:5%3C432:aid-asi4%3E3.0.co;2-7.Bar-Ilan, J. (2001). Data collection methods on the web for informetric purposes: A review and analysis. Scientometrics,50(1), 7–32.Bar-Ilan, J. (2002). Methods for measuring search engine performance over time. Journal of the American Society for Information Science and Technology,53(4), 308–319. https://doi.org/10.1002/asi.10047.Bar-Ilan, J. (2003). Search engine results over time: A case study on search engine stability. Cybermetrics,2/3, 1–16.Bar-Ilan, J. (2005a). Expectations versus reality—Search engine features needed for Web research at mid 2005. Cybermetrics,9, 1–26.Bar-Ilan, J. (2005b). Expectations versus reality—Web search engines at the beginning of 2005. In Proceedings of ISSI 2005: 10th international conference of the international society for scientometrics and informetrics (Vol. 1, pp. 87–96).Bar-Ilan, J. (2010). The WIF of Peter Ingwersen’s website. In B. Larsen, J. W. Schneider, & F. Åström (Eds.), The Janus Faced Scholar a Festschrift in honour of Peter Ingwersen (pp. 119–121). Det Informationsvidenskabelige Akademi. Retrieved 15 January 15, 2020, from https://vbn.aau.dk/ws/portalfiles/portal/90357690/JanusFacedScholer_Festschrift_PeterIngwersen_2010.pdf#page=122.Bar-Ilan, J. (2018). Eugene Garfield on the web in 2001. Scientometrics,114(2), 389–399. https://doi.org/10.1007/s11192-017-2590-9.Bar-Ilan, J., Mat-Hassan, M., & Levene, M. (2006). Methods for comparing rankings of search engine results. Computer Networks,50(10), 1448–1463. https://doi.org/10.1016/j.comnet.2005.10.020.Thelwall, M. (2017). Judit Bar-Ilan: Information scientist, computer scientist, scientometrician. Scientometrics,113(3), 1235–1244. https://doi.org/10.1007/s11192-017-2551-3

Financial literacy resources in US public libraries: website analysis

Purpose: This article explores the financial literacy resources patrons can discover and/or access on the webpages of the largest 48 U.S. public libraries in order to assess the strength of public libraries’ current support to patrons seeking assistance with personal financial matters. Design/methodology/approach: The author completed a website analysis of the largest 48 U.S. public libraries, as defined by the four sets of criteria in the American Library Association (ALA) publication The nation’s largest public libraries. Website analysis was completed via a standardized checklist assessment covering full-site searching, catalog content, the availability of relevant guides and/or workshops, and any other relevant online resources. Findings: Public libraries provide many resources relevant to patrons searching for personal finance topics, but some of these resources are not ideally highlighted on libraries’ websites. Site search tools are generally less efficient than catalog search tools. Only half of the studied libraries have relevant online guides, but all libraries have some relevant online resources. Originality/value: While there are a number of research articles exploring how public libraries support financial literacy in their communities, there has not yet been an in-depth exploration of how public libraries support this literacy specifically through the materials highlighted and/or available via their websites. This research addresses this gap in the literature.Publisher does not allow open access until after publicatio

Search Results: Predicting Ranking Algorithms With User Ratings and User-Driven Data

The purpose of this correlational quantitative study was to examine the possible relationship between user-driven parameters, user ratings, and ranking algorithms. The study’s population consisted of students and faculty in the information technology (IT) field at a university in Huntington, WV. Arrow’s impossibility theorem was used as the theoretical framework for this study. Complete survey data were collected from 47 students and faculty members in the IT field, and a multiple regression analysis was used to measure the correlations between the variables. The model was able to explain 85% of the total variability in the ranking algorithm. The overall model was able to significantly predict the algorithm ranking discounted cumulative gain, R2 = .852, F(3,115) = 220.13, p \u3c .01. The Respondent DCG and Search term variables were the most significant predictor with p = .0001. The overall findings can potentially be useful to content providers who focus their content on a specific niche. The content created by these providers would most likely be focused entirely on that subgroup of interested users. While it is necessary to focus content to the interested users, it may be beneficial to expand the content to more generic terms to help reach potential new users outside of the subgroups of interest. User’s searching for more generic terms could potentially be exposed to more content that would generally require more specific search terms. This exposure with more generic terms could help users expand their knowledge of new content more quickly

What Presentation of Search Engine Results Do Health Information Searchers Prefer?

A study of a sample of online health information searchers was conducted to see what their preferences are with respect to four different display styles for search engine results on health topics. Screen shots of search result display screens were presented to the participants via a Qualtrics (www.qualtrics.com) online survey. The other display types were Display 1: Google standard display, Display 2: Google enhanced with faceted browsable categories, Display 3: Google enhanced with a word cloud for each search result, and Display 4: Google enhanced with an overview word cloud for collection of search results. For each search task, participants were asked to rate the search engine results displays for quality indicators, using Likert-type item rating scales. At the end, in three concluding questions, the participants were asked to choose the display(s) that were best at meeting three specific criteria, based on overall impressions. The evaluations by the participants suggest that the standard Google search results display and the Google screen enhanced with faceted browsable categories were favored over the other two display types.Master of Science in Information Scienc