Evaluating the retrieval effectiveness of Web search engines using a representative query sample

Search engine retrieval effectiveness studies are usually small-scale, using only limited query samples. Furthermore, queries are selected by the researchers. We address these issues by taking a random representative sample of 1,000 informational and 1,000 navigational queries from a major German search engine and comparing Google's and Bing's results based on this sample. Jurors were found through crowdsourcing, data was collected using specialised software, the Relevance Assessment Tool (RAT). We found that while Google outperforms Bing in both query types, the difference in the performance for informational queries was rather low. However, for navigational queries, Google found the correct answer in 95.3 per cent of cases whereas Bing only found the correct answer 76.6 per cent of the time. We conclude that search engine performance on navigational queries is of great importance, as users in this case can clearly identify queries that have returned correct results. So, performance on this query type may contribute to explaining user satisfaction with search engines

Performance Evaluation of Selected Search Engines

Search Engines have become an integral part of daily internet usage. The search engine is the first stop for web users when they are looking for a product. Information retrieval may be viewed as a problem of classifying items into one of two classes corresponding to interesting and uninteresting items respectively. A natural performance metric in this context is classification accuracy, defined as the fraction of the system's interesting/uninteresting predictions that agree with the user's assessments. On the other hand, the field of information retrieval has two classical performance evaluation metrics: precision, the fraction of the items retrieved by the system that are interesting to the user, and recall, the fraction of the items of interest to the user that are retrieved by the system. Measuring the information retrieval effectiveness of World Wide Web search engines is costly because of human relevance judgments involved. However, both for business enterprises and people it is important to know the most effective Web search engines, since such search engines help their users find higher number of relevant Web pages with less effort. Furthermore, this information can be used for several practical purposes. This study evaluates the performance of three Web search engines. A set of measurements is proposed for evaluating Web search engine performance

Ribonucleic acid (RNA) virus and coronavirus in Google Dataset Search: their scope and epidemiological correlation

This paper presents an analysis of the publication of datasets collected via Google Dataset Search, specialized in families of RNA viruses, whose terminology was obtained from the National Cancer Institute (NCI) thesaurus developed by the US Department of Health and Human Services. The objective is to determine the scope and reuse capacity of the available data, determine the number of datasets and their free access, the proportion in reusable download formats, the main providers, their publication chronology, and to verify their scientific provenance. On the other hand, we also define possible relationships between the publication of datasets and the main pandemics that have occurred during the last 10 years. The results obtained highlight that only 52% of the datasets are related to scientific research, while an even smaller fraction (15%) are reusable. There is also an upward trend in the publication of datasets, especially related to the impact of the main epidemics, as clearly confirmed for the Ebola virus, Zika, SARS-CoV, H1N1, H1N5, and especially the SARS-CoV-2 coronavirus. Finally, it is observed that the search engine has not yet implemented adequate methods for filtering and monitoring the datasets. These results reveal some of the difficulties facing open science in the dataset field

Discovering semantic aspects of socially constructed knowledge hierarchy to boost the relevance of Web searching

The research intends to boost the relevance of Web search results by classifyingWebsnippet into socially constructed hierarchical search concepts, such as the mostcomprehensive human edited knowledge structure, the Open Directory Project (ODP). Thesemantic aspects of the search concepts (categories) in the socially constructed hierarchicalknowledge repositories are extracted from the associated textual information contributed bysocieties. The textual information is explored and analyzed to construct a category-documentset, which is subsequently employed to represent the semantics of the socially constructedsearch concepts. Simple API for XML (SAX), a component of JAXP (Java API for XMLProcessing) is utilized to read in and analyze the two RDF format ODP data files, structure.rdfand content.rdf. kNN, which is trained by the constructed category-document set, is used tocategorized the Web search results. The categorized Web search results are then ontologicallyfiltered based on the interactions of Web information seekers. Initial experimental resultsdemonstrate that the proposed approach can improve precision by 23.5%

A Coherent Measurement of Web-Search Relevance

We present a metric for quantitatively assessing the quality of Web searches. The relevance-of-searching-on-target index measures how relevant a search result is with respect to the searcher\u27s interest and intention. The measurement is established on the basis of the cognitive characteristics of common user\u27s online Web-browsing behavior and processes. We evaluated the accuracy of the index function with respect to a set of surveys conducted on several groups of our college students. While the index is primarily intended to be used to compare the Web-search results and tell which is more relevant, it can be extended to other applications. For example, it can be used to evaluate the techniques that people apply to improve the Web-search quality (including the quality of search engines), as well as other factors such as the expressiveness of search queries and the effectiveness of result-filtering processes

Learning to rank

Abstract. New general purpose ranking functions are discovered using genetic programming. The TREC WSJ collection was chosen as a training set. A baseline comparison function was chosen as the best of inner product, probability, cosine, and Okapi BM25. An elitist genetic algorithm with a population size 100 was run 13 times for 100 generations and the best performing algorithms chosen from these. The best learned functions, when evaluated against the best baseline function (BM25), demonstrate some significant performance differences, with improvements in mean average precision as high as 32% observed on one TREC collection not used in training. In no test is BM25 shown to significantly outperform the best learned function

A Theory and Practice of Website Engagibility

This thesis explores the domain of website quality. It presents a new study of website quality - an abstraction and synthesis, a measurement methodology, and analysis - and proposes metrics which can be used to quantify it. The strategy employed involved revisiting software quality, modelling its broader perspectives and identifying quality factors which are specific to the World Wide Web (WWW). This resulted in a detailed set of elements which constitute website quality, a method for quantifying a quality measure, and demonstrating an approach to benchmarking eCommerce websites. The thesis has two dimensions. The first is a contribution to the theory of software quality - specifically website quality. The second dimension focuses on two perspectives of website quality - quality-of-product and quality-of-use - and uses them to present a new theory and methodology which are important first steps towards understanding metrics and their use when quantifying website quality. Once quantified, the websites can be benchmarked by evaluators and website owners for comparison with competitor sites. The thesis presents a study of five mature eCommerce websites. The study involves identifying, defining and collecting data counts for 67 site-level criteria for each site. These counts are specific to website product quality and include criteria such as occurrences of hyperlinks and menus which underpin navigation, occurrences of activities which underpin interactivity, and counts relating to a site’s eCommerce maturity. Lack of automated count collecting tools necessitated online visits to 537 HTML pages and performing manual counts. The thesis formulates a new approach to measuring website quality, named Metric Ratio Analysis (MRA). The thesis demonstrates how one website quality factor - engagibility - can be quantified and used for website comparison analysis. The thesis proposes a detailed theoretical and empirical validation procedure for MRA