3,581 research outputs found
Evaluation of information retrieval systems using structural equation modeling
The interpretation of the experimental data collected by testing systems across input datasets and model parameters is of strategic importance for system design and implementation. In particular, finding relationships between variables and detecting the latent variables affecting retrieval performance can provide designers, engineers and experimenters with useful if not necessary information about how a system is performing. This paper discusses the use of Structural Equation Modeling (SEM) in providing an in-depth explanation of evaluation results and an explanation of failures and successes of a system; in particular, we focus on the case of evaluation of Information Retrieval systems
Predicting IR Personalization Performance using Pre-retrieval Query Predictors
Personalization generally improves the performance of queries but in a few cases it may also harms it. If we are able to predict and therefore to disable personalization for those situations, the overall performance will be higher and users will be more satisfied with personalized systems. We use some state-of-the-art pre-retrieval query performance predictors and propose some others including the user profile information for the previous purpose. We study the correlations among these predictors and the difference between the personalized and the original queries. We also use classification and regression techniques to improve the results and finally reach a bit more than one third of the maximum ideal performance. We think this is a good starting point within
this research line, which certainly needs more effort and improvements.This work has been supported by the Spanish Andalusian “Consejerı́a de Innovación, Ciencia y Empresa” postdoctoral phase of project P09-TIC-4526, the Spanish “Ministerio de Economı́a y Competitividad” projects TIN2013-42741-P and TIN2016-77902-C3-2-P, and the European Regional Development Fund (ERDF-FEDER)
Concept-based Interactive Query Expansion Support Tool (CIQUEST)
This report describes a three-year project (2000-03) undertaken in the Information Studies
Department at The University of Sheffield and funded by Resource, The Council for
Museums, Archives and Libraries. The overall aim of the research was to provide user
support for query formulation and reformulation in searching large-scale textual resources
including those of the World Wide Web. More specifically the objectives were: to investigate
and evaluate methods for the automatic generation and organisation of concepts derived from
retrieved document sets, based on statistical methods for term weighting; and to conduct
user-based evaluations on the understanding, presentation and retrieval effectiveness of
concept structures in selecting candidate terms for interactive query expansion.
The TREC test collection formed the basis for the seven evaluative experiments conducted in
the course of the project. These formed four distinct phases in the project plan. In the first
phase, a series of experiments was conducted to investigate further techniques for concept
derivation and hierarchical organisation and structure. The second phase was concerned with
user-based validation of the concept structures. Results of phases 1 and 2 informed on the
design of the test system and the user interface was developed in phase 3. The final phase
entailed a user-based summative evaluation of the CiQuest system.
The main findings demonstrate that concept hierarchies can effectively be generated from
sets of retrieved documents and displayed to searchers in a meaningful way. The approach
provides the searcher with an overview of the contents of the retrieved documents, which in
turn facilitates the viewing of documents and selection of the most relevant ones. Concept
hierarchies are a good source of terms for query expansion and can improve precision. The
extraction of descriptive phrases as an alternative source of terms was also effective. With
respect to presentation, cascading menus were easy to browse for selecting terms and for
viewing documents. In conclusion the project dissemination programme and future work are
outlined
Foundation Metrics: Quantifying Effectiveness of Healthcare Conversations powered by Generative AI
Generative Artificial Intelligence is set to revolutionize healthcare
delivery by transforming traditional patient care into a more personalized,
efficient, and proactive process. Chatbots, serving as interactive
conversational models, will probably drive this patient-centered transformation
in healthcare. Through the provision of various services, including diagnosis,
personalized lifestyle recommendations, and mental health support, the
objective is to substantially augment patient health outcomes, all the while
mitigating the workload burden on healthcare providers. The life-critical
nature of healthcare applications necessitates establishing a unified and
comprehensive set of evaluation metrics for conversational models. Existing
evaluation metrics proposed for various generic large language models (LLMs)
demonstrate a lack of comprehension regarding medical and health concepts and
their significance in promoting patients' well-being. Moreover, these metrics
neglect pivotal user-centered aspects, including trust-building, ethics,
personalization, empathy, user comprehension, and emotional support. The
purpose of this paper is to explore state-of-the-art LLM-based evaluation
metrics that are specifically applicable to the assessment of interactive
conversational models in healthcare. Subsequently, we present an comprehensive
set of evaluation metrics designed to thoroughly assess the performance of
healthcare chatbots from an end-user perspective. These metrics encompass an
evaluation of language processing abilities, impact on real-world clinical
tasks, and effectiveness in user-interactive conversations. Finally, we engage
in a discussion concerning the challenges associated with defining and
implementing these metrics, with particular emphasis on confounding factors
such as the target audience, evaluation methods, and prompt techniques involved
in the evaluation process.Comment: 13 pages, 4 figures, 2 tables, journal pape
Topic-independent modeling of user knowledge in informational search sessions
Web search is among the most frequent online activities. In this context, widespread informational queries entail user intentions to obtain knowledge with respect to a particular topic or domain. To serve learning needs better, recent research in the field of interactive information retrieval has advocated the importance of moving beyond relevance ranking of search results and considering a user’s knowledge state within learning oriented search sessions. Prior work has investigated the use of supervised models to predict a user’s knowledge gain and knowledge state from user interactions during a search session. However, the characteristics of the resources that a user interacts with have neither been sufficiently explored, nor exploited in this task. In this work, we introduce a novel set of resource-centric features and demonstrate their capacity to significantly improve supervised models for the task of predicting knowledge gain and knowledge state of users in Web search sessions. We make important contributions, given that reliable training data for such tasks is sparse and costly to obtain. We introduce various feature selection strategies geared towards selecting a limited subset of effective and generalizable features. © 2021, The Author(s)
GOOGLE TRENDS DATA AS A PROXY FOR INTEREST IN LEADERSHIP
The purpose of this quantitative study was to investigate the observable patterns of online search behavior in the topic of leadership using Google Trends data. Institutions have had a historically difficult time predicting good leadership candidates. Better predictions can be made by using the big data offered by groups such as Google to learn who, where, and when people are interested in leadership. The study utilized descriptive, comparative, and correlative methodologies to study Google users’ interest in leadership from 2004 to 2017. Society has placed great value into leadership throughout history, and though overall interest remains strong, it appears that the expression of that interest may have changed over time. Key findings revealed that interest in leadership often peaks during the spring and fall seasons while dipping during the summer and the winter holiday seasons. Leadership interest also appears to be more concentrated in geographic locations that home certain universities and political arenas
DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queries
In scientific research, the ability to effectively retrieve relevant
documents based on complex, multifaceted queries is critical. Existing
evaluation datasets for this task are limited, primarily due to the high cost
and effort required to annotate resources that effectively represent complex
queries. To address this, we propose a novel task, Scientific DOcument
Retrieval using Multi-level Aspect-based quEries (DORIS-MAE), which is designed
to handle the complex nature of user queries in scientific research. We
developed a benchmark dataset within the field of computer science, consisting
of 100 human-authored complex query cases. For each complex query, we assembled
a collection of 100 relevant documents and produced annotated relevance scores
for ranking them. Recognizing the significant labor of expert annotation, we
also introduce Anno-GPT, a scalable framework for validating the performance of
Large Language Models (LLMs) on expert-level dataset annotation tasks. LLM
annotation of the DORIS-MAE dataset resulted in a 500x reduction in cost,
without compromising quality. Furthermore, due to the multi-tiered structure of
these complex queries, the DORIS-MAE dataset can be extended to over 4,000
sub-query test cases without requiring additional annotation. We evaluated 17
recent retrieval methods on DORIS-MAE, observing notable performance drops
compared to traditional datasets. This highlights the need for better
approaches to handle complex, multifaceted queries in scientific research. Our
dataset and codebase are available at
https://github.com/Real-Doris-Mae/Doris-Mae-Dataset.Comment: To appear in NeurIPS 2023 Datasets and Benchmarks Trac
- …