68 research outputs found
On the Cost of Negation for Dynamic Pruning
Negated query terms allow documents containing such terms to be filtered out of a search results list, supporting disambiguation. In this work, the effect of negation on the efficiency of disjunctive, top-k retrieval is examined. First, we show how negation can be integrated efficiently into two popular dynamic pruning algorithms. Then, we explore the efficiency of our approach, and show that while often efficient, negation can negatively impact the dynamic pruning effectiveness for certain queries
An empirical analysis of pruning techniques performance, retrievability and bias
Prior work on using retrievability measures in the evaluation of information retrieval (IR) systems has laid out the foundations for investigating the relation between retrieval performance and retrieval bias. While various factors influencing retrievability have been examined, showing how the retrieval model may influence bias, no prior work has examined the impact of the index (and how it is optimized) on retrieval bias. Intuitively, how the documents are represented, and what terms they contain, will influence whether they are retrievable or not. In this paper, we investigate how the retrieval bias of a system changes as the inverted index is optimized for efficiency through static index pruning. In our analysis, we consider four pruning methods and examine how they affect performance and bias on the TREC GOV2 Collection. Our results show that the relationship between these factors is varied and complex-and very much dependent on the pruning algorithm. We find that more pruning results in relatively little change or a slight decrease in bias up to a point, and then a dramatic increase. The increase in bias corresponds to a sharp decrease in early precision such as NDCG@10 and is also indicative of a large decrease in MAP. The findings suggest that the impact of pruning algorithms can be quite varied-but retrieval bias could be used to guide the pruning process. Further work is required to determine precisely which documents are most affected and how this impacts upon performance
Examining the Impact of Uncontrolled Variables on Physiological Signals in User Studies for Information Processing Activities
Physiological signals can potentially be applied as objective measures to
understand the behavior and engagement of users interacting with information
access systems. However, the signals are highly sensitive, and many controls
are required in laboratory user studies. To investigate the extent to which
controlled or uncontrolled (i.e., confounding) variables such as task sequence
or duration influence the observed signals, we conducted a pilot study where
each participant completed four types of information-processing activities
(READ, LISTEN, SPEAK, and WRITE). Meanwhile, we collected data on blood volume
pulse, electrodermal activity, and pupil responses. We then used machine
learning approaches as a mechanism to examine the influence of controlled and
uncontrolled variables that commonly arise in user studies. Task duration was
found to have a substantial effect on the model performance, suggesting it
represents individual differences rather than giving insight into the target
variables. This work contributes to our understanding of such variables in
using physiological signals in information retrieval user studies.Comment: Accepted to the 46th International ACM SIGIR Conference on Research
and Development in Information Retrieval (SIGIR '23
Jury’s achter Lely’s meesterwerk
In dit rapport schetsen we allereerst een beeld van het proces, vanaf de voorbereiding van het voorstel om met burgerjury’s te werken tot en met de doorwerking van de adviezen in de besluitvorming over het Omgevingsplan door Provinciale Staten. We geven een korte beschrijving van de feitelijke gang van zaken, met daarbij de beelden die hierover in de interviews zijn verwoord door burgers, politici en projectmedewerkers. Vervolgens bespreken we een aantal belangrijke factoren die het proces hebben beïnvloed, namelijk de taakverdeling tussen GS en PS ten aanzien van de communicatie met de samenleving, de formulering van de vraagstellingen, de status van
de burgerjuryadviezen en verschillende factoren die de doorwerking van de adviezen hebben beïnvloed. De beschrijving van het proces vormt ook de basis voor het beantwoorden van de evaluatievragen die ons door de griffie zijn meegegeven. Daarmee komen we toe aan het beantwoorden van de vraag: was de burgerjury een juiste vorm voor burgerparticipatie over het Omgevingsplan en was de koppeling van de burgerjury aan Provinciale Staten een goede keuze
User Variability and IR System Evaluation
ABSTRACT Test collection design eliminates sources of user variability to make statistical comparisons among information retrieval (IR) systems more affordable. Does this choice unnecessarily limit generalizability of the outcomes to real usage scenarios? We explore two aspects of user variability with regard to evaluating the relative performance of IR systems, assessing effectiveness in the context of a subset of topics from three TREC collections, with the embodied information needs categorized against three levels of increasing task complexity. First, we explore the impact of widely differing queries that searchers construct for the same information need description. By executing those queries, we demonstrate that query formulation is critical to query effectiveness. The results also show that the range of scores characterizing effectiveness for a single system arising from these queries is comparable or greater than the range of scores arising from variation among systems using only a single query per topic. Second, our experiments reveal that searchers display substantial individual variation in the numbers of documents and queries they anticipate needing to issue, and there are underlying significant differences in these numbers in line with increasing task complexity levels. Our conclusion is that test collection design would be improved by the use of multiple query variations per topic, and could be further improved by the use of metrics which are sensitive to the expected numbers of useful documents
Designing and Evaluating Presentation Strategies for Fact-Checked Content
With the rapid growth of online misinformation, it is crucial to have
reliable fact-checking methods. Recent research on finding check-worthy claims
and automated fact-checking have made significant advancements. However,
limited guidance exists regarding the presentation of fact-checked content to
effectively convey verified information to users. We address this research gap
by exploring the critical design elements in fact-checking reports and
investigating whether credibility and presentation-based design improvements
can enhance users' ability to interpret the report accurately. We co-developed
potential content presentation strategies through a workshop involving
fact-checking professionals, communication experts, and researchers. The
workshop examined the significance and utility of elements such as veracity
indicators and explored the feasibility of incorporating interactive components
for enhanced information disclosure. Building on the workshop outcomes, we
conducted an online experiment involving 76 crowd workers to assess the
efficacy of different design strategies. The results indicate that proposed
strategies significantly improve users' ability to accurately interpret the
verdict of fact-checking articles. Our findings underscore the critical role of
effective presentation of fact reports in addressing the spread of
misinformation. By adopting appropriate design enhancements, the effectiveness
of fact-checking reports can be maximized, enabling users to make informed
judgments.Comment: Accepted to the 32nd ACM International Conference on Information and
Knowledge Management (CIKM '23
Boolean versus ranked querying for biomedical systematic reviews
Background: The process of constructing a systematic review, a document that compiles the published evidence pertaining to a specified medical topic, is intensely time-consuming, often taking a team of researchers over a year, with the identification of relevant published research comprising a substantial portion of the effort. The standard paradigm for this information-seeking task is to use Boolean search; however, this leaves the user(s) the requirement of examining every returned result. Further, our experience is that effective Boolean queries for this specific task are extremely difficult to formulate and typically require multiple iterations of refinement before being finalized. Methods: We explore the effectiveness of using ranked retrieval as compared to Boolean querying for the purpose of constructing a systematic review. We conduct a series of experiments involving ranked retrieval, using queries defined methodologically, in an effort to understand the practicalities of incorporating ranked retrieval into the systematic search task. Results: Our results show that ranked retrieval by itself is not viable for this search task requiring high recall. However, we describe a refinement of the standard Boolean search process and show that ranking within a Boolean result set can improve the overall search performance by providing early indication of the quality of the results, thereby speeding up the iterative query-refinement process. Conclusions: Outcomes of experiments suggest that an interactive query-development process using a hybrid ranked and Boolean retrieval system has the potential for significant time-savings over the current search process in the systematic reviewing
- …