Relevance thresholds in system evaluations

We introduce and explore the concept of an individual's relevance threshold as a way of reconciling differences in outcomes between batch and user experiments

User performance versus precision measures for simple search tasks

Several recent studies have demonstrated that the type of improvements in information retrieval system effectiveness reported in forums such as SIGIR and TREC do not translate into a benefit for users. Two of the studies used an instance recall task, and a third used a question answering task, so perhaps it is unsurprising that the precision based measures of IR system effectiveness on one-shot query evaluation do not correlate with user performance on these tasks. In this study, we evaluate two different information retrieval tasks on TREC Web-track data: a precision-based user task, measured by the length of time that users need to find a single document that is relevant to a TREC topic; and, a simple recall-based task, represented by the total number of relevant documents that users can identify within five minutes. Users employ search engines with controlled mean average precision (MAP) of between 55% and 95%. Our results show that there is no significant relationship between system effectiveness measured by MAP and the precision-based task. A significant, but weak relationship is present for the precision at one document returned metric. A weak relationship is present between MAP and the simple recall-based task

A self-regulatory approach to understanding boredom proneness

This is an Accepted Manuscript of an article published by Taylor & Francis in Cognition and Emotion on 2016-11-16, available online: http://dx.doi.org/10.1080/02699931.2015.1064363We investigated the relationship between self-regulation and two types of boredom proneness (perceived lack of internal stimulation, perceived lack of external stimulation) using a variety of measures of self-regulation. These included a general measure of self-control, measures of both regulatory focus (i.e., promotion or a sensitivity to gains/non-gains vs. prevention or a sensitivity to losses/non-losses) and regulatory mode (i.e., assessment or the tendency to compare means and goals vs. locomotion or the tendency to initiate and maintain commitment to action), and measures of cognitive flexibility (i.e., a perceived sense of control and the tendency to seek alternative solutions). Results identified a unique set of factors related to each boredom proneness component. Trait self-control and prevention focus were associated with lower boredom propensity due to a lack of external stimulation. Locomotion and the tendency to seek alternatives were associated with lower boredom propensity due to a lack of internal stimulation. These findings suggest that effective goal pursuit is associated with reduced likelihood of experiencing boredom.NSERC Discovery [grant no. 261628

Commitment to change from locomotion motivation during deliberation

The final publication is available at Springer via http://dx.doi.org/10.1007/s11031-011-9239-4The factors that motivate commitment to behavioral change (e.g., quitting smoking) are important in understanding self-regulation processes. The current research examines how an individual’s motivational orientation during deliberation affects the likelihood that they will commit to change. Building on the insights of regulatory mode theory (Higgins et al. in Advances in experimental social psychology. Academic Press, New York, vol 35, pp 293–344, 2003), we propose that increased commitment to change can result from increased locomotion motivation in the deliberation phase. Three studies provide evidence that increased commitment to change is related to locomotion motivation arising either from a chronic orientation or from a movement-focused deliberation tactic that intensifies that orientation. Although locomotion motivation is typically associated with goal pursuit, the current work highlights the impact that locomotion motivation can have on commitment to change in the initial deliberation phase.National Institute of Mental Health [Grant 39429

Language influences on tweeter geolocation

We investigate the influence of language on the accuracy of geolocating Twitter users. Our analysis, using a large corpus of tweets written in thirteen languages, provides a new understanding of the reasons behind reported performance disparities between languages. The results show that data imbalance has a greater impact on accuracy than geographical coverage. A comparison between micro and macro averaging demonstrates that existing evaluation approaches are less appropriate than previously thought. Our results suggest both averaging approaches should be used to effectively evaluate geolocation

Dodging Monsters and Dancing with Dreams: Success and Failure at Different Levels of Approach and Avoidance

To view the final version of this © The Authors, SAGE publication go here: http://dx.doi.org/10.1177/1754073913477506Many models of motivation suggest that goals can be arranged in a hierarchy, ranging from higher-level goals that represent desired end-states to lower-level means that operate in the service of those goals. We present a hierarchical model that distinguishes between three levels—goals, strategies, and tactics—and between approach/avoidance and regulatory focus motivations at different levels. We focus our discussion on how this hierarchical framework sheds light on the different ways that success and failure are defined within the promotion and prevention systems outlined in regulatory focus theory. Specifically, we review research that demonstrates that differences in what “counts” as success versus failure in these systems have important implications for motivational strength, emotional responses, and risky behavior

A comparative study of probabilistic and language models for information retrieval

Language models for information retrieval have received much attention in recent years, with many claims being made about their performance. However, previous studies evaluating the language modelling approach for information retrieval used different query sets and heterogeneous collections, which make reported results difficult to compare. This research is a broad-based study that evaluates language models against a variety of search tasks --- topic finding, named-page finding and topic distillation. The standard Text REtrieval Conference (TREC) methodology is used to compare language models to the probabilistic Okapi BM25 system. Using consistent parameter choices, we compare results of different language models on three different search tasks, multiple query sets and three different text collections. For ad hoc retrieval, the Dirichlet smoothing method was found to be significantly better than Okapi BM25, but for named-page finding Okapi BM25 was more effective than the language modelling methods. Optimal smoothing parameters for each method were found to be dependent on the collection and the query set. For longer queries, the language modelling approaches required more aggressive smoothing but they were found to be more effective than with shorter queries. The choice of smoothing method was also found to have a significant effect on the performance of language models for information retrieval

Answering English queries in automatically transcribed Arabic speech

There are several well-known approaches to parsing Arabic text in preparation for indexing and retrieval. Techniques such as stemming and stopping have been shown to improve search results on written newswire dispatches, but few comparisons are available on other data sources. In this paper, we apply several alternative stemming and stopping approaches to Arabic text automatically extracted from the audio soundtrack of news video footage, and compare these with approaches that rely on machine translation of the underlying text. Using the TRECVID video collection and queries, we show that normalisation, stopword- removal, and light stemming increase retrieval precision, but that heavy stemming and trigrams have a negative effect. We also show that the choice of machine translation engine plays a major role in retrieval effectiveness