Effects of search intent on eye-movement patterns in a change detection task

The goal of the present study was to examine whether intention type affects eye movement patterns in a change detection task In addition, we assessed whether the eye movement index could be used to identify human implicit intent. We attempted to generate three types of intent amongst the study participants, dividing them into one of three conditions; each condition received different information regarding an impending change to the visual stimuli. In the “navigational intent” condition, participants were asked to look for any interesting objects, and were not given any more information about the impending change. In the “low-specific intent” condition, participants were informed that a change would occur. In the “high-specific intent” condition, participants were told that a change would occur, and that an object would disappear. In addition to this main change detection task, participants also had to perform a primary task, in which they were required to name aloud the colors of objects in the pre-change scene. This allowed us to control for the visual searching process during the pre-change scene. The main results were as follows: firstly, the primary task successfully controlled for the visual search process during the pre-change scene, establishing that there were no differences in the patterns of eye movements across all three conditions despite differing intents. Secondly, we observed significantly different patterns of eye movement between the conditions in the post-change scene, suggesting that generating a specific intent for change detection yields a distinctive pattern of eye-movements. Finally, discriminant function analysis showed a reasonable classification rate for identifying a specific intent. Taken together, it was found that both participant intent and the specificity of information provided to the participants affect eye movements in a change detection task

The Influence of Commercial Intent of Search Results on Their Perceived Relevance

We carried out a retrieval effectiveness test on the three major web search engines (i.e., Google, Microsoft and Yahoo). In addition to relevance judgments, we classified the results according to their commercial intent and whether or not they carried any advertising. We found that all search engines provide a large number of results with a commercial intent. Google provides significantly more commercial results than the other search engines do. However, the commercial intent of a result did not influence jurors in their relevance judgments

Search engine user behaviour: How can users be guided to quality content?

The typical behaviour of the Web search engine user is widely known: a user only types in one or a few keywords and expects the search engine to produce relevant results in an instant. Search engines not only adapt to this behaviour. On the contrary, they are often faced with criticism that they themselves created this kind of behaviour. As search engines are trendsetters for the whole information world, it is important to know how they cope with their users’ behaviour. Recent developments show that search engines try to integrate results from different collections into their results lists and to guide their users to the right results. These results should not only be relevant in general, but also be pertinent in the sense of being relevant to the user in his current situation and in accordance to his background. The article focuses on the problems of guiding the user from his initial query to these results. It shows how the general users are searching and how the intents behind their queries can be used to deliver the right results. It will be shown that search engines try to give some good results for everyone instead of focusing on complete result sets for a specific user type. If the user wishes, he can follow the paths laid out by the engines to narrow the results to a result set suitable to him

Variance Reduction in Gradient Exploration for Online Learning to Rank

Online Learning to Rank (OL2R) algorithms learn from implicit user feedback on the fly. The key of such algorithms is an unbiased estimation of gradients, which is often (trivially) achieved by uniformly sampling from the entire parameter space. This unfortunately introduces high-variance in gradient estimation, and leads to a worse regret of model estimation, especially when the dimension of parameter space is large. In this paper, we aim at reducing the variance of gradient estimation in OL2R algorithms. We project the selected updating direction into a space spanned by the feature vectors from examined documents under the current query (termed the "document space" for short), after interleaved test. Our key insight is that the result of interleaved test solely is governed by a user's relevance evaluation over the examined documents. Hence, the true gradient introduced by this test result should lie in the constructed document space, and components orthogonal to the document space in the proposed gradient can be safely removed for variance reduction. We prove that the projected gradient is an unbiased estimation of the true gradient, and show that this lower-variance gradient estimation results in significant regret reduction. Our proposed method is compatible with all existing OL2R algorithms which rank documents using a linear model. Extensive experimental comparisons with several state-of-the-art OL2R algorithms have confirmed the effectiveness of our proposed method in reducing the variance of gradient estimation and improving overall performance.Comment: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '19); Key Words: Online learning to rank, Dueling bandit, Variance Reductio

5 − 4 ≠ 4 − 3: On the Uneven Gaps between Different Levels of Graded User Satisfaction in Interactive Information Retrieval Evaluation

Similar to other ground truth measures, graded user satisfaction has been frequently employed as a continuous variable in information retrieval evaluation based on the assumption that intervals between adjacent grades are quantitatively equal. To examine the validity of equal-gap assumption and explore dynamic perceptual thresholds triggering grade changes in search evaluation, we investigate the extent to which users are sensitive to changes in search efforts and outcomes across different gaps of graded satisfaction. Experiments on four user study datasets (15,337 queries) indicate that 1) User satisfaction sensitivity, especially to offline evaluation metrics, changes significantly across gaps in satisfaction scale; 2) the size and direction of changes in sensitivity vary across study settings, search types, and intentions, especially within “3-5” scale subrange. This study speaks to the fundamentals of user-centered evaluation and advances the knowledge of heterogeneity in satisfaction sensitivity to search efforts and gains and implicit changes in evaluation thresholds