Beyond ranking: integrating user interaction and implicit feedback in optimising search engine result pages

The process of searching for information through an information retrieval (IR) system is intrinsically interactive, involving users in a series of actions such as formulating queries and evaluating numerous result snippets and their corresponding documents to ascertain their relevance. These interactions, which can be regarded as costs, underscore a notable deficiency in conventional IR systems. The Probability Ranking Principle (PRP) states that ranking documents in decreasing order of relevance with respect to the user’s query is the optimal way to maximise their expected utility from the results. However, the PRP fails to account for the nuances of result presentation and the inherent costs associated with interacting with the search engine results page (SERP). Acknowledging the diversity in users’ preferences, which can range from closely aligned to significantly divergent, poses a challenge in optimising the display of relevant results on a page to accommodate these varied inclinations. Different users will prefer distinct types of result presentations, and the layout of the result pages significantly influences their ability to interact with the system, discover relevant information, and, consequently, their overall satisfaction. This variability necessitates a nuanced approach to designing IR systems, one that goes beyond traditional ranking methods to consider the individualised ways users engage with and perceive the utility of search results. To address these challenges, the interactive probability ranking principle (iPRP), implemented via the Card Model offers a robust theoretical framework within the interactive IR space, enabling us to model the user interaction process while accounting for constraints such as presentation and screen space. By incorporating users’ implicit feedback, it is possible to assess these costs and preferences towards certain result items. This assessment can then inform the calculation of the Expected Perceived Utility (EPU), thus offering a more nuanced understanding of user interaction with IR systems. This thesis builds upon the card model, by expanding it to estimate the costs associated with user interactions in terms of time and to re-rank search engine results pages (SERPs), thereby raising three compelling questions; (1) how does the operationalisation of EPU affect result ranking, (2) what is the relationship between system-sided and user-side search costs and lastly (2) how do we optimise the presentation and what is its impact on user satisfaction. Each of these questions is explored through user studies centred on a news search task, designed to understand user preferences, presentation effects, and optimisation strategies. Our first study operationalises the notion of EPU and examines the impact of different result presentation formats, revealing how presentation significantly influences user perception through metrics such as time spent and clicks, and how we can re-rank results beyond traditional ranking paradigms like the PRP. We find that changing the presentation of results significantly impacts system-side metrics such as DCG, RBO and TBG. In our second study, we uncover the dynamics of interaction costs, result presentation, and user satisfaction at both the query and session levels. Our findings indicate that while at the query level, user satisfaction is predominantly influenced by performance metrics such as nDCG rather than presentation, at the session level, satisfaction emerges from a complex interplay of factors, delineating a non-linear relationship with the presentation. In the concluding study, we propose and evaluate a novel optimisation technique that synchronises ranking with presentation, tailored to individual user preferences. Although presentation optimisations lead to several behavioural changes in user interactions, they do not consistently align with user-reported satisfaction metrics, highlighting a subtle yet crucial gap between objective system enhancements and subjective user experience. In this thesis, we operationalise and empirically validate the iPRP. The findings from this thesis advocate for a shift in the design of future interactive IR systems, emphasising the need to personalise and dynamically display search results to enhance user experience. This research not only lays the foundational work for further exploration but also paves the way for validating the universal applicability of EPU-based result ranking across various interactive IR platforms and user demographics, thereby setting the stage for future studies in this vital area.The process of searching for information through an information retrieval (IR) system is intrinsically interactive, involving users in a series of actions such as formulating queries and evaluating numerous result snippets and their corresponding documents to ascertain their relevance. These interactions, which can be regarded as costs, underscore a notable deficiency in conventional IR systems. The Probability Ranking Principle (PRP) states that ranking documents in decreasing order of relevance with respect to the user’s query is the optimal way to maximise their expected utility from the results. However, the PRP fails to account for the nuances of result presentation and the inherent costs associated with interacting with the search engine results page (SERP). Acknowledging the diversity in users’ preferences, which can range from closely aligned to significantly divergent, poses a challenge in optimising the display of relevant results on a page to accommodate these varied inclinations. Different users will prefer distinct types of result presentations, and the layout of the result pages significantly influences their ability to interact with the system, discover relevant information, and, consequently, their overall satisfaction. This variability necessitates a nuanced approach to designing IR systems, one that goes beyond traditional ranking methods to consider the individualised ways users engage with and perceive the utility of search results. To address these challenges, the interactive probability ranking principle (iPRP), implemented via the Card Model offers a robust theoretical framework within the interactive IR space, enabling us to model the user interaction process while accounting for constraints such as presentation and screen space. By incorporating users’ implicit feedback, it is possible to assess these costs and preferences towards certain result items. This assessment can then inform the calculation of the Expected Perceived Utility (EPU), thus offering a more nuanced understanding of user interaction with IR systems. This thesis builds upon the card model, by expanding it to estimate the costs associated with user interactions in terms of time and to re-rank search engine results pages (SERPs), thereby raising three compelling questions; (1) how does the operationalisation of EPU affect result ranking, (2) what is the relationship between system-sided and user-side search costs and lastly (2) how do we optimise the presentation and what is its impact on user satisfaction. Each of these questions is explored through user studies centred on a news search task, designed to understand user preferences, presentation effects, and optimisation strategies. Our first study operationalises the notion of EPU and examines the impact of different result presentation formats, revealing how presentation significantly influences user perception through metrics such as time spent and clicks, and how we can re-rank results beyond traditional ranking paradigms like the PRP. We find that changing the presentation of results significantly impacts system-side metrics such as DCG, RBO and TBG. In our second study, we uncover the dynamics of interaction costs, result presentation, and user satisfaction at both the query and session levels. Our findings indicate that while at the query level, user satisfaction is predominantly influenced by performance metrics such as nDCG rather than presentation, at the session level, satisfaction emerges from a complex interplay of factors, delineating a non-linear relationship with the presentation. In the concluding study, we propose and evaluate a novel optimisation technique that synchronises ranking with presentation, tailored to individual user preferences. Although presentation optimisations lead to several behavioural changes in user interactions, they do not consistently align with user-reported satisfaction metrics, highlighting a subtle yet crucial gap between objective system enhancements and subjective user experience. In this thesis, we operationalise and empirically validate the iPRP. The findings from this thesis advocate for a shift in the design of future interactive IR systems, emphasising the need to personalise and dynamically display search results to enhance user experience. This research not only lays the foundational work for further exploration but also paves the way for validating the universal applicability of EPU-based result ranking across various interactive IR platforms and user demographics, thereby setting the stage for future studies in this vital area

The Influence of Presentation and Performance on User Satisfaction

The effectiveness of an IR system is gauged not just by its ability to retrieve relevant results but also by how it presents these results to users; an engaging presentation often correlates with increased user satisfaction. While existing research has delved into the link between user satisfaction, IR performance metrics, and presentation, these aspects have typically been investigated in isolation. Our research aims to bridge this gap by examining the relationship between query performance, presentation and user satisfaction. For our analysis, we conducted a between-subjects experiment comparing the effectiveness of various result card layouts for an ad-hoc news search interface. Drawing data from the TREC WaPo 2018 collection, we centered our study on four specific topics. Within each of these topics, we assessed six distinct queries with varying nDCG values. Our study involved 164 participants who were exposed to one of five distinct layouts containing result cards, such as "title'', "title+image'', or "title+image+summary''. Our findings indicate that while nDCG is a strong predictor of user satisfaction at the query level, there exists no linear relationship between the performance of the query, presentation of results and user satisfaction. However, when considering the total gain on the initial result page, we observed that presentation does play a significant role in user satisfaction (at the query level) for certain layouts with result cards such as, title+image or title+image+summary. Our results also suggest that the layout differences have complex and multifaceted impacts on satisfaction. We demonstrate the capacity to equalize user satisfaction levels between queries of varying performance by changing how results are presented. This emphasizes the necessity to harmonize both performance and presentation in IR systems, considering users' diverse preferences.Comment: To appear as a full paper at CHIIR 2024, Sheffield, U

Ranking Heterogeneous Search Result Pages using the Interactive Probability Ranking Principle

The Probability Ranking Principle (PRP) ranks search results based on their expected utility derived solely from document contents, often overlooking the nuances of presentation and user interaction. However, with the evolution of Search Engine Result Pages (SERPs), now comprising a variety of result cards, the manner in which these results are presented is pivotal in influencing user engagement and satisfaction. This shift prompts the question: How does the PRP and its user-centric counterpart, the Interactive Probability Ranking Principle (iPRP), compare in the context of these heterogeneous SERPs? Our study draws a comparison between the PRP and the iPRP, revealing significant differences in their output. The iPRP, accounting for item-specific costs and interaction probabilities to determine the ``Expected Perceived Utility" (EPU), yields different result orderings compared to the PRP. We evaluate the effect of the EPU on the ordering of results by observing changes in the ranking within a heterogeneous SERP compared to the traditional ``ten blue links''. We find that changing the presentation affects the ranking of items according to the (iPRP) by up to 48\% (with respect to DCG, TBG and RBO) in ad-hoc search tasks on the TREC WaPo Collection. This work suggests that the iPRP should be employed when ranking heterogeneous SERPs to provide a user-centric ranking that adapts the ordering based on the presentation and user engagement.Comment: To be presented as a full paper at ECIR 2024 in Glasgow, U

The influence of presentation and performance on user satisfaction

Information Retrieval (IR) systems are designed to provide users with a ranked list of results based on their queries. The effectiveness of an IR system is gauged not just by its ability to retrieve relevant results but also by how it presents these results to users; an engaging presentation often correlates with increased user satisfaction. While existing research has delved into the link between user satisfaction, IR performance metrics, and presentation, these aspects have typically been investigated in isolation. Our research aims to bridge this gap by examining the relationship between query performance, presentation and user satisfaction. For our analysis, we conducted a between-subjects experiment comparing the effectiveness of various result card layouts for an ad-hoc news search interface. Drawing data from the TREC WaPo 2018 collection, we centered our study on four specific topics. Within each of these topics, we assessed six distinct queries with varying nDCG values. Our study involved 164 participants who were exposed to one of five distinct layouts containing result cards, such as “title”, “title+image”, or “title+image+summary”. Our findings indicate that while nDCG is a strong predictor of user satisfaction at the query level, there exists no linear relationship between the performance of the query, presentation of results and user satisfaction. However, when considering the total gain on the initial result page, we observed that presentation does play a significant role in user satisfaction (at the query level) for certain layouts with result cards such as, title+image or title+image+summary. Our results also suggest that the layout differences have complex and multifaceted impacts on satisfaction. We demonstrate the capacity to equalize user satisfaction levels between queries of varying performance by changing how results are presented. This emphasizes the necessity to harmonize both performance and presentation in IR systems, considering users’ diverse preferences. Ultimately, our insights can steer the evolution of more user-aligned IR systems, underscoring the balance between system performance and result presentation

The influence of presentation and performance on user satisfaction

The effectiveness of an IR system is gauged not just by its ability to retrieve relevant results but also by how it presents these results to users; an engaging presentation often correlates with increased user satisfaction. While existing research has delved into the link between user satisfaction, IR performance metrics, and presentation, these aspects have typically been investigated in isolation. Our research aims to bridge this gap by examining the relationship between query performance, presentation and user satisfaction. For our analysis, we conducted a between-subjects experiment comparing the effectiveness of various result card layouts for an ad-hoc news search interface. Drawing data from the TREC WaPo 2018 collection, we centered our study on four specific topics. Within each of these topics, we assessed six distinct queries with varying nDCG values. Our study involved 164 participants who were exposed to one of five distinct layouts containing result cards, such as "title'', "title+image'', or "title+image+summary''. Our findings indicate that while nDCG is a strong predictor of user satisfaction at the query level, there exists no linear relationship between the performance of the query, presentation of results and user satisfaction. However, when considering the total gain on the initial result page, we observed that presentation does play a significant role in user satisfaction (at the query level) for certain layouts with result cards such as, title+image or title+image+summary. Our results also suggest that the layout differences have complex and multifaceted impacts on satisfaction. We demonstrate the capacity to equalize user satisfaction levels between queries of varying performance by changing how results are presented. This emphasizes the necessity to harmonize both performance and presentation in IR systems, considering users' diverse preferences

Ranking heterogeneous search result pages using the interactive Probability Ranking Principle

The Probability Ranking Principle (PRP) ranks search results based on their expected utility derived solely from document contents, often overlooking the nuances of presentation and user interaction. However, with the evolution of Search Engine Result Pages (SERPs), now comprising a variety of result cards, the manner in which these results are presented is pivotal in influencing user engagement and satisfaction. This shift prompts the question: How does the PRP and its user-centric counterpart, the Interactive Probability Ranking Principle (iPRP), compare in the context of these heterogeneous SERPs? Our study draws a comparison between the PRP and the iPRP, revealing significant differences in their output. The iPRP, accounting for item-specific costs and interaction probabilities to determine the ``Expected Perceived Utility" (EPU), yields different result orderings compared to the PRP. We evaluate the effect of the EPU on the ordering of results by observing changes in the ranking within a heterogeneous SERP compared to the traditional ``ten blue links''. We find that changing the presentation affects the ranking of items according to the (iPRP) by up to 48\% (with respect to DCG, TBG and RBO) in ad-hoc search tasks on the TREC WaPo Collection. This work suggests that the iPRP should be employed when ranking heterogeneous SERPs to provide a user-centric ranking that adapts the ordering based on the presentation and user engagement

Report on the Collab-a-thon at ECIR 2024

We present a report on the Collab-a-thon, a series of sessions at the European Conference on Information Retrieval (ECIR) 2024 designed to help foster new collaborations during a conference. This report presents the motivation and design of the Collab-a-thon, a summary of the discussions covered at each session, and a set of recommendations for conducting similar events in the future. The event is set to run again at ECIR 2025 and planning is underway to pilot the event in a different community at th

Report on the Collab-a-thon at ECIR 2024

We present a report on the Collab-a-thon, a series of sessions at the European Conference on Information Retrieval (ECIR) 2024 designed to help foster new collaborations during a conference. This report presents the motivation and design of the Collab-a-thon, a summary of the discussions covered at each session, and a set of recommendations for conducting similar events in the future. The event is set to run again at ECIR 2025 and planning is underway to pilot the event in a different community at th

Report on the Collab-a-thon at ECIR 2024

We present a report on the Collab-a-thon, a series of sessions at the European Conference on Information Retrieval (ECIR) 2024 designed to help foster new collaborations during a conference. This report presents the motivation and design of the Collab-a-thon, a summary of the discussions covered at each session, and a set of recommendations for conducting similar events in the future. The event is set to run again at ECIR 2025 and planning is underway to pilot the event in a different community at th