Modeling facial expressions and peripheral physiological signals to predict topical relevance

By analyzing explicit and implicit feedback information retrieval systems can determine topical relevance and tailor search criteria to the user's needs. In this paper we investigate whether it is possible to infer what is relevant by observing user affective behaviour. The sensory data employed range between facial expressions and peripheral physiological signals. We extract a set of features from the signals and analyze the data using classification methods, such as SVM and KNN. The results of our initial evaluation indicate that prediction of relevance is possible, to a certain extent, and implicit feedback models can benefit from taking into account user affective behavior

Engaged or Frustrated? Disambiguating Engagement and Frustration in Search

One of the primary ways researchers have characterized engagement is by an increase in search actions. Another possibility is that instead of experiencing increased engagement, people who click and query frequently are actually frustrated; several studies have shown that frustration is also characterized by increases in clicking and querying behaviors. This research seeks to illuminate the differences in search behavior between participants who are engaged and frustrated, as well as investigate the effect of task interest on engagement and frustration. To accomplish this, a laboratory experiment was conducted with 40 participants. Participants completed four tasks, and responded to questionnaires that measured their engagement, frustration, and stress. Participants were asked to rank eight topics based on interest, and were given their two most interesting and two least interesting tasks. Poor search result quality was introduced to induce frustration during their most interesting and least interesting tasks. This study found that physiological signals hold some promise for disambiguating engagement and frustration, but this depends on the time frame and manner in which they are examined. Frustrated participants had significantly more skin conductance responses during the task, while engaged participants had greater increases in skin conductance during the first 60 seconds of the task. Significant main and interaction effects for interest and frustration were found for heart rate in the window analysis, indicating that heart rate fluctuations over time can be most effective in distinguishing engagement from frustration. The multilevel modeling of engagement and frustration confirmed this, showing that interest contributed significantly to the model of skin conductance, while frustration contributed significantly to the model of heart rate. This study also found that interest had a significant effect on engagement, while the frustrator effectively created frustration. Frustration also had a significant effect on self-reported stress. Participants exhibited increases in search actions such as clicks and scrolls during periods of both engagement and frustration, but a regression analyses showed that scrolls, clicks on documents, and SERP clicks were most predictive of a frustrating episode. A significant main effect for interest was found for time between queries, indicating that this could be a useful signal of engagement. A model including the physiological signals and search behaviors showed that physiological signals aided in the prediction of engagement and frustration. Findings of this research have provided insight into the utility of physiological signals in distinguishing emotional states as well as provided evidence about the relationship among search actions, engagement and frustration. These findings have also increased our understanding of the role emotions play in search behavior and how information about a searcher’s emotional state can be used to improve the search experience.Doctor of Philosoph