Combining Eye Movements and Collaborative Filtering for Proactive Information Retrieval

We study a new task, proactive information retrieval by combining implicit relevance feedback and collaborative filtering. We have constructed a controlled experimental setting, a prototype application, in which the users try to find interesting scientific articles by browsing their titles. Implicit feedback is inferred from eye movement signals, with discriminative hidden Markov models estimated from existing data in which explicit relevance feedback is available. Collaborative filtering is carried out using the User Rating Profile model, a state-of-the-art probabilistic latent variable model, computed using Markov Chain Monte Carlo techniques. For new document titles the prediction accuracy with eye movements, collaborative filtering, and their combination was significantly better than by chance. The best prediction accuracy still leaves room for improvement but shows that proactive information retrieval and combination of many sources of relevance feedback is feasible

Investigating online reading with eye tracking and EEG : The influence of text format, reading task and parafoveal stimuli on reading processes

Research on reading has been successful in revealing how attention guides eye movements when people read single sentences or text paragraphs in simplified and strictly controlled experimental conditions. However, less is known about reading processes in more naturalistic and applied settings, such as reading Web pages. This thesis investigates online reading processes by recording participants eye movements. The thesis consists of four experimental studies that examine how location of stimuli presented outside the currently fixated region (Study I and III), text format (Study II), animation and abrupt onset of online advertisements (Study III), and phase of an online information search task (Study IV) affect written language processing. Furthermore, the studies investigate how the goal of the reading task affects attention allocation during reading by comparing reading for comprehension with free browsing, and by varying the difficulty of an information search task. The results show that text format affects the reading process, that is, vertical text (word/line) is read at a slower rate than a standard horizontal text, and the mean fixation durations are longer for vertical text than for horizontal text. Furthermore, animated online ads and abrupt ad onsets capture online readers attention and direct their gaze toward the ads, and distract the reading process. Compared to a reading-for-comprehension task, online ads are attended to more in a free browsing task. Moreover, in both tasks abrupt ad onsets result in rather immediate fixations toward the ads. This effect is enhanced when the ad is presented in the proximity of the text being read. In addition, the reading processes vary when Web users proceed in online information search tasks, for example when they are searching for a specific keyword, looking for an answer to a question, or trying to find a subjectively most interesting topic. A scanning type of behavior is typical at the beginning of the tasks, after which participants tend to switch to a more careful reading state before finishing the tasks in the states referred to as decision states. Furthermore, the results also provided evidence that left-to-right readers extract more parafoveal information to the right of the fixated word than to the left, suggesting that learning biases attentional orienting towards the reading direction.Aikaisemmissa lukututkimuksissa on selvitetty lukijan tarkkaavaisuuden ja katseen ohjautumista, kun ärsykkeinä on käytetty yksittäisiä lauseita tai lyhyitä tekstejä. Sen sijaan soveltavissa ympäristöissä, kuten Internetissä, katseen ohjautumista ja lukuprosesseja on tutkittu vähemmän. Tässä väitöskirjatyössä tutkittiin katseen ohjautumista, kun koehenkilöt suorittivat Internet ympäristölle tyypillisiä lukutehtäviä. Lisäksi ensimmäisessä osatyössä tutkittiin, miten opittu lukusuunta vaikuttaa tarkannäön (fovean) ulkopuolella esitetyn teksti-informaation prosessointiin. Osatyöt koostuivat kokeista, joissa tarkasteltiin tekstin esitystavan sekä tekstin ympärillä esitettyjen kuvien ja sanojen vaikutusta lukuprosessiin. Lisäksi tutkittiin lukuprosessin vaihtelua ajallisesti tehtävän edetessä tai tehtäväohjeen muuttuessa. Tulokset osoittivat, että kapeapalstaista (sana/rivi) tekstiä luettiin hitaammin kuin normaalia vaakasuorilla riveillä esitettyä tekstiä ja että katseen pysähdysten (fiksaatioiden) keskimääräiset kestot olivat pidempiä kapeapalstaista tekstiä luettaessa. Lisäksi havaittiin, että tekstin ympärillä esitetyt mainokset häiritsevät lukemista, sillä lukuprosessi keskeytyi lukijan katseen ohjautuessa mainokseen. Lisäksi mainosten ilmestymisaika ruudulle oli ajallisesti yhteydessä hetkeen, jolloin katse siirtyi kohti mainosta. Mainoksia katsottiin enemmän tilanteessa, jossa koehenkilöt saivat vapaasti tutustua sivustoon verrattuna vaativampaan lukutehtävään, jossa koehenkilöitä pyydettiin vastaamaan tekstin sisältöä koskeviin kysymyksiin. Tulokset osoittivat myös, että lukuprosessit vaihtelevat tiedonhakutehtävien eri vaiheissa, kun koehenkilöt etsivät lauselistojen joukosta joko tiettyä avainsanaa, vastausta kysymykseen tai henkilökohtaisesti kiinnostavaa aihetta. Tehtävän alussa koehenkilöiden silmänliikkeiden piirteet viittasivat silmäilyyn, jonka jälkeen koehenkilöt siirtyivät huolellisempaan lukemiseen. Huolellisen lukemisen jälkeen silmänliikkeet muuttuivat jälleen koehenkilöiden siirtyessä lukuprosessista päätöksenteko-prosessiin, johon tehtävät lopetettiin. Lisäksi tulokset osoittivat, että opitulla lukusuunnalla oli vaikutusta tarkkaavaisuuden ohjautumiseen, sillä vasemmalta oikealle lukevat koehenkilöt hahmottivat enemmän informaatiota keskellä esitetyn sanan oikealta puolelta vasempaan puoleen verrattuna

Eye Tracking: A Perceptual Interface for Content Based Image Retrieval

In this thesis visual search experiments are devised to explore the feasibility of an eye gaze driven search mechanism. The thesis first explores gaze behaviour on images possessing different levels of saliency. Eye behaviour was predominantly attracted by salient locations, but appears to also require frequent reference to non-salient background regions which indicated that information from scan paths might prove useful for image search. The thesis then specifically investigates the benefits of eye tracking as an image retrieval interface in terms of speed relative to selection by mouse, and in terms of the efficiency of eye tracking mechanisms in the task of retrieving target images. Results are analysed using ANOVA and significant findings are discussed. Results show that eye selection was faster than a computer mouse and experience gained during visual tasks carried out using a mouse would benefit users if they were subsequently transferred to an eye tracking system. Results on the image retrieval experiments show that users are able to navigate to a target image within a database confirming the feasibility of an eye gaze driven search mechanism. Additional histogram analysis of the fixations, saccades and pupil diameters in the human eye movement data revealed a new method of extracting intentions from gaze behaviour for image search, of which the user was not aware and promises even quicker search performances. The research has two implications for Content Based Image Retrieval: (i) improvements in query formulation for visual search and (ii) new methods for visual search using attentional weighting. Futhermore it was demonstrated that users are able to find target images at sufficient speeds indicating that pre-attentive activity is playing a role in visual search. A current review of eye tracking technology, current applications, visual perception research, and models of visual attention is discussed. A review of the potential of the technology for commercial exploitation is also presented

The design and study of pedagogical paper recommendation

For learners engaging in senior-level courses, tutors in many cases would like to pick some articles as supplementary reading materials for them each week. Unlike researchers ‘Googling’ papers from the Internet, tutors, when making recommendations, should consider course syllabus and their assessment of learners along many dimensions. As such, simply ‘Googling’ articles from the Internet is far from enough. That is, learner models of each individual, including their learning interest, knowledge, goals, etc. should be considered when making paper recommendations, since the recommendation should be carried out so as to ensure that the suitability of a paper for a learner is calculated as the summation of the fitness of the appropriateness of it to help the learner in general. This type of the recommendation is called a Pedagogical Paper Recommender.In this thesis, we propose a set of recommendation methods for a Pedagogical Paper Recommender and study the various important issues surrounding it. Experimental studies confirm that making recommendations to learners in social learning environments is not the same as making recommendation to users in commercial environments such as Amazon.com. In such learning environments, learners are willing to accept items that are not interesting, yet meet their learning goals in some way or another; learners’ overall impression towards each paper is not solely dependent on the interestingness of the paper, but also other factors, such as the degree to which the paper can help to meet their ‘cognitive’ goals.It is also observed that most of the recommendation methods are scalable. Although the degree of this scalability is still unclear, we conjecture that those methods are consistent to up to 50 papers in terms of recommendation accuracy. The experiments conducted so far and suggestions made on the adoption of recommendation methods are based on the data we have collected during one semester of a course. Therefore, the generality of results needs to undergo further validation before more certain conclusion can be drawn. These follow up studies should be performed (ideally) in more semesters on the same course or related courses with more newly added papers. Then, some open issues can be further investigated. Despite these weaknesses, this study has been able to reach the research goals set out in the proposed pedagogical paper recommender which, although sounding intuitive, unfortunately has been largely ignored in the research community. Finding a ‘good’ paper is not trivial: it is not about the simple fact that the user will either accept the recommended items, or not; rather, it is a multiple step process that typically entails the users navigating the paper collections, understanding the recommended items, seeing what others like/dislike, and making decisions. Therefore, a future research goal to proceed from the study here is to design for different kinds of social navigation in order to study their respective impacts on user behavior, and how over time, user behavior feeds back to influence the system performance

Analysis of Family-Health-Related Topics on Wikipedia

New concepts, terms, and topics always emerge; and meanings of existing terms and topics keep changing all the time. These phenomena occur more frequently on social media than on conventional media because social media allows a huge number of users to generate information online. Retrieving relevant results in different time periods of a fast-changing topic becomes one of the most difficult challenges in the information retrieval field. Among numerous topics discussed on social media, health-related topics are a major category which attracts increasing attention from the general public. This study investigated and explored the evolution patterns of family-health-related topics on Wikipedia. Three family-health-related topics (Child Maltreatment, Family Planning, and Women’s Health) were selected from the World Health Organization Website and their associated entries were retrieved on Wikipedia. Historical numeric and text data of the entries from 2010 to 2017 were collected from a Wikipedia data dump and the Wikipedia Web pages. Four periods were defined: 2010 to 2011, 2012 to 2013, 2014 to 2015, and 2016 to 2017. Coding, subject analysis, descriptive statistical analysis, inferential statistical analysis, SOM approach, and n-gram approach were employed to explore the internal characteristics and external popularity evolutions of the topics. The findings illustrate that the external popularities of the family-health-related topics declined from 2010 to 2017, although their content on Wikipedia kept increasing. The emerged entries had three features: specialization, summarization, and internationalization. The subjects derived from the entries became increasingly diverse during the investigated periods. Meanwhile, the developing trajectories of the subjects varied from one to another. According to the developing trajectories, the subjects were grouped into three categories: growing subject, diminishing subject, and fluctuating subject. The popularities of the topics among the Wikipedia viewers were consistent, while among the editors were not. For each topic, its popularity trend among the editors and the viewers was inconsistent. Child Maltreatment was the most popular among the three topics, Women’s Health was the second most popular, while Family Planning was the least popular among the three. The implications of this study include: (1) helping health professionals and general users get a more comprehensive understanding of the investigated topics; (2) contributing to the developments of health ontologies and consumer health vocabularies; (3) assisting Website designers in organizing online health information and helping them identify popular family-health-related topics; (4) providing a new approach for query recommendation in information retrieval systems; (5) supporting temporal information retrieval by presenting the temporal changes of family-health-related topics; and (6) providing a new combination of data collection and analysis methods for researchers

Interactive video retrieval using implicit user feedback.

PhDIn the recent years, the rapid development of digital technologies and the low cost of recording media have led to a great increase in the availability of multimedia content worldwide. This availability places the demand for the development of advanced search engines. Traditionally, manual annotation of video was one of the usual practices to support retrieval. However, the vast amounts of multimedia content make such practices very expensive in terms of human effort. At the same time, the availability of low cost wearable sensors delivers a plethora of user-machine interaction data. Therefore, there is an important challenge of exploiting implicit user feedback (such as user navigation patterns and eye movements) during interactive multimedia retrieval sessions with a view to improving video search engines. In this thesis, we focus on automatically annotating video content by exploiting aggregated implicit feedback of past users expressed as click-through data and gaze movements. Towards this goal, we have conducted interactive video retrieval experiments, in order to collect click-through and eye movement data in not strictly controlled environments. First, we generate semantic relations between the multimedia items by proposing a graph representation of aggregated past interaction data and exploit them to generate recommendations, as well as to improve content-based search. Then, we investigate the role of user gaze movements in interactive video retrieval and propose a methodology for inferring user interest by employing support vector machines and gaze movement-based features. Finally, we propose an automatic video annotation framework, which combines query clustering into topics by constructing gaze movement-driven random forests and temporally enhanced dominant sets, as well as video shot classification for predicting the relevance of viewed items with respect to a topic. The results show that exploiting heterogeneous implicit feedback from past users is of added value for future users of interactive video retrieval systems

Evaluating sources of implicit feedback for web search

This dissertation investigated several important issues in using implicit feedback techniques to assist searchers with difficulties in formulating effective search strategies. The study focused on examining the relationship between types of behavioral evidence that can be captured from Web searches and searchers’ interests. Web search cases which involved underspecification of information needs at the beginning and modification of search strategies during the search process were collected and reviewed by human analysts (reference librarians) who tried to infer searchers’ interests from behavioral traces. Analysts’ rationales for making the inferences were elicited and analyzed with the focus on understanding what evidence was used to support the inferences and how it was used. The analysis revealed the complexities and nuances in using behavioral evidence for implicit feedback and led to the proposal of an implicit feedback model for Web search that bridged previous studies on behavioral evidence and implicit feedback measures. A new level of analysis termed an analytical lens emerged from the data and provides a road map for future research on this topic. The study also put forward design recommendations for implicit feedback systems based on the signals that analysts identified and the rules that they used in making inferences