Classification of Eye Fixation Related Potentials for Variable Stimulus Saliency

Objective: Electroencephalography (EEG) and eye tracking can possibly provide information about which items displayed on the screen are relevant for a person. Exploiting this implicit information promises to enhance various software applications. The specific problem addressed by the present study is that items shown in real applications are typically diverse. Accordingly, the saliency of information, which allows to discriminate between relevant and irrelevant items, varies. As a consequence, recognition can happen in foveal or in peripheral vision, i.e., either before or after the saccade to the item. Accordingly, neural processes related to recognition are expected to occur with a variable latency with respect to the eye movements. The aim was to investigate if relevance estimation based on EEG and eye tracking data is possible despite of the aforementioned variability. Approach:Sixteen subjects performed a search task where the target saliency was varied while the EEG was recorded and the unrestrained eye movements were tracked. Based on the acquired data, it was estimated which of the items displayed were targets and which were distractors in the search task. Results: Target prediction was possible also when the stimulus saliencies were mixed. Information contained in EEG and eye tracking data was found to be complementary and neural signals were captured despite of the unrestricted eye movements. The classification algorithm was able to cope with the experimentally induced variable timing of neural activity related to target recognition. Significance: It was demonstrated how EEG and eye tracking data can provide implicit information about the relevance of items on the screen for potential use in online applications.EC/FP7/611570/EU/Symbiotic Mind Computer Interaction for Information Seeking/MindSeeBMBF, 01GQ0850, Bernstein Fokus Neurotechnologie - Nichtinvasive Neurotechnologie für Mensch-Maschine Interaktio

Implicit relevance feedback from electroencephalography and eye tracking in image search

Objective. Methods from brain–computer interfacing (BCI) open a direct access to the mental processes of computer users, which offers particular benefits in comparison to standard methods for inferring user-related information. The signals can be recorded unobtrusively in the background, which circumvents the time-consuming and distracting need for the users to give explicit feedback to questions concerning the individual interest. The obtained implicit information makes it possible to create dynamic user interest profiles in real-time, that can be taken into account by novel types of adaptive, personalised software. In the present study, the potential of implicit relevance feedback from electroencephalography (EEG) and eye tracking was explored with a demonstrator application that simulated an image search engine. Approach. The participants of the study queried for ambiguous search terms, having in mind one of the two possible interpretations of the respective term. Subsequently, they viewed different images arranged in a grid that were related to the query. The ambiguity of the underspecified search term was resolved with implicit information present in the recorded signals. For this purpose, feature vectors were extracted from the signals and used by multivariate classifiers that estimated the intended interpretation of the ambiguous query. Main result. The intended interpretation was inferred correctly from a combination of EEG and eye tracking signals in 86% of the cases on average. Information provided by the two measurement modalities turned out to be complementary. Significance. It was demonstrated that BCI methods can extract implicit user-related information in a setting of human-computer interaction. Novelties of the study are the implicit online feedback from EEG and eye tracking, the approximation to a realistic use case in a simulation, and the presentation of a large set of photographies that had to be interpreted with respect to the content.EC/FP7/611570/EU/Symbiotic Mind Computer Interaction for Information Seeking/MindSeeBMBF, 01GQ0850, Bernstein Fokus Neurotechnologie - Nichtinvasive Neurotechnologie für Mensch-Maschine Interaktio

Search Process as Transitions Between Neural States

Search is one of the most performed activities on the World Wide Web. Various conceptual models postulate that the search process can be broken down into distinct emotional and cognitive states of searchers while they engage in a search process. These models significantly contribute to our understanding of the search process. However, they are typically based on self-report measures, such as surveys, questionnaire, etc. and therefore, only indirectly monitor the brain activity that supports such a process. With this work, we take one step further and directly measure the brain activity involved in a search process. To do so, we break down a search process into five time periods: a realisation of Information Need, Query Formulation, Query Submission, Relevance Judgment and Satisfaction Judgment. We then investigate the brain activity between these time periods. Using functional Magnetic Resonance Imaging (fMRI), we monitored the brain activity of twenty-four participants during a search process that involved answering questions carefully selected from the TREC-8 and TREC 2001 Q/A Tracks. This novel analysis that focuses on transitions rather than states reveals the contrasting brain activity between time periods – which enables the identification of the distinct parts of the search process as the user moves through them. This work, therefore, provides an important first step in representing the search process based on the transitions between neural states. Discovering more precisely how brain activity relates to different parts of the search process will enable the development of brain-computer interactions that better support search and search interactions, which we believe our study and conclusions advance

Ranking algorithms for implicit feedback

This report presents novel algorithms to use eye movements as an implicit relevance feedback in order to improve the performance of the searches. The algorithms are evaluated on "Transport Rank Five" Dataset which were previously collected in Task 8.3. We demonstrated that simple linear combination or tensor product of eye movement and image features can improve the retrieval accuracy

Evaluating the retrieval effectiveness of Web search engines using a representative query sample

Search engine retrieval effectiveness studies are usually small-scale, using only limited query samples. Furthermore, queries are selected by the researchers. We address these issues by taking a random representative sample of 1,000 informational and 1,000 navigational queries from a major German search engine and comparing Google's and Bing's results based on this sample. Jurors were found through crowdsourcing, data was collected using specialised software, the Relevance Assessment Tool (RAT). We found that while Google outperforms Bing in both query types, the difference in the performance for informational queries was rather low. However, for navigational queries, Google found the correct answer in 95.3 per cent of cases whereas Bing only found the correct answer 76.6 per cent of the time. We conclude that search engine performance on navigational queries is of great importance, as users in this case can clearly identify queries that have returned correct results. So, performance on this query type may contribute to explaining user satisfaction with search engines