Object Segmentation in Images using EEG Signals

This paper explores the potential of brain-computer interfaces in segmenting objects from images. Our approach is centered around designing an effective method for displaying the image parts to the users such that they generate measurable brain reactions. When an image region, specifically a block of pixels, is displayed we estimate the probability of the block containing the object of interest using a score based on EEG activity. After several such blocks are displayed, the resulting probability map is binarized and combined with the GrabCut algorithm to segment the image into object and background regions. This study shows that BCI and simple EEG analysis are useful in locating object boundaries in images.Comment: This is a preprint version prior to submission for peer-review of the paper accepted to the 22nd ACM International Conference on Multimedia (November 3-7, 2014, Orlando, Florida, USA) for the High Risk High Reward session. 10 page

Exploring EEG for Object Detection and Retrieval

This paper explores the potential for using Brain Computer Interfaces (BCI) as a relevance feedback mechanism in content-based image retrieval. We investigate if it is possible to capture useful EEG signals to detect if relevant objects are present in a dataset of realistic and complex images. We perform several experiments using a rapid serial visual presentation (RSVP) of images at different rates (5Hz and 10Hz) on 8 users with different degrees of familiarization with BCI and the dataset. We then use the feedback from the BCI and mouse-based interfaces to retrieve localized objects in a subset of TRECVid images. We show that it is indeed possible to detect such objects in complex images and, also, that users with previous knowledge on the dataset or experience with the RSVP outperform others. When the users have limited time to annotate the images (100 seconds in our experiments) both interfaces are comparable in performance. Comparing our best users in a retrieval task, we found that EEG-based relevance feedback outperforms mouse-based feedback. The realistic and complex image dataset differentiates our work from previous studies on EEG for image retrieval.Comment: This preprint is the full version of a short paper accepted in the ACM International Conference on Multimedia Retrieval (ICMR) 2015 (Shanghai, China

Act quickly, decide later: long latency visual processing underlies perceptual decisions but not reflexive behavior

Jolij J, Scholte H, Van Gaal S, Hodgson TL, Lamme VAF (2011) Act quickly, decide later: Long latency visual processing underlies perceptual decisions but not reflexive behavior. Journal of Cognitive Neuroscience 23(12), p 3734-3745

BRIAN (Brain image analysis): A toolkit for the analysis of multimodal brain datasets

The analysis of cognitive processes in man usually involves multiple examina­tion modalities which map different aspects of the brain. Among these proce­dures, at least one modality yielding anatomical information (i.e. MRI*) besidesone or more functional modalities (fMRI, PET, SPECT, EEG, MEG) are involved.Because these different examination methods yield complimentary informationabout the anatomical, metabolical and neurophysiological state of the brain, acombined data evaluation is highly desirable and will lead to results not achiev­able within one examination domain.Such studies are of importance in research (cognitive neuroscience) and ­ withan emphasis on pathological processes ­ in clinical disciplines like neurology,neurosurgery and psychiatry.We have developed a program package for the handling of image datasets(MRI, PET, SPECT, CCT) and signal datasets (EEG, MEG) which allows a com­bined analysis of these data sources in a four­dimensional coordinate space (x, y,z, and time)

Improving object segmentation by using EEG signals and rapid serial visual presentation

This paper extends our previous work on the potential of EEG-based brain computer interfaces to segment salient objects in images. The proposed system analyzes the Event Related Potentials (ERP) generated by the rapid serial visual presentation of windows on the image. The detection of the P300 signal allows estimating a saliency map of the image, which is used to seed a semi-supervised object segmentation algorithm. Thanks to the new contributions presented in this work, the average Jaccard index was improved from $0.47$ to $0.66$ when processed in our publicly available dataset of images, object masks and captured EEG signals. This work also studies alternative architectures to the original one, the impact of object occupation in each image window, and a more robust evaluation based on statistical analysis and a weighted F-score