Remembered or Forgotten?-An EEG-Based Computational Prediction Approach.

Prediction of memory performance (remembered or forgotten) has various potential applications not only for knowledge learning but also for disease diagnosis. Recently, subsequent memory effects (SMEs)-the statistical differences in electroencephalography (EEG) signals before or during learning between subsequently remembered and forgotten events-have been found. This finding indicates that EEG signals convey the information relevant to memory performance. In this paper, based on SMEs we propose a computational approach to predict memory performance of an event from EEG signals. We devise a convolutional neural network for EEG, called ConvEEGNN, to predict subsequently remembered and forgotten events from EEG recorded during memory process. With the ConvEEGNN, prediction of memory performance can be achieved by integrating two main stages: feature extraction and classification. To verify the proposed approach, we employ an auditory memory task to collect EEG signals from scalp electrodes. For ConvEEGNN, the average prediction accuracy was 72.07% by using EEG data from pre-stimulus and during-stimulus periods, outperforming other approaches. It was observed that signals from pre-stimulus period and those from during-stimulus period had comparable contributions to memory performance. Furthermore, the connection weights of ConvEEGNN network can reveal prominent channels, which are consistent with the distribution of SME studied previously

Top-3 channels for pre-stimulus period, during-stimulus period and entire period.

<p>Top-3 channels for pre-stimulus period, during-stimulus period and entire period.</p

Timings of the auditory memory task in the study phase (A) and memory test (B).

<p>The two shaded areas in the study phase are the lasting time for an auditory cue and an auditory word respectively. The participants were instructed to make a semantic judgment about the word with the “animate or not” question showing on a screen. In the memory test, the two shade areas have the same meaning as those in the study phase. The participants made a judgment about the scale of familiarity by pressing a key from 1 to 5. The ConvEEGNN approach is designed to predict whether the participant remembered the word in the study phase by analyzing the EEG recorded from the study phase.</p

Prediction performance of various approaches.

<p>Prediction performance of various approaches.</p

Prediction accuracy using pre-stimulus, during-stimulus and entire period.

<p>Prediction accuracy using pre-stimulus, during-stimulus and entire period.</p

Grand-averaged ERP waveforms for remembered/forgotten words at a representative frontal electrode site (site Fp1 of the 10/10 system).

<p>Positive values are plotted upwards. (a) Pre-stimulus neural activity of auditory events. After a cue about an upcoming word, ERPs were elicited and analyzed by overlaid according to whether the word was remembered or forgotten. (b) During-stimulus neural activity of auditory events. After an auditory presented word, ERPs were elicited and analyzed by overlaid according to the judgments made in the memory test.</p

Prediction performance of different ConvEEGNN network structures.

<p>Prediction performance of different ConvEEGNN network structures.</p

Weight map averaged across all 9 participants. (a) pre-stimulus period, (b) during-stimulus period, (c) entire period.

<p>The map is range-scaled. The contour maps show the position of all the channels used. The bold dots represent the top 3 channels for corresponding period.</p

Detailed performances for all 9 participants using different approaches.

<p>The prediction accuracy and significance for ConvEEGNN are compared to: (1) LDA (2) ANN-1 (3) ANN-2 (4) SVM (5) SVM + LDA [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0167497#pone.0167497.ref020" target="_blank">20</a>] (6) CWT + SVM [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0167497#pone.0167497.ref021" target="_blank">21</a>]. The red bold line represents average prediction accuracy for each approach. The dots indicate the accuracies for every participant predicted by the approach next to it. Solid dot means that the prediction accuracy is significantly over chance otherwise soft dot is used.</p

Architecture of ConvEEGNN.

<p>Architecture of ConvEEGNN.</p