Hybrid brain computer interfaces (BCIs) combining multiple brain imaging modalities have been proposed recently to boost the performance of single modality BCIs. We advance the state of hybrid BCIs by introducing a novel system that measures electrical brain activity as well as cerebral blood flow velocity using Electroencephalography (EEG) and functional transcranial Doppler ultrasound (fTCD), respectively. The system we developed employs two different paradigms to induce changes simultaneously in EEG and fTCD and to infer user intent. One of these paradigms includes visual stimuli to simultaneously induce steady state visually evoked potentials (SSVEPs) and instructs users to perform word generation (WG) and mental rotation (MR) tasks, while the other paradigm instructs users to perform left and right arm motor imagery (MI) tasks through visual stimuli.
To improve accuracy and information transfer rate (ITR) of the proposed system compared to those obtained through our preliminary analysis, using classical feature extraction approaches, we mainly contribute to multi-modal fusion of EEG and fTCD features. Specifically, we proposed a probabilistic fusion of EEG and fTCD evidences instead of simple concatenation of EEG and fTCD feature vectors that we performed in our preliminary analysis. Experimental results showed that the MI paradigm outperformed the MR/WG one in terms of both accuracy and ITR. In particular, 93.85%, 93.71%, and 100% average accuracies and 19.89, 26.55, and 40.83 bits/min
v
average ITRs were achieved for right MI vs baseline, left MI versus baseline, and right MI versus left MI, respectively. Moreover, for both paradigms, the EEG-fTCD BCI with the proposed analysis techniques outperformed all EEG- fNIRS BCIs in terms of accuracy and ITR.
In addition, to investigate the feasibility of increasing the possible number of BCI commands, we extended our approaches to solve the 3-class problems for both paradigms. It was found that the MI paradigm outperformed the MR/WG paradigm and achieved 96.58% average accuracy and 45 bits/min average ITR. Finally, we introduced a transfer learning approach to reduce the calibration requirements of the proposed BCI. This approach was found to be very efficient especially with the MI paradigm as it reduced the calibration requirements by at least 60.43%
