A Ternary Hybrid EEG-NIRS Brain-Computer Interface for the Classification of Brain Activation Patterns during Mental Arithmetic, Motor Imagery, and Idle State

Supplementary information for the research articl

Data_Sheet_1_Novel hybrid visual stimuli incorporating periodic motions into conventional flickering or pattern-reversal visual stimuli for steady-state visual evoked potential-based brain-computer interfaces.pdf

In this study, we proposed a new type of hybrid visual stimuli for steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs), which incorporate various periodic motions into conventional flickering stimuli (FS) or pattern reversal stimuli (PRS). Furthermore, we investigated optimal periodic motions for each FS and PRS to enhance the performance of SSVEP-based BCIs. Periodic motions were implemented by changing the size of the stimulus according to four different temporal functions denoted by none, square, triangular, and sine, yielding a total of eight hybrid visual stimuli. Additionally, we developed the extended version of filter bank canonical correlation analysis (FBCCA), which is a state-of-the-art training-free classification algorithm for SSVEP-based BCIs, to enhance the classification accuracy for PRS-based hybrid visual stimuli. Twenty healthy individuals participated in the SSVEP-based BCI experiment to discriminate four visual stimuli with different frequencies. An average classification accuracy and information transfer rate (ITR) were evaluated to compare the performances of SSVEP-based BCIs for different hybrid visual stimuli. Additionally, the user's visual fatigue for each of the hybrid visual stimuli was also evaluated. As the result, for FS, the highest performances were reported when the periodic motion of the sine waveform was incorporated for all window sizes except for 3 s. For PRS, the periodic motion of the square waveform showed the highest classification accuracies for all tested window sizes. A significant statistical difference in the performance between the two best stimuli was not observed. The averaged fatigue scores were reported to be 5.3 ± 2.05 and 4.05 ± 1.28 for FS with sine-wave periodic motion and PRS with square-wave periodic motion, respectively. Consequently, our results demonstrated that FS with sine-wave periodic motion and PRS with square-wave periodic motion could effectively improve the BCI performances compared to conventional FS and PRS. In addition, thanks to its low visual fatigue, PRS with square-wave periodic motion can be regarded as the most appropriate visual stimulus for the long-term use of SSVEP-based BCIs, particularly for window sizes equal to or larger than 2 s.</p

Performance Enhancement of a Brain-Computer Interface using High-Density Multi-Distance NIRS

This study investigated the effectiveness of using high-density multi-distance source-detector (SD) separations in near-infrared spectroscopy (NIRS) for enhancing the performance of functional NIRS (fNIRS)-based brain-computer interface (BCI). The NIRS system that was used for the experiment is capable of measuring signals from four SD separations: 15, 21.2, 30 and 33.5 mm; this allowed the measurement of hemodynamic response alterations at various depths. Fifteen participants were asked to perform mental arithmetic and word chain tasks to induce task-related hemodynamic response variations or asked to stay relaxed to acquire a baseline signal. To evaluate the degree of BCI performance enhancement by the high-density channel configuration, the classification accuracy obtained using the typical low-density lattice SD arrangement was compared with that obtained using the high-density SD arrangement, while maintaining the SD separation at 30 mm. The analysis results demonstrated that the use of high-density channel configuration did not result in noticeable enhancement of classification accuracy; however, combining hemodynamic variations measured by two multi-distance SD separations resulted in significant enhancement of the overall classification accuracy. The results of this study indicate that the use of high-density multi-distance SD separations is likely to provide a new method of enhancing the performance of an fNIRS-BCI

Changes in network connectivity during motor imagery and execution

<div><p>Background</p><p>Recent studies of functional or effective connectivity in the brain have reported that motor-related brain regions were activated during motor execution and motor imagery, but the relationship between motor and cognitive areas has not yet been completely understood. The objectives of our study were to analyze the effective connectivity between motor and cognitive networks in order to define network dynamics during motor execution and motor imagery in healthy individuals. Second, we analyzed the differences in effective connectivity between correct and incorrect responses during motor execution and imagery using dynamic causal modeling (DCM) of electroencephalography (EEG) data.</p><p>Method</p><p>Twenty healthy subjects performed a sequence of finger tapping trials using either motor execution or motor imagery, and the performances were recorded. Changes in effective connectivity between the primary motor cortex (M1), supplementary motor area (SMA), premotor cortex (PMC), and dorsolateral prefrontal cortex (DLPFC) were estimated using dynamic causal modeling. Bayesian model averaging with family-level inference and fixed-effects analysis was applied to determine the most likely connectivity model for these regions.</p><p>Results</p><p>Motor execution and imagery showed inputs to distinct brain regions, the premotor cortex and the supplementary motor area, respectively. During motor execution, the coupling strength of a feedforward network from the DLPFC to the PMC was greater than that during motor imagery. During motor imagery, the coupling strengths of a feedforward network from the PMC to the SMA and of a feedback network from M1 to the PMC were higher than that during motor execution. In imagined movement, although there were connectivity differences between correct and incorrect task responses, each motor imagery task that included correct and incorrect responses showed similar network connectivity characteristics. Correct motor imagery responses showed connectivity from the PMC to the DLPFC, while the incorrect responses had characteristic connectivity from the SMA to the DLPFC.</p><p>Conclusions</p><p>These findings provide an understanding of effective connectivity between motor and cognitive areas during motor execution and imagery as well as the basis for future connectivity studies for patients with stroke.</p></div

Event-related spectral perturbation over the C3 and C4 electrodes and topography for the mu and beta bands in each trial.

<p>(A) Event-related spectral perturbation over the C3 and C4 electrodes during motor execution (ME) and motor imagery (MI), (B) Topography at the mu band in each trial, (C) Topography at beta bands in each trial.</p

Motor task block components.

<p>The starting finger was indicated with a red dot. The experimental paradigm was divided into two sessions of motor execution and imagery. Each session was performed individually. A white dot was presented in the middle of the monitor every 1300 ms as a signal to progress to the next finger, both for motor execution and imagery. Each block was composed of at least three trials. Subjects pressed the appropriate button at the end of the task block as directed.</p

Participant MIQ-RS score.

<p>Participant MIQ-RS score.</p

Coupling parameters from dynamic causal modeling.

<p>Coupling parameters from dynamic causal modeling.</p

Family-level analysis and Bayesian model selection (BMS).

<p>(A) Correct responses during motor execution (ME), (B) Incorrect response during ME, (C) Correct responses during motor imagery (MI), and (D) Incorrect responses during MI.</p

DCM coupling strength based on modulatory connectivity (DCM-B matrix).

<p>(A) Correct responses during motor execution (ME), (B) Incorrect response during ME, (C) Correct responses during motor imagery (MI), (D) Incorrect responses during MI, (E) Higher coupling strength during ME compared with correct MI, (F) Higher coupling strength for MI compared with correct ME, (G) Connectivity characteristic of correct MI responses compared with incorrect MI, and (H) Connectivity characteristics of incorrect MI responses compared with correct MI.</p