Video_1_Training in Use of Brain–Machine Interface-Controlled Robotic Hand Improves Accuracy Decoding Two Types of Hand Movements.MOV

<p>Objective: Brain-machine interfaces (BMIs) are useful for inducing plastic changes in cortical representation. A BMI first decodes hand movements using cortical signals and then converts the decoded information into movements of a robotic hand. By using the BMI robotic hand, the cortical representation decoded by the BMI is modulated to improve decoding accuracy. We developed a BMI based on real-time magnetoencephalography (MEG) signals to control a robotic hand using decoded hand movements. Subjects were trained to use the BMI robotic hand freely for 10 min to evaluate plastic changes in the cortical representation due to the training.</p><p>Method: We trained nine young healthy subjects with normal motor function. In open-loop conditions, they were instructed to grasp or open their right hands during MEG recording. Time-averaged MEG signals were then used to train a real decoder to control the robotic arm in real time. Then, subjects were instructed to control the BMI-controlled robotic hand by moving their right hands for 10 min while watching the robot's movement. During this closed-loop session, subjects tried to improve their ability to control the robot. Finally, subjects performed the same offline task to compare cortical activities related to the hand movements. As a control, we used a random decoder trained by the MEG signals with shuffled movement labels. We performed the same experiments with the random decoder as a crossover trial. To evaluate the cortical representation, cortical currents were estimated using a source localization technique. Hand movements were also decoded by a support vector machine using the MEG signals during the offline task. The classification accuracy of the movements was compared among offline tasks.</p><p>Results: During the BMI training with the real decoder, the subjects succeeded in improving their accuracy in controlling the BMI robotic hand with correct rates of 0.28 ± 0.13 to 0.50 ± 0.11 (p = 0.017, n = 8, paired Student's t-test). Moreover, the classification accuracy of hand movements during the offline task was significantly increased after BMI training with the real decoder from 62.7 ± 6.5 to 70.0 ± 11.1% (p = 0.022, n = 8, t₍₇₎ = 2.93, paired Student's t-test), whereas accuracy did not significantly change after BMI training with the random decoder from 63.0 ± 8.8 to 66.4 ± 9.0% (p = 0.225, n = 8, t₍₇₎ = 1.33).</p><p>Conclusion: BMI training is a useful tool to train the cortical activity necessary for BMI control and to induce some plastic changes in the activity.</p

Color-map of the normalized ECoG signals and coordinates at the left wrist joint.

<p>Signals were obtained from channels 1∼30 in session 2 of patient 1(channels 31∼60 are not shown). This session includes 11 cycles. We treated each cycle as an independent trial.Start and end points were respectively defined as the instances where tangential velocity of the arm exceeded or fell below 5% of maximum velocity. Unused sampling points are colored yellow (yellow vertical lines). Precise wave forms of <i>z</i>-score on channel 27 inside of a red rectangle were already displayed in detail in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0072085#pone-0072085-g003" target="_blank">Figure 3</a>.</p

Prediction of Three-Dimensional Arm Trajectories Based on ECoG Signals Recorded from Human Sensorimotor Cortex

<div><p>Brain-machine interface techniques have been applied in a number of studies to control neuromotor prostheses and for neurorehabilitation in the hopes of providing a means to restore lost motor function. Electrocorticography (ECoG) has seen recent use in this regard because it offers a higher spatiotemporal resolution than non-invasive EEG and is less invasive than intracortical microelectrodes. Although several studies have already succeeded in the inference of computer cursor trajectories and finger flexions using human ECoG signals, precise three-dimensional (3D) trajectory reconstruction for a human limb from ECoG has not yet been achieved. In this study, we predicted 3D arm trajectories in time series from ECoG signals in humans using a novel preprocessing method and a sparse linear regression. Average Pearson’s correlation coefficients and normalized root-mean-square errors between predicted and actual trajectories were 0.44∼0.73 and 0.18∼0.42, respectively, confirming the feasibility of predicting 3D arm trajectories from ECoG. We foresee this method contributing to future advancements in neuroprosthesis and neurorehabilitation technology.</p></div

Examples of the predicted (red lines) and actual 3D trajectories (blue lines).

<p>A part of the 10th trial (6 s) in session 2 of patient 1 is shown (see <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0072085#pone.0072085.s002" target="_blank">Video S1</a>). Markers (circles, triangles, squares, and diamonds) represent 2 s time intervals. Circles and diamonds indicate the earliest and the latest positions, respectively. The red trajectories were computed using predicted data q1∼q4 and patient 1′s actual arm length. The timings (positions of the markers) and trajectory curves of the predicted data were similar to those of the actual data.</p

Accuracies for onset detection and classification of movement type.

<p>(A) The mean onset detection rate is shown from −2000 to 1000 ms with the standard deviation (<i>N</i> = 6). The N.D. (not detected) denotes the rate of trials in which no onset was detected from −2000 to 1000 ms (mean and standard deviation). Time 0 ms denotes target time to detect, which is the training time of the class decoder in training data sets. (B) Blue and red lines show the mean onset detection rates of contra-eSCP and ipsi-eSCP, respectively. The shaded area denotes the standard deviation (<i>N</i> = 6). (C) The green and red bars denote the average of the sensitivity and specificity of onset detection, respectively, and the error bars denote 95% confidence intervals. Asterisks show statistical significances (*<i>p</i> < 0.05, **<i>p</i> < 0.01, paired two-tailed Student’s <i>t</i>-test, <i>N</i> = 6) (D) The classification accuracies of movement type were compared among three types of features for decoding (**<i>p</i> < 0.01, paired two-tailed Student’s <i>t</i>-test, <i>N</i> = 6). The mean and 95% confidence interval are shown. Dotted line denotes chance level (50%).</p

Prediction results for all patients.

<p>Averaged correlation coefficients (CC) for joint angle (A) and <i>x</i>, <i>y</i>, <i>z</i> coordination (B), and the normalized root-mean-square error (nRMSE) for joint angles (C) and <i>x</i>, <i>y</i>, <i>z</i> coordination (D) were obtained using LOO-CV on 20, 19 and 73 trials for patients 1, 2, and 3 (blue, red, and green bars), respectively.</p

Contribution of each frequency band for trajectory prediction.

<p>Each panel (A: joint angles; B: <i>xyz</i> coordinates of the elbow; C: <i>xyz</i> coordinates of the wrist) shows the results of prediction using each sensorimotor rhythm, one by one. Noteworthy significant differences between CC values of frequency bands are marked with * (p<0.05) and ** (p<0.001). Other significance comparisons are omitted for visualization purposes.</p

Example of movement-type specific activation during the open-loop session.

<p>(A) The <i>z</i>-scored MEG signals (SMFs) averaged at the time of execution cue (0 ms) when subject 1 grasped or opened his right hand are color-coded at the location of the sensors. R, right; L, left. (B) The time course of mean SMFs at the sensor indicated by the black arrow in A. The shaded area shows the standard error. (C) The <i>z</i>-scored cortical potentials (eSCPs) averaged at 0 ms are color-coded on the normalized brain surface for each movement of subject 1. (D) The <i>F-</i>values of one-way ANOVA comparing eSCPs for the two types of movements shown in C are color-coded on the normalized brain surface only for values with <i>p</i> < 0.05.</p

Examples of predicted joint angles and positions in time series.

<p>Blue lines are actual recoded joint angles (left column), and actual positions at the left elbow (center column) and left wrist joint (right column) in the 10th trial shown in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0072085#pone-0072085-g003" target="_blank">Figure 3</a> and <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0072085#pone-0072085-g004" target="_blank">Figure 4</a>. The joint angles and coordinates predicted with sparse linear regression are plotted in red. The Pearson’s correlation coefficient (CC) and the normalized root-mean-square error (nRMSE) are shown at the top of each graph.</p

Electrodes placed on the sensorimotor cortex of patient 1.

<p>(A) Positions of the electrodes (circles). (B) Two 5 × 6 electrode arrays were placed on the right hemisphere, covering the sensorimotor cortex. Yellow lines depict the right central sulcus.</p