Electromyography is the gold-standard among sensors for prosthetic control. However, stable and reliable

myocontrol remains an unsolved problem in the community. Amid improvements currently under investigation, one

focuses on alternative or complementary sensors. In this study, we compare different techniques, recording surface

and deep muscle activity. Ten subjects were involved in an experiment in which three different modalities were

attached on their forearm: force myography, electro-impedance tomography and ultrasound. They were asked to

perform wrist and grasp movements. For the first time, we evaluate and compare in an offline analysis these three

different modalities while recording several hand gestures

Claudio, Castellini

Connan, Mathilde

Gibas, Christian

Kirchner, Elsa Andrea

Rainer, Brück

Yu, Bingbin

Castellini, Claudio

Institute of Transport Research:Publications

DEEP AND SURFACE SENSOR MODALITIES FOR MYO-INTENT DETECTION  Mathilde Connan1, Bingbin Yu2,3, Christian Gibas4, Rainer Brück4,  Elsa Andrea Kirchner3,5, Claudio Castellini1,6 1German Aerospace Center (DLR), Institute of Robotics and Mechatronics, Wessling, Germany 2University of Bremen, Robotics Research Group, Bremen, Germany 3German Research Center for Artificial Intelligence (DFKI), Bremen, Germany 4University of Siegen, Medical Informatics and Microsystems Engineering, Siegen, Germany 5University of Duisburg, Institute of Medical Technology Systems, Duisburg, Germany 6Friedrich-Alexander University of Erlangen-Nuremberg, Artificial Intelligence in Biomedical Engineering, Erlangen, Germany  ABSTRACT Electromyography is the gold-standard among sensors for prosthetic control. However, stable and reliable myocontrol remains an unsolved problem in the community. Amid improvements currently under investigation, one focuses on alternative or complementary sensors. In this study, we compare different techniques, recording surface and deep muscle activity. Ten subjects were involved in an experiment in which three different modalities were attached on their forearm: force myography, electro-impedance tomography and ultrasound. They were asked to perform wrist and grasp movements. For the first time, we evaluate and compare in an offline analysis these three different modalities while recording several hand gestures.  INTRODUCTION Although surface electromyography (sEMG) has been used for decades in myoelectric control, it is subject to several drawbacks, such as sweat, electrode shift, muscular fatigue, or cross-talk among others [1]. Possible alternatives are being investigated in order to potentially replace or complement sEMG. For instance, in [2], it was shown that force myography (FMG), which is based on the deformation of the forearm due to muscular contractions, provides a more stable signal compared to electromyography. Although there are of course still some steps to go before integration, FMG showed a higher separateness of clusters and a higher accuracy when compared to sEMG. However, both sEMG and FMG are surface techniques, meaning that they record information mostly from surface muscles. Indeed, even though FMG indirectly contains some information about deeper muscle activity, these two techniques are still considered surface modalities. Deeper acquisition sources could therefore potentially provide important missing information. For example, ultrasound (US) imaging has already been evaluated for myocontrol with positive results for single finger movements [3]. Another deep sensing modality has also gained interest in recent years due to the search for alternatives to sEMG: Electrical Impedance Tomography (EIT). In medical EIT, particularly for myocontrol of the hand, a certain number of electrodes are placed around the forearm and a micro non-invasive alternating current is applied to one of them while the others measure bioimpedance. This process is repeated by applying the current in each electrode, in turn, until completion of the circle. The collected measurements can be reconstructed into a tomographic image using a back-projection algorithm [4]. The technique has already shown that it can be used to discriminate different hand movements offline with good classification accuracy using a support vector machine algorithm [5] EIT has also been integrated in an armband together with sEMG [6]. In this paper, we describe an experiment comparing FMG, US and EIT and discuss their potential for myocontrol applications. Ten subjects were fitted with the three modalities simultaneously and asked to perform wrist and grasp movements. To the best of our knowledge, this is the first time these three modalities have been compared in an experiment. The results show that US is always within the first two best performing algorithms for each hand/wrist gesture.   MATERIALS AND METHODS Experimental setup The EIT system used in this experiment was developed by the University of Siegen [7]. It consists of 16 electrodes placed around the forearm and provides 256 raw values. Its output frequency was approximately 2.7Hz. Post-filtering and processing to reconstruct the tomographic image were performed using the EIDORS library [4]. FMG data were collected using a custom-made armband with a Velcro strap and 10 force sensitive resistors developed at DLR [2]. The data were filtered with a second-order Butterworth low-pass filter with a cut-off frequency of 1Hz and saved at 94.2Hz. FMG was preferred over EMG due to the potential interference with the EIT system, which would have injected microcurrents into the same area where the EMG sensors would have recorded electrical muscle activity. The newly developed portable US system was developed by the Fraunhofer IBMT [8] and is one of the smallest systems available for ultrasound imaging, as the probe is flat and circular, unlike the normally bulky probes of medical systems. The 1161px by 162px B-mode displayed image was streamed into our software and stored at a frequency of approximately 5.1Hz. A bird’s eye view of the experiment can be seen in Figure 1(A).        Figure 1: (A) Bird’s eye view of the experiment. (B) Zoom on the modalities. The three devices were placed in the following order from proximal to distal: EIT, FMG and US, as shown in Figure 1(B). Both the EIT and FMG systems were sending the data to our Interactive Myocontrol software via Bluetooth while the US device sent its data via USB. Subjects and experimental protocol Ten people (8 men, 2 women, 32.5 +/- 6.3 years old) took part in this experiment. Half of the subjects wore the sensors on their left arm, and the other half on their right arm. A sequence consisted of eight actions: rest, power, point, precision (tridigital), wrist flexion, wrist extension, wrist supination and wrist pronation. After a familiarisation phase, three repetitions of the sequence followed. All subjects signed an inform consent form and the experiment was previously approved by the DLR Work Ethical Committee. The subjects sat in front of a table and placed their elbow on the table so that the arm was in line with the shoulder and the forearm formed an angle of about 90° to the upper arm, with the palm facing the side of the body. A 3D hand model on a screen indicated which hand movement had to be performed. For each action 2 seconds of data were recorded, with up and down phases lasting 1 second each and a further 2 seconds of non-captured rest between each action. For each subject and each repetition, the order of the actions was randomized. Data analysis The machine learning used for the analysis was Ridge Regression (RR), as it has the advantage with high-dimensional (HD) data that combined movements are possible without having to be trained, as was the case with HD-FMG [9]. The hyperparameters were evaluated for each subject using cross-validation. For each modality, different feature selection methods were chosen for comparison. Each modality was rescaled between 0 and 1. The 255-feature vector of EIT data was compared with restored images from different reconstruction algorithms of the EIDORS library. The first two methods use a basic solver using unfiltered back-projection. This is one of the simplest algorithms for reconstructing an EIT image. The second method uses additional artificially generated data compared to the first one. The third method is the Gauss-Newton approach with one-step iteration: it is the most commonly used for back projection in clinical and experimental publications. US data cannot be processed directly in the RR algorithm due to its size and must be reduced by feature selection algorithms. The first feature selection algorithm for US is Region of Interest Gradient (ROI-G) [10]. It has already been used successfully in experiments on US [11] and HD-FMG [9]. After some preliminary tests, a square of 40px with a step size of 30px was chosen as ROI. The other feature selection algorithms consisted of rescaling the matrices to a smaller size by selecting one row in n with n in {14, 20, 25, 30} as the different step sizes. For all methods, the features were amplified by a factor of 10 and filtered through a second-order Butterworth low-pass filter with a cut-off frequency of 1Hz. In order to compare the accuracy of the individual feature selection method, the normalized root mean square error (nRMSE) was calculated. This was averaged across all subjects, using the first two repetitions as the training samples and the last repetition as the test set. We also performed a comparative analysis of cluster separability for each modality. For each subject and each cluster pair (𝐶𝑖, 𝐶𝑗), we evaluated a numerical index called the Safety Index [11], which indicates how separated two clusters are in a given input space. The Safety Separateness Index of the clusters was calculated as follows: it is the ratio between the maximum standard deviation of cluster 𝐶𝑖 (evaluated over all dimensions) and the Euclidean distance between cluster Ci and 𝐶𝑗, 𝑠𝑖𝑗 =𝑚𝑎𝑥(𝜎𝑖)||𝐶𝑖−𝐶𝑗|| where σi is the standard deviation of cluster 𝐶𝑖 and 𝐶 is the mean of cluster 𝐶. In addition, the average number of principal components to reach 99% of the variance of the input space was calculated across all subjects.  RESULTS The nRMSE of RR for each modality was calculated action-wise as shown in Figure 2. The results were evaluated statistically across all actions using Friedman test for non-parametric data, showing that the nRMSE was statistically significantly different across the different modalities X2(9) = 40.8, p < 0.0001, with a moderate effect size W=0.453. The post-hoc Wilcoxon paired test could not conclude between which methods after the Holm correction for multiple comparisons.  Figure 2: nRMSE on the three modalities and their respective feature selections: FMG in green, EIT in red, US in purple. The modalities are sorted in an increasing order of the median of nRMSE (best performing first) for each action. The number of principal components (PCs) for reaching 99% of the variance of the input space was calculated for each subject and averaged for each modality in Table 1. This needs to be compared with the actual input space, i.e. the number of features, of each modality, which is also reported in the table. Table 1: Number of features for each method. Average and standard deviation (SD) of the number of principal components in order to reach 99% of the variance. Separateness index (mean and SD) indicating the separateness of the clusters (the lower the better).  Modality EIT EIT_GN EIT_lin EIT_lin_art FMG US_red14 US_red20 US_red25 US_red30 US_ROIG Number of features 256 740 740 740 10 996 531 329 234 348 Number of PCs (mean) 14.3 3.6 4.1 3.6 6.3 25.7 27.2 24.6 23 51.7 Number of PCs (SD) 2.9 0.7 1 0.7 0.8 6.4 6.5 5.8 5.5 7.4 Separateness index (mean) 0.495 0.098 0.089 0.093 0.181 0.067 0.1 0.111 0.126 0.143 Separateness index (SD) 0.341 0.034 0.025 0.03 0.05 0.015 0.039 0.025 0.029 0.023  The safety index to estimate the separateness of the clusters was evaluated and averaged across all subjects for each modality. Non-reconstructed EIT shows the lowest separability, while US rescaled every 14 rows shows the highest one. DISCUSSION  Figure 2 shows that, despite no clear method standing out from the others, at least one feature selection algorithm from US is always in the top two according to nRMSE. FMG, with its 10 features, surprisingly performs better than the other algorithms for wrist extension and wrist pronation. The basic solver EIT_lin_art is the best performing for wrist supination. Comparing the number of principal components necessary to reach 99% of the input space, the US-related number of components are the highest ones, but generally exhibits a good cluster separateness index, especially the US_red14, which has the best separateness. Non-reconstructed EIT has the worst overall Safety Index. However, this could be explained by electrodes with high impedance that would negatively affect other measurements and that are filtered in the reconstruction algorithms. The EIT reconstruction algorithms have surprisingly a low number of PCs and better cluster separability than most of US methods. However, the fact that the number of PCs is lower than the number of gestures to be controlled could explain the generally lower nRMSE results compared to US and possibly indicate that important data is lost during reconstruction. This could also be due to the lower sampling rate of EIT. Several feature selection algorithms were tested here on the different modalities. Unfortunately, none of them has yet been able to clearly outperform the others, with some modalities performing better than others for some movements. This might indicate that sensor fusion could be the ultimate solution. However, US was among the best performing methods according to nRMSE and to some extent cluster separateness index. Further feature selection algorithms should be evaluated to confirm this indication. In addition, the three modalities should be evaluated online and on amputees. However, due to the limited length of the stumps, it might be necessary to reduce the number of modalities to the two best performing ones.  ACKNOWLEDGEMENTS We would like to thank all the subjects who participated in this experiment. This work was partially supported by the German Research Society project Deep-Hand (DFG Sachbeihilfe CA-1389/1-2).  REFERENCES  [1]  J. R. Cram and G. S. Kasman, “Cram's Introduction To Surface Electromyography,” 2nd editio ed., Jones and Bartlett Publishers, 2010, p. 1–163. [2]  M. Connan, E. R. Ramı́rez, B. Vodermayer and C. Castellini, “Assessment of a Wearable Force- and Electromyography Device and Comparison of the Related Signals for Myocontrol,” Frontiers in Neurorobotics, vol. 10, p. 1–13, November 2016.  [3]  M. Sagardia, K. Hertkorn, D. Sierra González and C. Castellini, “Ultrapiano: A Novel Human-Machine Interface Applied to Virtual Reality,” in Proceedings of {ICRA} - International Conference on Robotics and Automation, 2014.  [4]  A. Adler and W. R. B. Lionheart, “Uses and abuses of EIDORS: an extensible software base for EIT,” Physiological measurement, vol. 27, p. S25, 2006.  [5]  Y. Zhang and C. Harrison, “Tomo: Wearable, low-cost electrical impedance tomography for hand gesture recognition,” in Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, 2015.  [6]  A. Briko, V. Kapravchuk, A. Kobelev, A. Hammoud, S. Leonhardt, C. Ngo, Y. Gulyaev and S. Shchukin, “A Way of Bionic Control Based on EI, EMG, and FMG Signals,” Sensors, vol. 22, p. 152, 2022.  [7]  C. Gibas, A. Grünewald, S. Büchner and R. Brück, “An EIT system for mobile medical diagnostics,” in Medical Imaging 2018: Biomedical Applications in Molecular, Structural, and Functional Imaging, 2018.  [8]  M. Fournelle, T. Grün, D. Speicher, S. Weber, M. Yilmaz, D. Schoeb, A. Miernik, G. Reis, S. Tretbar and H. Hewener, “Portable Ultrasound Research System for Use in Automated Bladder Monitoring with Machine-Learning-Based Segmentation,” Sensors, vol. 21, p. 6481, 2021.  [9]  M. Connan, R. Kõiva and C. Castellini, “Online natural myocontrol of combined hand and wrist actions using tactile myography and the biomechanics of grasping,” Frontiers in Neurorobotics, vol. 14, p. 1–16, 2020.  [10]  R. M. Haralick and L. G. Shapiro, “Computer and Robot Vision, vol. 1, Addison-Welsey,” Reading, Mass, USA, 1992.  [11]  D. Sierra González and C. Castellini, “A realistic implementation of ultrasound imaging as a human-machine interface for upper-limb amputees,” Frontiers in Neurorobotics, vol. 7, p. 1–11, October 2013.  

Deep and Surface Sensor Modalities for Myo-intent Detection

https://elib.dlr.de/191142/1/Connan2022%20-%20Deep%20and%20Surface%20Sensor%20Modalities%20for%20Myo-intent%20Detection.pdf

Deep and Surface Sensor Modalities for Myo-intent Detection

Abstract

Similar works

Full text

Available Versions

Institute of Transport Research:Publications