The article is devoted to the problem of recognition of motor imagery based on electroencephalogram (EEG) signals, which is associated with many difficulties, such as the physical and mental state of a person, measurement accuracy, etc. Artificial neural networks are a good tool in solving this class of problems. Electroencephalograms are time signals, Gramian Angular Fields (GAF) and Markov Transition Field (MTF) transformations are used to represent time series as images. The paper shows the possibility of using GAF and MTF EEG signal transforms for recognizing motor patterns, which is further applicable, for example, in building a brain-computer interface

Bragin, Aleksandr Dmitrievich

Spitsin, Vladislav Vladimirovich

English

Electronic archive of Tomsk Polytechnic University

 978-1-7281-2896-2 / 19 / $ 31.00 ©2019 IEEE Neural network analysis of electroencephalograms based on their graphical representation Aleksandr Bragin  Department of Information Technology National Research Tomsk Polytechnic University Tomsk, Russia lflenylol@gmail.com Vladimir Spitsyn Department of Information Technology National Research Tomsk Polytechnic University Tomsk, Russia spvg@tpu.ru  Abstract—The article is devoted to the problem of recognition of motor imagery based on electroencephalogram (EEG) signals, which is associated with many difficulties, such as the physical and mental state of a person, measurement accuracy, etc. Artificial neural networks are a good tool in solving this class of problems. Electroencephalograms are time signals, Gramian Angular Fields (GAF) and Markov Transition Field (MTF) transformations are used to represent time series as images. The paper shows the possibility of using GAF and MTF EEG signal transforms for recognizing motor patterns, which is further applicable, for example, in building a brain-computer interface. Keywords—motor imagery recognition, electroencephalogram, Gramian Angular Field, Markov Transition Field, Convolutional Neural Network  I. INTRODUCTION Electroencephalography is one of the most popular non-invasive methods for studying brain activity today. Electroencephalogram (EEG) signals show the total electrical activity of neurons in the cerebral cortex. You can get a lot of useful information about the human condition by studying these data. The study of EEG is associated with many difficulties, such as the dependence of signals on age, time of day, the presence of noise, interference, and weak structuring. Classical mathematical methods based on time-frequency, wave or component analysis can be used for EEG studies. However, often the use of these methods does not give stable results of human states recognition, and their application becomes extremely complicated due to the complexity of the algorithms [1–4]. Brain signals are very complex. Classical mathematical techniques (Fourier transform, wavelet analysis, etc.) are based on extracting the useful signal from the whole data array and algorithmic work with it. For signals recorded in difficult conditions of psychophysiological experiments, it is often difficult to isolate the useful signal, and the technique may stop working with the slightest change in state. The use of artificial neural network (ANN) in applied areas, such as the brain-computer interface, is a perspective direction today [5–9]. The ability of the ANN to adaptive learning, resistance to signal distortion and a good generalizing effect makes them an excellent tool for classification [10-12]. There are several approaches to the classification of time series using ANN [13]. A key factor in the success of human activities recognition using EEG is the effective use of data obtained from measurement sensors. In this paper, the method proposed in [14] is used. In this method, the time series is converted into images, after which the convolutional neural network is used to analyze them. This work is devoted to the creation of systems for the recognition of motor imagery (movement of the right and left hands) based on EEG signals. The processes associated with motor imagery are extremely complex, changes in the time-frequency structure of electroencephalograms are not systematic and vary for each person. The change in the EEG signal can be used to recognize motor patterns. II. TIME SERIES RECOGNITION Convolutional Neural Network (CNN) is a special architecture of artificial neural networks aimed at effective pattern recognition. This type of network uses some features of the visual cortex and responds to straight lines at different angles and their combinations. Thus, the idea of convolutional neural networks is the alternation of convolutional layers and pooling layers. The network structure is unidirectional (without feedback), essentially multi-layered. Convolutional neural networks are an effective tool in pattern recognition tasks. It is widely used in video and image processing tasks, natural language processing systems [15]. The use of such networks helps to reduce the number of parameters of the model being taught, to reduce hardware costs in the process of learning and working. However, such a network does not work directly with time series. In [14], a method was proposed for using convolutional neural networks to classify time series. In this method, the time series is converted into images in two ways, after which the usual convolutional neural network is used to image. A. GAF transform To classify EEG signals using convolutional neural networks, the GAF (Gramian Angular Field) method was used [16, 17].  In this method, the time series is converted into a polar coordinate system. Based on the data obtained, a matrix G is constructed. Each element of the matrix is equal to the cosine of the sum of the angles. The resulting matrix is converted into an image, which is fed to the input of the convolutional neural network. Preservation of time dependence is provided with this conversion. The main diagonal is a special case with k = 0, containing the original values and angular information. The GAF matrix is a matrix built on the series as follows: 1. First, the series is normalized to the segment [−1, 1], (1): )min()max())min(())max((ˆXXXixXixix−−+−=  (1) 2. Next, the resulting values are translated to the polar coordinate system: Information & Communication Technologies302==Nit)arccos(irixiϕ  (2) 3. The GAF matrix is calculated by the following expression:  ++++++=)cos()1cos()2cos()12cos()1cos()11cos(nnnnnGϕϕϕϕϕϕϕϕϕϕϕϕ  (3) The final matrix saves all the information about the series, except for the initial bounds of the values that we lose in step (1) after the normalization procedure - that is, we can restore the original series using the obtained matrix but only scaled to the segment [−1, 1]. Images for further use are formed on the basis of the obtained matrices. Since the color channel in this case does not carry useful information, binary images are used in the work, which made it possible to reduce the number of image channels by 3 times in comparison with the RGB version. Binary GAF image is shown in Fig. 1. B. MTF transform Another method of converting the original time series into images is the MTF (Markov Transition Field) method. Unlike GAF, the original bounds of the series and the distribution of the values of the series that are lost when applying convolutions and pooling to GAF are stored in the MTF matrix. The idea of this method is to consistently represent the probabilities of Markov transitions to store information in the time dimension. The first step is the quantization of time series to build a first-order Markov matrix. The Markov matrix, which includes Markov dynamics, rejects the conditional connection between the distribution of X and the time dependence on the time steps ti. MTF extends the Markov matrix to equalize each probability in a temporal order.   The data conversion order: 1. The whole set of observation values is divided into m quantile bins — segments with the same probability that the observation value falls into each of them. This can be done simply with the help of a training sample - to combine all the values of a variable into one set, sort it, and then arrange m - 1 boundaries in it so that there is approximately the same number of values between two adjacent boundaries. The space between two adjacent borders will be a quantile bin. 2. Let ɷij be the number of pairs of neighboring observations in the time series, which we are considering - such that the left observation lies in the bin i and the right observation in the bin j. ijωˆ  is a normalized ɷij such that ji ij∀= ,1ωˆ . Then the MTF matrix for a time series is a matrix of transition probabilities between bins:  =mmmmmGωωωωωωˆ1ˆ2ˆ21ˆ1ˆ11ˆ   Example of MTF image generated from EEG data is shown in Fig. 2. III. DEEP CONVOLUTIONAL NEURAL NETWORK ARCHITECTURE The architecture of the deep convolutional neural network (CNN) adapted for the classification of the EEG signal according to the number of measuring electrodes was developed in this paper. Input images are fed to the network input in the form of a 64 channel image, where each channel is a transformed electroencephalogram signal. The main layers of the deep CNN are three convolutional and three fully connected layers. Network parameters were determined experimentally. IV. DATASET The data presented in [18] were selected for the study. Each subject was in a chair with armrests and watched the image on the monitor. At the beginning of each test, a black screen with a fixing cross was displayed on the monitor for two seconds, and then the subject had to imagine a hand movement depending on the instructions on the monitor for next three seconds. Then followed a short break for a few seconds, after which the action was repeated. The data set is EEG signals recorded using the BCI 2000 system [19] and using 64 electrodes at a sampling frequency of 512 Hz. Frequency filters for data conversion were not used. The order of the experiment and the transformation of the original data into GAF images are shown in Fig.3. Data from the first subject were used for the training of neural networks.   Fig. 1. GAF image example  Fig. 2. MTF image example  Information & Communication Technologies303 Fig. 3. The order of the experiment  Fig. 4. Model structure V. MODEL STRUCTURE The signals received from 64 electrodes in the electroencephalograms under study are presented, so we have the corresponding number of images as input parameters of the model. The images are converted by analogy with the RGB image into the input vector, however, if in the case of the RGB image 3 channels are used, one for each color, in this case a 64 channel image containing the converted EEG signal is input. The block diagram of the final model is presented in Fig. 4. VI. RECOGNITION OF MOTOR IMAGERY  For the recognition of motor imagery were used data of the first test subject [18]. At the input of the neural network were fed the image size of 128 by 128 pixels. The recognition accuracy for the GAF method was about 97%, for the MTF method – 93%, recognition accuracy from the data source [18] is equal to 80%. The full EEG signal (in the case of the GAF transformation, normalized in the [-1.1] range) was used to generate images, without removing any artifacts, as well as highlighting characteristic rhythms and patterns. This approach avoids the need to filter the EEG signal for further investigation, which may result in the loss of useful information.  VII. CONCLUSION  The article showed the possibility of using the GAF and MTF transformations methods for the detection of motor imagery in EEG signals. It is shown that without the use of additional filtering of the initial data (selection of brain rhythms) it is possible to achieve high results. The accuracy of recognition of motor imagery of the movement of the right and left hand, as well as the state of rest for the studied EEG signals was 97% (GAF) and 93% (MTF). General results indicate that these methods provide higher accuracy than the methods described in the data source. The GAF method showed better results in the recognition of motor imagery, at the same time the MTF conversion allows you to restore the original signal, after processing, for further study. The discussed methods of electroencephalogram classification can later be used to build a brain-computer interface. ACKNOWLEDGMENT The reported study was funded by RFBR according to the research project № 18-08-00977 А. REFERENCES [1] G. van Luijtelaar, A. Lüttjohann, V.V. Makarov, V.A. Maksimenko, A.A. Koronovskii, A.E. Hramov, “Methods of automated absence seizure detection, interference bystimulation, and possibilities for prediction in genetic absence models”, Journal of Neuroscience Methods. 2016. vol. 260, p. 144. [2] A.A. Koronovskii, A.E. Hramov, V.V. Grubov, O.I. Moskalenko, E.Y. Sitnikova, A.N. Pavlov, “Coexistence of intermittencies in the neuronal network of the epileptic brain”, Phys. Rev. E. 2016. V. 93. doi: 10.1103/PhysRevE.93.032220 [3] V.V. Grubov, A.E. Runnova, M.K. Kurovskaуa, A.N. Pavlov, A.A. Koronovskii, A.E. Hramov, “Demonstration of brain noise on human EEG signals in perception of bistable images”, Proc. SPIE. 2016. V. 9707. DOI: 10.1117/12.2207390 [4] A.E. Hramov, A.A. Koronovskii, V.A. Makarov, A.N. Pavlov, E.Y. Sitnikova, “Wavelets in neuroscience”, Heidelberg; New York; Dordrecht; London, Springer, 2015, p. 318. Information & Communication Technologies304[5] P. Sotnikov, K. Finagin, S. Vidunova, “Selection of optimal frequency bands of the electroencephalogram signal in eye-brain-computer interface”, Procedia Computer Science. 2017, vol. 103, pp. 168-175. [6] A.N. Vasilyev, S.P. Liburkina, A.Y. Kaplan, “Lateralization of EEG patterns in humans during motor imagery of arm movements in the brain-computer interface”, Zhurnal Vysshei Nervnoi Deyatelnosti Imeni I.P. Pavlova, 2016, vol. 66, № 3, pp. 302-312. [7] V.A. Maksimenko, S. Heukelum, V.V. Makarov, J. Kelderhuis, A. Lüttjohann, A.A. Koronovskii, A.E. Hramov, G. Luijtelaar, “Absence seizure control by a brain computer interface”, Scientific Reports. 2017, vol. 7, p. 2487. [8] W. Hsu, I. Chiang, “Application of neural network to brain-computer interface”, 2012 IEEE International Conference on Granular Computing, Hangzhou, 2012, pp. 163-168. doi: 10.1109/GrC.2012.6468559 [9] K. Nakayama, K. Inagaki, “A brain computer interface based on neural network with efficient pre-processing”, 2006 International Symposium on Intelligent Signal Processing and Communications, Tottori, 2006, pp. 673-676. doi: 10.1109/ISPACS.2006.364745 [10] R. Östberg, “Robustness of a neural network used for image classification: The effect of applying distortions on adversarial examples”, Dissertation, 2018. [11] Q. Wang, W. Guo, K. Zhang, A. G. Ororbia, II, X. Xing, X. Liu, C. Lee Giles, “Adversary resistant deep neural networks with an application to malware detection”, Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 13-17, 2017, Halifax, NS, Canada   [12] J. Yim, K. Sohn, “Enhancing the performance of convolutional neural networks on quality degraded datasets”, 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Sydney, NSW, 2017, pp. 1-8. doi: 10.1109/DICTA.2017.8227427 [13] N. Hatami, Y. Gavet, J. Debayle, “Classification of time-series images using deep convolutional neural networks” 2017 The 10th International Conference on Machine Vision (ICMV 2017), ICMV Committees, Nov 2017, Vienne, Austria. 10.1117/12.2309486. hal-01743695 [14] Z. Wang, T. Oates, “Spatially Encoding Temporal Correlations to Classify Temporal Data Using Convolutional Neural Networks”, В: CoRR abs/1509.07481 (2015). url: http://arxiv.org/abs/1509.07481. [15] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Adv. Neural Inf. Process. Syst. (F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, eds.), vol. 25, pp. 1097–1105, Curran Associates, Inc., 2012. [16] Z. Wang, T. Oates, “Imaging time-series to improve classification and imputation”, In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina, 25–31 July 2015, pp. 3939–3945. [17] Z. Wang, T. Oates, “Encoding time series as images for visual inspection and classification using tiled convolutional neural networks”, Association for the Advancement of Artificial Intelligence (AAAI) conference,2015. [18] Hohyun Cho, Minkyu Ahn, Sangtae Ahn, Moonyoung Kwon, Sung Chan Jun, “EEG datasets for motor imagery brain–computer interface”, GigaScience, Volume 6, Issue 7, July 2017, gix034, https://doi.org/10.1093/gigascience/gix034 [19] B. Blankertz, Kr. Müller, G. Curio, Tm. Vaughan, G. Schalk, Jr. Wolpaw, A. Schlögl, C. Neuper, G. Pfurtscheller, T. Hinterberger, M. Schröder, N. Birbaumer, “The BCI competition 2003: Progress and perspectives in detection and discrimination of EEG single trials”, IEEE Transactions on Biomedical Engineering, 2004, №6 (51), pp. 1044–1051.  Information & Communication Technologies305

Neural network analysis of electroencephalograms based on their graphical representation

http://earchive.tpu.ru/bitstream/11683/57471/1/conference_tpu-2019-C127_p302-305.pdf

Neural network analysis of electroencephalograms based on their graphical representation

Abstract

Similar works

Full text

Available Versions

Electronic archive of Tomsk Polytechnic University