Modelling of Human Control and Performance Evaluation using Artificial Neural Network and Brainwave

Conventionally, a human has to learn to operate a machine by himself / herself. Human Adaptive Mechatronics (HAM) aims to investigate a machine that has the capability to learn its operator skills in order to provide assistance and guidance appropriately. Therefore, the understanding of human behaviour during the human-machine interaction (HMI) from the machine’s side is essential. The focus of this research is to propose a model of human-machine control strategy and performance evaluation from the machine’s point of view. Various HAM simulation scenarios are developed for the investigations of the HMI. The first case study that utilises the classic pendulum-driven capsule system reveals that a human can learn to control the unfamiliar system and summarise the control strategy as a set of rules. Further investigation of the case study is conducted with nine participants to explore the performance differences and control characteristics among them. High performers tend to control the pendulum at high frequency in the right portion of the angle range while the low performers perform inconsistent control behaviour. This control information is used to develop a human-machine control model by adopting an Artificial Neural Network (ANN) and 10-time- 10-fold cross-validation. Two models of capsule direction and position predictions are obtained with 88.3% and 79.1% accuracies, respectively. An Electroencephalogram (EEG) headset is integrated into the platform for monitoring brain activity during HMI. A number of preliminary studies reveal that the brain has a specific response pattern to particular stimuli compared to normal brainwaves. A novel human-machine performance evaluation based on the EEG brainwaves is developed by utilising a classical target hitting task as a case study of HMI. Six models are obtained for the evaluation of the corresponding performance aspects including the Fitts index of performance. The averaged evaluation accuracy of the models is 72.35%. However, the accuracy drops to 65.81% when the models are applied to unseen data. In general, it can be claimed that the accuracy is satisfactory since it is very challenging to evaluate the HMI performance based only on the EEG brainwave activity

Data S1: Data

We present the evaluation of two well-known, low-cost consumer-grade EEG devices: the Emotiv EPOC and the Neurosky MindWave. Problems with using the consumer-grade EEG devices (BCI illiteracy, poor technical characteristics, and adverse EEG artefacts) are discussed. The experimental evaluation of the devices, performed with 10 subjects asked to perform concentration/relaxation and blinking recognition tasks, is given. The results of statistical analysis show that both devices exhibit high variability and non-normality of attention and meditation data, which makes each of them difficult to use as an input to control tasks. BCI illiteracy may be a significant problem, as well as setting up of the proper environment of the experiment. The results of blinking recognition show that using the Neurosky device means recognition accuracy is less than 50%, while the Emotiv device has achieved a recognition accuracy of more than 75%; for tasks that require concentration and relaxation of subjects, the Emotiv EPOC device has performed better (as measured by the recognition accuracy) by ∼9%. Therefore, the Emotiv EPOC device may be more suitable for control tasks using the attention/meditation level or eye blinking than the Neurosky MindWave device

Real-Time Electroencephalogram Sonification for Neurofeedback

Electroencephalography (EEG) is the measurement via the scalp of the electrical activity of the brain. The established therapeutic intervention of neurofeedback involves presenting people with their own EEG in real-time to enable them to modify their EEG for purposes of improving performance or health. The aim of this research is to develop and validate real-time sonifications of EEG for use in neurofeedback and methods for assessing such sonifications. Neurofeedback generally uses a visual display. Where auditory feedback is used, it is mostly limited to pre-recorded sounds triggered by the EEG activity crossing a threshold. However, EEG generates time-series data with meaningful detail at fine temporal resolution and with complex temporal dynamics. Human hearing has a much higher temporal resolution than human vision, and auditory displays do not require people to focus on a screen with their eyes open for extended periods of time – e.g. if they are engaged in some other task. Sonification of EEG could allow more rapid, contingent, salient and temporally detailed feedback. This could improve the efficiency of neurofeedback training and reduce the number and duration of sessions for successful neurofeedback. The same two deliberately simple sonification techniques were used in all three experiments of this research: Amplitude Modulation (AM) sonification, which maps the fluctuations in the power of the EEG to the volume of a pure tone; and Frequency Modulation (FM) sonification, which uses the changes in the EEG power to modify the frequency. Measures included, a listening task, NASA task load index; a measure of how much work it was to do the task, Pre & post measures of mood, and EEG. The first experiment used pre-recorded single channel EEG and participants were asked to listen to the sound of the sonified EEG and try and track the activity that they could hear by moving a slider on a computer screen using a computer mouse. This provided a quantitative assessment of how well people could perceive the sonified fluctuations in EEG level. The tracking accuracy scores were higher for the FM sonification but self-assessments of task load rated the AM sonification as easier to track. The second experiment used the same two sonifications, in a real neurofeedback task using participants own live EEG. Unbeknownst to the participants the neurofeedback task was designed to improve mood. A Pre-Post questionnaire showed that participants changed their self-rated mood in the intended direction with the EEG training, but there was no statistically significant change in EEG. Again the FM sonification showed a better performance but AM was rated as less effortful. The performance of sonifications in the tracking task in experiment 1 was found to predict their relative efficacy at blind self-rated mood modification in experiment 2. The third experiment used both the tracking as in experiment 1 and neurofeedback tasks as in experiment 2, but with modified versions of the AM and FM sonifications to allow two-channel EEG sonifications. This experiment introduced a physical slider as opposed to a mouse for the tracking task. Tracking accuracy increased, but this time no significant difference was found between the two sonification techniques on the tracking task. In the training task, once more the blind self-rated mood did improve in the intended direction with the EEG training, but as again there was no significant change in EEG, this cannot necessarily be attributed to the neurofeedback. There was only a slight difference between the two sonification techniques in the effort measure. In this way, a prototype method has been devised and validated for the quantitative assessment of real-time EEG sonifications. Conventional evaluations of neurofeedback techniques are expensive and time consuming. By contrast, this method potentially provides a rapid, objective and efficient method for evaluating the suitability of candidate sonifications for EEG neurofeedback

Motivation Modelling and Computation for Personalised Learning of People with Dyslexia

The increasing development of e-learning systems in recent decades has benefited ubiquitous computing and education by providing freedom of choice to satisfy various needs and preferences about learning places and paces. Automatic recognition of learners’ states is necessary for personalised services or intervention to be provided in e-learning environments. In current literature, assessment of learners’ motivation for personalised learning based on the motivational states is lacking. An effective learning environment needs to address learners’ motivational needs, particularly, for those with dyslexia. Dyslexia or other learning difficulties can cause young people not to engage fully with the education system or to drop out due to complex reasons: in addition to the learning difficulties related to reading, writing or spelling, psychological difficulties are more likely to be ignored such as lower academic self-worth and lack of learning motivation caused by the unavoidable learning difficulties. Associated with both cognitive processes and emotional states, motivation is a multi-facet concept that consequences in the continued intention to use an e-learning system and thus a better chance of learning effectiveness and success. It consists of factors from intrinsic motivation driven by learners’ inner feeling of interest or challenges and those from extrinsic motivation associated with external reward or compliments. These factors represent learners’ various motivational needs; thus, understanding this requires a multidisciplinary approach. Combining different perspectives of knowledge on psychological theories and technology acceptance models with the empirical findings from a qualitative study with dyslexic students conducted in the present research project, motivation modelling for people with dyslexia using a hybrid approach is the main focus of this thesis. Specifically, in addition to the contribution to the qualitative conceptual motivation model and ontology-based computational model that formally expresses the motivational factors affecting users’ continued intention to use e-learning systems, this thesis also conceives a quantitative approach to motivation modelling. A multi-item motivation questionnaire is designed and employed in a quantitative study with dyslexic students, and structural equation modelling techniques are used to quantify the influences of the motivational factors on continued use intention and their interrelationships in the model. In addition to the traditional approach to motivation computation that relies on learners’ self-reported data, this thesis also employs dynamic sensor data and develops classification models using logistic regression for real-time assessment of motivational states. The rule-based reasoning mechanism for personalising motivational strategies and a framework of motivationally personalised e-learning systems are introduced to apply the research findings to e-learning systems in real-world scenarios. The motivation model, sensor-based computation and rule-based personalisation have been applied to a practical scenario with an essential part incorporated in the prototype of a gaze-based learning application that can output personalised motivational strategies during the learning process according to the real-time assessment of learners’ motivational states based on both the eye-tracking data in addition to users’ self-reported data. Evaluation results have indicated the advantage of the application implemented compared to the traditional one without incorporating the present research findings for monitoring learners’ motivation states with gaze data and generating personalised feedback. In summary, the present research project has: 1) developed a conceptual motivation model for students with dyslexia defining the motivational factors that influence their continued intention to use e-learning systems based on both a qualitative empirical study and prior research and theories; 2) developed an ontology-based motivation model in which user profiles, factors in the motivation model and personalisation options are structured as a hierarchy of classes; 3) designed a multi-item questionnaire, conducted a quantitative empirical study, used structural equation modelling to further explore and confirm the quantified impacts of motivational factors on continued use intention and the quantified relationships between the factors; 4) conducted an experiment to exploit sensors for motivation computation, and developed classification models for real-time assessment of the motivational states pertaining to each factor in the motivation model based on empirical sensor data including eye gaze data and EEG data; 5) proposed a sensor-based motivation assessment system architecture with emphasis on the use of ontologies for a computational representation of the sensor features used for motivation assessment in addition to the representation of the motivation model, and described the semantic rule-based personalisation of motivational strategies; 6) proposed a framework of motivationally personalised e-learning systems based on the present research, with the prototype of a gaze-based learning application designed, implemented and evaluated to guide future work

BRAIN-COMPUTER MUSIC INTERFACING: DESIGNING PRACTICAL SYSTEMS FOR CREATIVE APPLICATIONS

Brain-computer music interfacing (BCMI) presents a novel approach to music making, as it requires only the brainwaves of a user to control musical parameters. This presents immediate benefits for users with motor disabilities that may otherwise prevent them from engaging in traditional musical activities such as composition, performance or collaboration with other musicians. BCMI systems with active control, where a user can make cognitive choices that are detected within brain signals, provide a platform for developing new approaches towards accomplishing these activities. BCMI systems that use passive control present an interesting alternate to active control, where control over music is accomplished by harnessing brainwave patterns that are associated with subconscious mental states. Recent developments in brainwave measuring technologies, in particular electroencephalography (EEG), have made brainwave interaction with computer systems more affordable and accessible and the time is ripe for research into the potential such technologies can offer for creative applications for users of all abilities. This thesis presents an account of BCMI development that investigates methods of active, passive and hybrid (multiple control methods) control that include control over electronic music, acoustic instrumental music, multi-brain systems and combining methods of brainwave control. In practice there are many obstacles associated with detecting useful brainwave signals, in particular when scaling systems otherwise designed for medical studies for use outside of laboratory settings. Two key areas are addressed throughout this thesis. Firstly, improving the accuracy of meaningful brain signal detection in BCMI, and secondly, exploring the creativity available in user control through ways in which brainwaves can be mapped to musical features. Six BCMIs are presented in this thesis, each with the objective of exploring a unique aspect of user control. Four of these systems are designed for live BCMI concert performance, one evaluates a proof-of-concept through end-user testing and one is designed as a musical composition tool. The thesis begins by exploring the field of brainwave detection and control and identifies the steady-state visually evoked potential (SSVEP) method of eliciting brainwave control as a suitable technique for use in BCMI. In an attempt to improve signal accuracy of the SSVEP technique a new modular hardware unit is presented that provides accurate SSVEP stimuli, suitable for live music performance. Experimental data confirms the performance of the unit in tests across three different EEG hardware platforms. Results across 11 users indicate that a mean accuracy of 96% and an average response time of 3.88 seconds are attainable with the system. These results contribute to the development of the BCMI for Activating Memory, a multi-user system. Once a stable SSVEP platform is developed, control is extended through the integration of two more brainwave control techniques: affective (emotional) state detection and motor imagery response. In order to ascertain the suitability of the former an experiment confirms the accuracy of EEG when measuring affective states in response to music in a pilot study. This thesis demonstrates how a range of brainwave detection methods can be used for creative control in musical applications. Video and audio excerpts of BCMI pieces are also included in the Appendices

Cue reactivity to self-harm cues: the development of a systematic treatment intervention for deliberate self-harm

There is increasing awareness of the prevalence of deliberate self-harm (DSH) although the phenomenon is still poorly understood. Those who self-harm often have a poor long-term prognosis, yet systematic focused treatment interventions are scarce. DSH appears to share fundamental characteristics with addictive behaviour, including; impulsive or compulsive urges to act in the presence of triggers, positive and negative reinforcing consequences and endorsement of the diagnostic criteria for clinical dependence. Given this fact, a behavioural mode of DSH may be appropriate. A range of events are anecdotally reported to trigger DSH. This thesis was designed to identify these cues, to develop an understanding of how those who self-harm respond to these cues and the processes by which these cues may operate to maintain DSH. An intervention based on the management of urges to self-harm in the presence of these cues was developed.Study I identified that triggers for DSH (interpersonal, intrapersonal and environmental) were similar to those that reliably predict addictive behaviour. Respondents endorsed the diagnostic criteria for dependency and reported that the act of DSH reduced negative emotions. The second two studies identified self-reported cue reactivity, and generalised hyperarousal to both DSH and neutral stimuli in those who self-harm but no evidence of psychophysiological cue reactivity. Study IV used ERP methodology to evaluate cue reactivity at the CNS level and to evaluate two mechanisms by which cues might operate to maintain DSH. There was some preliminary support for enhanced preconscious attentional bias towards emotional, but not environmental DSH cues, and no support for emotional interference. Study V identified that those who self-harm exhibited enhanced tolerance to physical and psychological stressors, and that priming with interpersonal distress did not impact on this tolerance. Finally, a single case intervention study identified a reduction in DSH, reduced psychophysiological arousal and urges to self-harm and improved clinical symptomatology. However, clinical improvements were not time-locked to targeted exposure intervention phases. The clinical and theoretical implications for these findings are discussed

Brain Waves, A Cultural History: Oscillations of Neuroscience, Technology, Telepathy, and Transcendence

This project proceeds from a narrow question: What, if anything, is a brain wave? Beguiling in its simplicity, this question prompts a cultural-historical investigation that spans over 150 years of science, technology, and society. Proposed in 1869, the original theory of brain waves cites etheric undulations to explain reports of apparent thought transference. Though most modern thinkers no longer believe in outright telepathy, I argue that dreams of thought transmission and other mental miracles subtly persist—not in obscure and occult circles, but at the forefront of technoscience. A hybrid of science and fiction, brain waves represent an ideal subject through which to explore the ways in which technical language shrouds spiritual dreams. Today, the phrase “brain waves” often function as shorthand for electrical changes in the brain, particularly in the context of technologies that purport to “read” some aspect of mental function, or to transmit neural data to a digital device. While such technologies appear uniquely modern, the history of brain waves reveals that they are merely the millennial incarnation of a much older hope—a hope for transmission and transcendence via the brain’s emanations

Traiter la dépendance à la nicotine par le neurofeedback chez les adultes ayant un trouble déficitaire de l'attention

Le trouble déficitaire de l'attention avec hyperactivité (TDAH) et le tabagisme ont un lien établi. Les personnes atteintes d’un TDAH sont plus portées à développer une dépendance à la cigarette, à débuter le tabagisme plus jeune et ont plus de difficultés à cesser de fumer. Il est maintenant connu que le traitement du TDAH modifie ces interactions. Bien que les psychostimulants soient le traitement pharmacologique de choix pour le TDAH, les effets secondaires indésirables de ces substances réduisent considérablement l’utilisation par ceux qui veulent cesser de fumer, surtout s’ils utilisent déjà des substances ayant des propriétés stimulantes, tel que la nicotine, pour les aider. Cette étude a comme objectif d’évaluer l’efficacité potentielle d’un traitement de neurofeedback chez des adultes atteints d’un TDAH et qui, malgré l’utilisation d’un timbre de nicotine, n’arrivaient toujours pas à cesser de fumer. Quatre participantes qui ont rencontré les critères de recherche pour le TDAH ont reçu 12 à 14 séances de neurofeedback pendant qu’elles continuaient un traitement avec un timbre de nicotine. L’efficacité de l’intervention en neurofeedback est évaluée selon un devis de recherche à cas unique avec lignes de base multiples établies en fonction des participants. L’analyse post-intervention révèle que trois des quatre participantes ont réduit de façon significative leur dépendance sur la nicotine à la suite du traitement. Le neurofeedback déjà connu comme traitement efficace du TDAH, dans le cas de dépendance à la nicotine, peut améliorer la tolérance aux symptômes de sevrage en passant par une amélioration de l’attention