Whole brain Probabilistic Generative Model toward Realizing Cognitive Architecture for Developmental Robots

Building a humanlike integrative artificial cognitive system, that is, an artificial general intelligence, is one of the goals in artificial intelligence and developmental robotics. Furthermore, a computational model that enables an artificial cognitive system to achieve cognitive development will be an excellent reference for brain and cognitive science. This paper describes the development of a cognitive architecture using probabilistic generative models (PGMs) to fully mirror the human cognitive system. The integrative model is called a whole-brain PGM (WB-PGM). It is both brain-inspired and PGMbased. In this paper, the process of building the WB-PGM and learning from the human brain to build cognitive architectures is described.Comment: 55 pages, 8 figures, submitted to Neural Network

Cortically coupled image computing

In the 1970s, researchers at the University of California started to investigate communication between humans and computers using neural signals, which lead to the emergence of brain- computer interfaces (BCIs). In the past 40 years, significant progress has been achieved in ap- plication areas such as neuroprosthetics and rehabilitation. BCIs have been recently applied to media analytics (e.g., image search and information retrieval) as we are surrounded by tremen- dous amounts of media information today. A cortically coupled computer vision (CCCV) sys- tem is a type of BCI that exposes users to high throughput image streams via the rapid serial visual presentation (RSVP) protocol. Media analytics has also been transformed through the enormous advances in artificial intelligence (AI) in recent times. Understanding and presenting the nature of the human-AI relationship will play an important role in our society in the future. This thesis explores two lines of research in the context of traditional BCIs and AI. Firstly, we study and investigate the fundamental processing methods such as feature extraction and clas- sification for CCCV systems. Secondly, we discuss the feasibility of interfacing neural systems with AI technology through CCCV, an area we identify as neuro-AI interfacing. We have made two electroencephalography (EEG) datasets available to the community that support our inves- tigation of these two research directions. These are the neurally augmented image labelling strategies (NAILS) dataset and the neural indices for face perception analysis (NIFPA) dataset, which are introduced in Chapter 2. The first line of research focuses on studying and investigating fundamental processing methods for CCCV. In Chapter 3, we present a review on recent developments in processing methods for CCCV. This review introduces CCCV related components, specifically the RSVP experimental setup, RSVP-EEG phenomena such as the P300 and N170, evaluation metrics, feature extraction and classification. We then provide a detailed study and an analysis on spatial filtering pipelines in Chapter 4, which are the most widely used feature extraction and reduction methods in a CCCV system. In this context, we propose a spatial filtering technique named multiple time window LDA beamformers (MTWLB) and compare it to two other well-known techniques in the literature, namely xDAWN and common spatial patterns (CSP). Importantly, we demonstrate the efficacy of MTWLB for time-course source signal reconstruction compared to existing methods, which we then use as a source signal information extraction method to support a neuro-AI interface. This will be further discussed in this thesis i.e. Chapter 6 and Chapter 7. The latter part of this thesis investigates the feasibility of neuro-AI interfaces. We present two research studies which contribute to this direction. Firstly, we explore the idea of neuro- AI interfaces based on stimulus and neural systems i.e., observation of the effects of stimuli produced by different AI systems on neural signals. We use generative adversarial networks (GANs) to produce image stimuli in this case as GANs are able to produce higher quality images compared to other deep generative models. Chapter 5 provides a review on GAN-variants in terms of loss functions and architectures. In Chapter 6, we design a comprehensive experiment to verify the effects of images produced by different GANs on participants’ EEG responses. In this we propose a biologically-produced metric called Neuroscore for evaluating GAN per- formance. We highlight the consistency between Neuroscore and human perceptual judgment, which is superior to conventional metrics (i.e., Inception Score (IS), Fre ́chet Inception Distance (FID) and Kernel Maximum Mean Discrepancy (MMD) discussed in this thesis). Secondly, in order to generalize Neuroscore, we explore the use of a neuro-AI interface to help convolutional neural networks (CNNs) predict a Neuroscore with only an image as the input. In this scenario, we feed the reconstructed P300 source signals to the intermediate layer as supervisory informa- tion. We demonstrate that including biological neural information can improve the prediction performance for our proposed CNN models and the predicted Neuroscore is highly correlated with the real Neuroscore (as directly calculated from human neural signals)

Deep learning and feature engineering techniques applied to the myoelectric signal for accurate prediction of movements

Técnicas de reconhecimento de padrões no Sinal Mioelétrico (EMG) são empregadas no desenvolvimento de próteses robóticas, e para isso, adotam diversas abordagens de Inteligência Artificial (IA). Esta Tese se propõe a resolver o problema de reconhecimento de padrões EMG através da adoção de técnicas de aprendizado profundo de forma otimizada. Para isso, desenvolveu uma abordagem que realiza a extração da característica a priori, para alimentar os classificadores que supostamente não necessitam dessa etapa. O estudo integrou a plataforma BioPatRec (estudo e desenvolvimento avançado de próteses) a dois algoritmos de classificação (Convolutional Neural Network e Long Short-Term Memory) de forma híbrida, onde a entrada fornecida à rede já possui características que descrevem o movimento (nível de ativação muscular, magnitude, amplitude, potência e outros). Assim, o sinal é rastreado como uma série temporal ao invés de uma imagem, o que nos permite eliminar um conjunto de pontos irrelevantes para o classificador, tornando a informação expressivas. Na sequência, a metodologia desenvolveu um software que implementa o conceito introduzido utilizando uma Unidade de Processamento Gráfico (GPU) de modo paralelo, esse incremento permitiu que o modelo de classificação aliasse alta precisão com um tempo de treinamento inferior a 1 segundo. O modelo paralelizado foi chamado de BioPatRec-Py e empregou algumas técnicas de Engenharia de Features que conseguiram tornar a entrada da rede mais homogênea, reduzindo a variabilidade, o ruído e uniformizando a distribuição. A pesquisa obteve resultados satisfatórios e superou os demais algoritmos de classificação na maioria dos experimentos avaliados. O trabalho também realizou uma análise estatística dos resultados e fez o ajuste fino dos hiper-parâmetros de cada uma das redes. Em última instancia, o BioPatRec-Py forneceu um modelo genérico. A rede foi treinada globalmente entre os indivíduos, permitindo a criação de uma abordagem global, com uma precisão média de 97,83%.Pattern recognition techniques in the Myoelectric Signal (EMG) are employed in the development of robotic prostheses, and for that, they adopt several approaches of Artificial Intelligence (AI). This Thesis proposes to solve the problem of recognition of EMG standards through the adoption of profound learning techniques in an optimized way. The research developed an approach that extracts the characteristic a priori to feed the classifiers that supposedly do not need this step. The study integrated the BioPatRec platform (advanced prosthesis study and development) to two classification algorithms (Convolutional Neural Network and Long Short-Term Memory) in a hybrid way, where the input provided to the network already has characteristics that describe the movement (level of muscle activation, magnitude, amplitude, power, and others). Thus, the signal is tracked as a time series instead of an image, which allows us to eliminate a set of points irrelevant to the classifier, making the information expressive. In the sequence, the methodology developed software that implements the concept introduced using a Graphical Processing Unit (GPU) in parallel this increment allowed the classification model to combine high precision with a training time of less than 1 second. The parallel model was called BioPatRec-Py and employed some Engineering techniques of Features that managed to make the network entry more homogeneous, reducing variability, noise, and standardizing distribution. The research obtained satisfactory results and surpassed the other classification algorithms in most of the evaluated experiments. The work performed a statistical analysis of the outcomes and fine-tuned the hyperparameters of each of the networks. Ultimately, BioPatRec-Py provided a generic model. The network was trained globally between individuals, allowing the creation of a standardized approach, with an average accuracy of 97.83%

Hyperspectral Imaging from Ground Based Mobile Platforms and Applications in Precision Agriculture

This thesis focuses on the use of line scanning hyperspectral sensors on mobile ground based platforms and applying them to agricultural applications. First this work deals with the geometric and radiometric calibration and correction of acquired hyperspectral data. When operating at low altitudes, changing lighting conditions are common and inevitable, complicating the retrieval of a surface's reflectance, which is solely a function of its physical structure and chemical composition. Therefore, this thesis contributes the evaluation of an approach to compensate for changes in illumination and obtain reflectance that is less labour intensive than traditional empirical methods. Convenient field protocols are produced that only require a representative set of illumination and reflectance spectral samples. In addition, a method for determining a line scanning camera's rigid 6 degree of freedom (DOF) offset and uncertainty with respect to a navigation system is developed, enabling accurate georegistration and sensor fusion. The thesis then applies the data captured from the platform to two different agricultural applications. The first is a self-supervised weed detection framework that allows training of a per-pixel classifier using hyperspectral data without manual labelling. The experiments support the effectiveness of the framework, rivalling classifiers trained on hand labelled training data. Then the thesis demonstrates the mapping of mango maturity using hyperspectral data on an orchard wide scale using efficient image scanning techniques, which is a world first result. A novel classification, regression and mapping pipeline is proposed to generate per tree mango maturity averages. The results confirm that maturity prediction in mango orchards is possible in natural daylight using a hyperspectral camera, despite complex micro-illumination-climates under the canopy

On the Utility of Representation Learning Algorithms for Myoelectric Interfacing

Electrical activity produced by muscles during voluntary movement is a reflection of the firing patterns of relevant motor neurons and, by extension, the latent motor intent driving the movement. Once transduced via electromyography (EMG) and converted into digital form, this activity can be processed to provide an estimate of the original motor intent and is as such a feasible basis for non-invasive efferent neural interfacing. EMG-based motor intent decoding has so far received the most attention in the field of upper-limb prosthetics, where alternative means of interfacing are scarce and the utility of better control apparent. Whereas myoelectric prostheses have been available since the 1960s, available EMG control interfaces still lag behind the mechanical capabilities of the artificial limbs they are intended to steer—a gap at least partially due to limitations in current methods for translating EMG into appropriate motion commands. As the relationship between EMG signals and concurrent effector kinematics is highly non-linear and apparently stochastic, finding ways to accurately extract and combine relevant information from across electrode sites is still an active area of inquiry.This dissertation comprises an introduction and eight papers that explore issues afflicting the status quo of myoelectric decoding and possible solutions, all related through their use of learning algorithms and deep Artificial Neural Network (ANN) models. Paper I presents a Convolutional Neural Network (CNN) for multi-label movement decoding of high-density surface EMG (HD-sEMG) signals. Inspired by the successful use of CNNs in Paper I and the work of others, Paper II presents a method for automatic design of CNN architectures for use in myocontrol. Paper III introduces an ANN architecture with an appertaining training framework from which simultaneous and proportional control emerges. Paper Iv introduce a dataset of HD-sEMG signals for use with learning algorithms. Paper v applies a Recurrent Neural Network (RNN) model to decode finger forces from intramuscular EMG. Paper vI introduces a Transformer model for myoelectric interfacing that do not need additional training data to function with previously unseen users. Paper vII compares the performance of a Long Short-Term Memory (LSTM) network to that of classical pattern recognition algorithms. Lastly, paper vIII describes a framework for synthesizing EMG from multi-articulate gestures intended to reduce training burden