A Unique Method of Using Information Entropy to Evaluate the Reliability of Deep Neural Network Predictions on Intracranial Electroencephalogram

Deep Neural networks (DNN) are fundamentally information processing machines, which synthesize the complex patterns in input to arrive at solutions, with applications in various fields. One major question when working with the DNN is, which features in the input lead to a specific decision by DNN. One of the common methods of addressing this question involve generation of heatmaps. Another pertinent question is how effectively DNN has captured the entire information presented in the input, which can potentially be addressed with complexity measures of the inputs. In the case of patients with intractable epilepsy, appropriate clinical decision making depends on the interpretation of the brain signals, as recorded in the form of Electroencephalogram (EEG), which in most of the cases will be recorded through intracranial monitoring (iEEG)). In current clinical settings, the iEEG is visually inspected by the clinicians to arrive at decisions regarding the location of the epileptogenic zones which is used in the determination of surgical planning. Visual inspection and decision making is a very tedious and potentially error prone approach, given the massive amount of data that need to be evaluated in a limited amount of time. We developed a DNN model to evaluate iEEG to classify signals arising from epileptic and non-epileptic zones. One of the challenges of incorporating the deep neural network tools in the medical decision making is the black box nature of these tools. To further analyze the underlying reasons for DNN\u27s decision regarding iEEG, we used heatmapping and signal processing tools to better understand the decision-making process of DNN. We were able to demonstrate that the energy rich regions, as captured by analytical signals, is identified by DNN as potentially epileptogenic, when arriving at decisions. We explored the DNN\u27s ability to capture the details of the signal with information theoretical approaches. We introduced a measure of confidence of DNN predictions, named certainty index, which is calculated based on the overall outputs in the penultimate layer of the network. We employed the method of Sample Entropy (SampEn) and were able to demonstrate that the DNN\u27s prediction certainty is related to how effectively the heatmap is correlated to the SampEn of the entire signal. We explored the parameter space of the SampEn calculation and demonstrate that the relationship between SampEn and certainty of DNN predictions hold even on changing the estimation parameters. Further we were able to demonstrate that the rate of change of relationship between the DNN output and activation map, as a function of the sequential DNN layers, is related to the SampEn of the signal. This observation suggests that the speed at which DNN captures the results is directly proportional to the information content in the signal

Synthesizing multi-layer perceptron network with ant lion biogeography-based dragonfly algorithm evolutionary strategy invasive weed and league champion optimization hybrid algorithms in predicting heating load in residential buildings

The significance of accurate heating load (HL) approximation is the primary motivation of this research to distinguish the most efficient predictive model among several neural-metaheuristic models. The proposed models are formulated through synthesizing a multi-layer perceptron network (MLP) with ant lion optimization (ALO), biogeography-based optimization (BBO), the dragonfly algorithm (DA), evolutionary strategy (ES), invasive weed optimization (IWO), and league champion optimization (LCA) hybrid algorithms. Each ensemble is optimized in terms of the operating population. Accordingly, the ALO-MLP, BBO-MLP, DA-MLP, ES-MLP, IWO-MLP, and LCA-MLP presented their best performance for population sizes of 350, 400, 200, 500, 50, and 300, respectively. The comparison was carried out by implementing a ranking system. Based on the obtained overall scores (OSs), the BBO (OS = 36) featured as the most capable optimization technique, followed by ALO (OS = 27) and ES (OS = 20). Due to the efficient performance of these algorithms, the corresponding MLPs can be promising substitutes for traditional methods used for HL analysis

Computational Optimizations for Machine Learning

The present book contains the 10 articles finally accepted for publication in the Special Issue “Computational Optimizations for Machine Learning” of the MDPI journal Mathematics, which cover a wide range of topics connected to the theory and applications of machine learning, neural networks and artificial intelligence. These topics include, among others, various types of machine learning classes, such as supervised, unsupervised and reinforcement learning, deep neural networks, convolutional neural networks, GANs, decision trees, linear regression, SVM, K-means clustering, Q-learning, temporal difference, deep adversarial networks and more. It is hoped that the book will be interesting and useful to those developing mathematical algorithms and applications in the domain of artificial intelligence and machine learning as well as for those having the appropriate mathematical background and willing to become familiar with recent advances of machine learning computational optimization mathematics, which has nowadays permeated into almost all sectors of human life and activity

Towards a Brain-inspired Information Processing System: Modelling and Analysis of Synaptic Dynamics: Towards a Brain-inspired InformationProcessing System: Modelling and Analysis ofSynaptic Dynamics

Biological neural systems (BNS) in general and the central nervous system (CNS) specifically exhibit a strikingly efficient computational power along with an extreme flexible and adaptive basis for acquiring and integrating new knowledge. Acquiring more insights into the actual mechanisms of information processing within the BNS and their computational capabilities is a core objective of modern computer science, computational sciences and neuroscience. Among the main reasons of this tendency to understand the brain is to help in improving the quality of life of people suffer from loss (either partial or complete) of brain or spinal cord functions. Brain-computer-interfaces (BCI), neural prostheses and other similar approaches are potential solutions either to help these patients through therapy or to push the progress in rehabilitation. There is however a significant lack of knowledge regarding the basic information processing within the CNS. Without a better understanding of the fundamental operations or sequences leading to cognitive abilities, applications like BCI or neural prostheses will keep struggling to find a proper and systematic way to help patients in this regard. In order to have more insights into these basic information processing methods, this thesis presents an approach that makes a formal distinction between the essence of being intelligent (as for the brain) and the classical class of artificial intelligence, e.g. with expert systems. This approach investigates the underlying mechanisms allowing the CNS to be capable of performing a massive amount of computational tasks with a sustainable efficiency and flexibility. This is the essence of being intelligent, i.e. being able to learn, adapt and to invent. The approach used in the thesis at hands is based on the hypothesis that the brain or specifically a biological neural circuitry in the CNS is a dynamic system (network) that features emergent capabilities. These capabilities can be imported into spiking neural networks (SNN) by emulating the dynamic neural system. Emulating the dynamic system requires simulating both the inner workings of the system and the framework of performing the information processing tasks. Thus, this work comprises two main parts. The first part is concerned with introducing a proper and a novel dynamic synaptic model as a vital constitute of the inner workings of the dynamic neural system. This model represents a balanced integration between the needed biophysical details and being computationally inexpensive. Being a biophysical model is important to allow for the abilities of the target dynamic system to be inherited, and being simple is needed to allow for further implementation in large scale simulations and for hardware implementation in the future. Besides, the energy related aspects of synaptic dynamics are studied and linked to the behaviour of the networks seeking for stable states of activities. The second part of the thesis is consequently concerned with importing the processing framework of the dynamic system into the environment of SNN. This part of the study investigates the well established concept of binding by synchrony to solve the information binding problem and to proposes the concept of synchrony states within SNN. The concepts of computing with states are extended to investigate a computational model that is based on the finite-state machines and reservoir computing. Biological plausible validations of the introduced model and frameworks are performed. Results and discussions of these validations indicate that this study presents a significant advance on the way of empowering the knowledge about the mechanisms underpinning the computational power of CNS. Furthermore it shows a roadmap on how to adopt the biological computational capabilities in computation science in general and in biologically-inspired spiking neural networks in specific. Large scale simulations and the development of neuromorphic hardware are work-in-progress and future work. Among the applications of the introduced work are neural prostheses and bionic automation systems

Simulation and implementation of novel deep learning hardware architectures for resource constrained devices

Corey Lammie designed mixed signal memristive-complementary metal–oxide–semiconductor (CMOS) and field programmable gate arrays (FPGA) hardware architectures, which were used to reduce the power and resource requirements of Deep Learning (DL) systems; both during inference and training. Disruptive design methodologies, such as those explored in this thesis, can be used to facilitate the design of next-generation DL systems