10 research outputs found

    A survey on modern trainable activation functions

    Full text link
    In neural networks literature, there is a strong interest in identifying and defining activation functions which can improve neural network performance. In recent years there has been a renovated interest of the scientific community in investigating activation functions which can be trained during the learning process, usually referred to as "trainable", "learnable" or "adaptable" activation functions. They appear to lead to better network performance. Diverse and heterogeneous models of trainable activation function have been proposed in the literature. In this paper, we present a survey of these models. Starting from a discussion on the use of the term "activation function" in literature, we propose a taxonomy of trainable activation functions, highlight common and distinctive proprieties of recent and past models, and discuss main advantages and limitations of this type of approach. We show that many of the proposed approaches are equivalent to adding neuron layers which use fixed (non-trainable) activation functions and some simple local rule that constraints the corresponding weight layers.Comment: Published in "Neural Networks" journal (Elsevier

    Dynamic Complexity and Causality Analysis of Scalp EEG for Detection of Cognitive Deficits

    Get PDF
    This dissertation explores the potential of scalp electroencephalography (EEG) for the detection and evaluation of neurological deficits due to moderate/severe traumatic brain injury (TBI), mild cognitive impairment (MCI), and early Alzheimer’s disease (AD). Neurological disorders often cannot be accurately diagnosed without the use of advanced imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET). Non-quantitative task-based examinations are also used. None of these techniques, however, are typically performed in the primary care setting. Furthermore, the time and expense involved often deters physicians from performing them, leading to potential worse prognoses for patients. If feasible, screening for cognitive deficits using scalp EEG would provide a fast, inexpensive, and less invasive alternative for evaluation of TBI post injury and detection of MCI and early AD. In this work various measures of EEG complexity and causality are explored as means of detecting cognitive deficits. Complexity measures include eventrelated Tsallis entropy, multiscale entropy, inter-regional transfer entropy delays, and regional variation in common spectral features, and graphical analysis of EEG inter-channel coherence. Causality analysis based on nonlinear state space reconstruction is explored in case studies of intensive care unit (ICU) signal reconstruction and detection of cognitive deficits via EEG reconstruction models. Significant contributions in this work include: (1) innovative entropy-based methods for analyzing event-related EEG data; (2) recommendations regarding differences in MCI/AD of common spectral and complexity features for different scalp regions and protocol conditions; (3) development of novel artificial neural network techniques for multivariate signal reconstruction; and (4) novel EEG biomarkers for detection of dementia

    Improving classification models with context knowledge and variable activation functions

    Get PDF
    This work proposes two methods to boost the performances of a given classifier: the first one, which works on a Neural Network classifier, is a new type of trainable activation function, that is a function which is adjusted during the learning phase, allowing the network to exploit the data better respect to use a classic activation function with fixed-shape; the second one provides two frameworks to use an external knowledge base to improve the classification results

    Simple identification tools in FishBase

    Get PDF
    Simple identification tools for fish species were included in the FishBase information system from its inception. Early tools made use of the relational model and characters like fin ray meristics. Soon pictures and drawings were added as a further help, similar to a field guide. Later came the computerization of existing dichotomous keys, again in combination with pictures and other information, and the ability to restrict possible species by country, area, or taxonomic group. Today, www.FishBase.org offers four different ways to identify species. This paper describes these tools with their advantages and disadvantages, and suggests various options for further development. It explores the possibility of a holistic and integrated computeraided strategy

    An empirical study towards efficient learning in artificial neural networks by neuronal diversity

    Get PDF
    Artificial Neural Networks (ANN) are biologically inspired algorithms, and it is natural that it continues to inspire research in artificial neural networks. From the recent breakthrough of deep learning to the wake-sleep training routine, all have a common source of drawing inspiration: biology. The transfer functions of artificial neural networks play the important role of forming decision boundaries necessary for learning. However, there has been relatively little research on transfer function optimization compared to other aspects of neural network optimization. In this work, neuronal diversity - a property found in biological neural networks- is explored as a potentially promising method of transfer function optimization. This work shows how neural diversity can improve generalization in the context of literature from the bias-variance decomposition and meta-learning. It then demonstrates that neural diversity - represented in the form of transfer function diversity- can exhibit diverse and accurate computational strategies that can be used as ensembles with competitive results without supplementing it with other diversity maintenance schemes that tend to be computationally expensive. This work also presents neural network meta-features described as problem signatures sampled from models with diverse transfer functions for problem characterization. This was shown to meet the criteria of basic properties desired for any meta-feature, i.e. consistency for a problem and discriminatory for different problems. Furthermore, these meta-features were also used to study the underlying computational strategies adopted by the neural network models, which lead to the discovery of the strong discriminatory property of the evolved transfer function. The culmination of this study is the co-evolution of neurally diverse neurons with their weights and topology for efficient learning. It is shown to achieve significant generalization ability as demonstrated by its average MSE of 0.30 on 22 different benchmarks with minimal resources (i.e. two hidden units). Interestingly, these are the properties associated with neural diversity. Thus, showing the properties of efficiency and increased computational capacity could be replicated with transfer function diversity in artificial neural networks

    An empirical study towards efficient learning in artificial neural networks by neuronal diversity

    Get PDF
    Artificial Neural Networks (ANN) are biologically inspired algorithms, and it is natural that it continues to inspire research in artificial neural networks. From the recent breakthrough of deep learning to the wake-sleep training routine, all have a common source of drawing inspiration: biology. The transfer functions of artificial neural networks play the important role of forming decision boundaries necessary for learning. However, there has been relatively little research on transfer function optimization compared to other aspects of neural network optimization. In this work, neuronal diversity - a property found in biological neural networks- is explored as a potentially promising method of transfer function optimization. This work shows how neural diversity can improve generalization in the context of literature from the bias-variance decomposition and meta-learning. It then demonstrates that neural diversity - represented in the form of transfer function diversity- can exhibit diverse and accurate computational strategies that can be used as ensembles with competitive results without supplementing it with other diversity maintenance schemes that tend to be computationally expensive. This work also presents neural network meta-features described as problem signatures sampled from models with diverse transfer functions for problem characterization. This was shown to meet the criteria of basic properties desired for any meta-feature, i.e. consistency for a problem and discriminatory for different problems. Furthermore, these meta-features were also used to study the underlying computational strategies adopted by the neural network models, which lead to the discovery of the strong discriminatory property of the evolved transfer function. The culmination of this study is the co-evolution of neurally diverse neurons with their weights and topology for efficient learning. It is shown to achieve significant generalization ability as demonstrated by its average MSE of 0.30 on 22 different benchmarks with minimal resources (i.e. two hidden units). Interestingly, these are the properties associated with neural diversity. Thus, showing the properties of efficiency and increased computational capacity could be replicated with transfer function diversity in artificial neural networks

    Design Optimization of Wind Energy Conversion Systems with Applications

    Get PDF
    Modern and larger horizontal-axis wind turbines with power capacity reaching 15 MW and rotors of more than 235-meter diameter are under continuous development for the merit of minimizing the unit cost of energy production (total annual cost/annual energy produced). Such valuable advances in this competitive source of clean energy have made numerous research contributions in developing wind industry technologies worldwide. This book provides important information on the optimum design of wind energy conversion systems (WECS) with a comprehensive and self-contained handling of design fundamentals of wind turbines. Section I deals with optimal production of energy, multi-disciplinary optimization of wind turbines, aerodynamic and structural dynamic optimization and aeroelasticity of the rotating blades. Section II considers operational monitoring, reliability and optimal control of wind turbine components

    Symmetry Induction in Computational Intelligence

    Get PDF
    Symmetry has been a very useful tool to researchers in various scientific fields. At its most basic, symmetry refers to the invariance of an object to some transformation, or set of transformations. Usually one searches for, and uses information concerning an existing symmetry within given data, structure or concept to somehow improve algorithm performance or compress the search space. This thesis examines the effects of imposing or inducing symmetry on a search space. That is, the question being asked is whether only existing symmetries can be useful, or whether changing reference to an intuition-based definition of symmetry over the evaluation function can also be of use. Within the context of optimization, symmetry induction as defined in this thesis will have the effect of equating the evaluation of a set of given objects. Group theory is employed to explore possible symmetrical structures inherent in a search space. Additionally, conditions when the search space can have a symmetry induced on it are examined. The idea of a neighborhood structure then leads to the idea of opposition-based computing which aims to induce a symmetry of the evaluation function. In this context, the search space can be seen as having a symmetry imposed on it. To be useful, it is shown that an opposite map must be defined such that it equates elements of the search space which have a relatively large difference in their respective evaluations. Using this idea a general framework for employing opposition-based ideas is proposed. To show the efficacy of these ideas, the framework is applied to popular computational intelligence algorithms within the areas of Monte Carlo optimization, estimation of distribution and neural network learning. The first example application focuses on simulated annealing, a popular Monte Carlo optimization algorithm. At a given iteration, symmetry is induced on the system by considering opposite neighbors. Using this technique, a temporary symmetry over the neighborhood region is induced. This simple algorithm is benchmarked using common real optimization problems and compared against traditional simulated annealing as well as a randomized version. The results highlight improvements in accuracy, reliability and convergence rate. An application to image thresholding further confirms the results. Another example application, population-based incremental learning, is rooted in estimation of distribution algorithms. A major problem with these techniques is a rapid loss of diversity within the samples after a relatively low number of iterations. The opposite sample is introduced as a remedy to this problem. After proving an increased diversity, a new probability update procedure is designed. This opposition-based version of the algorithm is benchmarked using common binary optimization problems which have characteristics of deceptivity and attractive basins characteristic of difficult real world problems. Experiments reveal improvements in diversity, accuracy, reliability and convergence rate over the traditional approach. Ten instances of the traveling salesman problem and six image thresholding problems are used to further highlight the improvements. Finally, gradient-based learning for feedforward neural networks is improved using opposition-based ideas. The opposite transfer function is presented as a simple adaptive neuron which easily allows for efficiently jumping in weight space. It is shown that each possible opposite network represents a unique input-output mapping, each having an associated effect on the numerical conditioning of the network. Experiments confirm the potential of opposite networks during pre- and early training stages. A heuristic for efficiently selecting one opposite network per epoch is presented. Benchmarking focuses on common classification problems and reveals improvements in accuracy, reliability, convergence rate and generalization ability over common backpropagation variants. To further show the potential, the heuristic is applied to resilient propagation where similar improvements are also found

    Design Optimization of Wind Energy Conversion Systems with Applications

    Get PDF
    Modern and larger horizontal-axis wind turbines with power capacity reaching 15 MW and rotors of more than 235-meter diameter are under continuous development for the merit of minimizing the unit cost of energy production (total annual cost/annual energy produced). Such valuable advances in this competitive source of clean energy have made numerous research contributions in developing wind industry technologies worldwide. This book provides important information on the optimum design of wind energy conversion systems (WECS) with a comprehensive and self-contained handling of design fundamentals of wind turbines. Section I deals with optimal production of energy, multi-disciplinary optimization of wind turbines, aerodynamic and structural dynamic optimization and aeroelasticity of the rotating blades. Section II considers operational monitoring, reliability and optimal control of wind turbine components

    Tools for identifying biodiversity: progress and problems

    Get PDF
    The correct identification of organisms is fundamental not only for the assessment and the conservation of biodiversity, but also in agriculture, forestry, the food and pharmaceutical industries, forensic biology, and in the broad field of formal and informal education at all levels. In this book, the reader will find short presentations of current and upcoming projects (EDIT, KeyToNature, STERNA, Species 2000, Fishbase, BHL, ViBRANT, etc.), plus a large panel of short articles on software, taxonomic applications, use of e-keys in the educational field, and practical applications. Single-access keys are now available on most recent electronic devices; the collaborative and semantic web opens new ways to develop and to share applications; the automatic processing of molecular data and images is now based on validated systems; identification tools appear as an efficient support for environmental education and training; the monitoring of invasive and protected species and the study of climate change require intensive identifications of specimens, which opens new markets for identification research
    corecore