Operational Neural Networks

Feed-forward, fully-connected Artificial Neural Networks (ANNs) or the so-called Multi-Layer Perceptrons (MLPs) are well-known universal approximators. However, their learning performance varies significantly depending on the function or the solution space that they attempt to approximate. This is mainly because of their homogenous configuration based solely on the linear neuron model. Therefore, while they learn very well those problems with a monotonous, relatively simple and linearly separable solution space, they may entirely fail to do so when the solution space is highly nonlinear and complex. Sharing the same linear neuron model with two additional constraints (local connections and weight sharing), this is also true for the conventional Convolutional Neural Networks (CNNs) and, it is, therefore, not surprising that in many challenging problems only the deep CNNs with a massive complexity and depth can achieve the required diversity and the learning performance. In order to address this drawback and also to accomplish a more generalized model over the convolutional neurons, this study proposes a novel network model, called Operational Neural Networks (ONNs), which can be heterogeneous and encapsulate neurons with any set of operators to boost diversity and to learn highly complex and multi-modal functions or spaces with minimal network complexity and training data. Finally, a novel training method is formulated to back-propagate the error through the operational layers of ONNs. Experimental results over highly challenging problems demonstrate the superior learning capabilities of ONNs even with few neurons and hidden layers.Comment: 21 page

On the neural substrates leading to the emergence of mental operational structures

A developmental approach to the study of the emergence of mental operational structures in neural networks is presented. Neural architectures proposed to underlie the six stages of the sensory-motor period are discussed

A walk in the statistical mechanical formulation of neural networks

Neural networks are nowadays both powerful operational tools (e.g., for pattern recognition, data mining, error correction codes) and complex theoretical models on the focus of scientific investigation. As for the research branch, neural networks are handled and studied by psychologists, neurobiologists, engineers, mathematicians and theoretical physicists. In particular, in theoretical physics, the key instrument for the quantitative analysis of neural networks is statistical mechanics. From this perspective, here, we first review attractor networks: starting from ferromagnets and spin-glass models, we discuss the underlying philosophy and we recover the strand paved by Hopfield, Amit-Gutfreund-Sompolinky. One step forward, we highlight the structural equivalence between Hopfield networks (modeling retrieval) and Boltzmann machines (modeling learning), hence realizing a deep bridge linking two inseparable aspects of biological and robotic spontaneous cognition. As a sideline, in this walk we derive two alternative (with respect to the original Hebb proposal) ways to recover the Hebbian paradigm, stemming from ferromagnets and from spin-glasses, respectively. Further, as these notes are thought of for an Engineering audience, we highlight also the mappings between ferromagnets and operational amplifiers and between antiferromagnets and flip-flops (as neural networks -built by op-amp and flip-flops- are particular spin-glasses and the latter are indeed combinations of ferromagnets and antiferromagnets), hoping that such a bridge plays as a concrete prescription to capture the beauty of robotics from the statistical mechanical perspective.Comment: Contribute to the proceeding of the conference: NCTA 2014. Contains 12 pages,7 figure

Neural Networks, Cell Genome and Interactome Nonlinear Dynamic Models

Operational logic and bioinformatics models of nonlinear dynamics in complex functional systems such as neural networks, genomes and cell interactomes are proposed. Łukasiewicz Algebraic Logic models of genetic networks and signaling pathways in cells are formulated in terms of nonlinear dynamic systems with n-state components that allow for the generalization of previous logical models of both genetic activities and neural networks. An algebraic formulation of variable 'next-state functions' is extended to a Łukasiewicz Topos with an n-valued Łukasiewicz Algebraic Logic subobject classifier description that represents non-random and nonlinear network activities as well as their transformations in developmental processes and carcinogenesis

APPLICATION OF NEURAL NETWORKS IN PREDICTIVE DATA MINING

Neural Networks represent a meaningfully different approach to using computers in the workplace. A neural network is used to learn patterns and relationships in data. The data may be the results of a market research effort, or the results of a production process given varying operational conditions. Regardless of the specifics involved, applying a neural network is a substantial departure from traditional approaches. In this paper we will look into how neural networks is used in data mining. The ultimate goal of data mining is prediction - and predictive data mining is the most common type of data mining and one that has the most direct business applications. Therefore, we will consider how this technique can be used to classify the performance status of a departmental store in monitoring their productsNeural networks, data mining, prediction

The neural network art which uses the Hamming distance to measure an image similarity score

This study reports a new discrete neural network of Adaptive Resonance Theory (ART-1H) in which the Hamming distance is used for the first time to estimate the measure of binary images (vectors) proximity. For the development of a new neural network of adaptive resonance theory, architectures and operational algorithms of discrete neural networks ART-1 and discrete Hamming neural networks are used. Unlike the discrete neural network adaptive resonance theory ART-1 in which the similarity parameter which takes into account single images components only is used as a measure of images (vectors) proximity in the new network in the Hamming distance all the components of black and white images are taken into account. In contrast to the Hamming network, the new network allows the formation of typical vector classes representatives in the learning process not using information from the teacher which is not always reliable. New neural network can combine the advantages of the Hamming neural network and ART-1 by setting a part of source information in the form of reference images (distinctive feature and advantage of the Hamming neural network) and obtaining some of typical image classes representatives using learning algorithms of the neural network ART-1 (the dignity of the neural network ART-1). The architecture and functional algorithms of the new neural network ART which has the properties of both neural network ART-1 and the Hamming network were proposed and investigated. The network can use three methods to get information about typical image classes representatives: teacher information, neural network learning process, third method uses a combination of first two methods. Property of neural network ART-1 and ART-1H, related to the dependence of network learning outcomes or classification of input information to the order of the vectors (images) can be considered not as a disadvantage of the networks but as a virtue. This property allows to receive various types of input information classification which cannot be obtained using other neural networks

Exploring efficient neural architectures for linguistic-acoustic mapping in text-to-speech

Conversion from text to speech relies on the accurate mapping from linguistic to acoustic symbol sequences, for which current practice employs recurrent statistical models such as recurrent neural networks. Despite the good performance of such models (in terms of low distortion in the generated speech), their recursive structure with intermediate affine transformations tends to make them slow to train and to sample from. In this work, we explore two different mechanisms that enhance the operational efficiency of recurrent neural networks, and study their performance–speed trade-off. The first mechanism is based on the quasi-recurrent neural network, where expensive affine transformations are removed from temporal connections and placed only on feed-forward computational directions. The second mechanism includes a module based on the transformer decoder network, designed without recurrent connections but emulating them with attention and positioning codes. Our results show that the proposed decoder networks are competitive in terms of distortion when compared to a recurrent baseline, whilst being significantly faster in terms of CPU and GPU inference time. The best performing model is the one based on the quasi-recurrent mechanism, reaching the same level of naturalness as the recurrent neural network based model with a speedup of 11.2 on CPU and 3.3 on GPU.Peer ReviewedPostprint (published version