Output Reachable Set Estimation and Verification for Multi-Layer Neural Networks

In this paper, the output reachable estimation and safety verification problems for multi-layer perceptron neural networks are addressed. First, a conception called maximum sensitivity in introduced and, for a class of multi-layer perceptrons whose activation functions are monotonic functions, the maximum sensitivity can be computed via solving convex optimization problems. Then, using a simulation-based method, the output reachable set estimation problem for neural networks is formulated into a chain of optimization problems. Finally, an automated safety verification is developed based on the output reachable set estimation result. An application to the safety verification for a robotic arm model with two joints is presented to show the effectiveness of proposed approaches.Comment: 8 pages, 9 figures, to appear in TNNL

Neural networks in geophysical applications

Neural networks are increasingly popular in geophysics. Because they are universal approximators, these tools can approximate any continuous function with an arbitrary precision. Hence, they may yield important contributions to finding solutions to a variety of geophysical applications. However, knowledge of many methods and techniques recently developed to increase the performance and to facilitate the use of neural networks does not seem to be widespread in the geophysical community. Therefore, the power of these tools has not yet been explored to their full extent. In this paper, techniques are described for faster training, better overall performance, i.e., generalization,and the automatic estimation of network size and architecture

Enhanced classification of network traffic data captured by intrusion prevention systems

A common practice in modern computer networks is the deployment of Intrusion Prevention Systems (IPSs) for the purpose of identifying security threats. Such systems provide alerts on suspicious activities based on a predefined set of rules. These alerts almost always contain high percentages of false positives and false negatives, which may impede the efficacy of their use. Therefore, with the presence of high numbers of false positives and false negatives, the analysis of network traffic data can be ineffective for decision makers which normally require concise, and preferably, visual forms to base their decisions upon. Machine learning techniques can help extract useful information from large datasets. Combined with visualisation, classification could provide a solution to false alerts and text-based outputs of IPSs. This research developed two new classification techniques that outperformed the traditional classification methods in accurate classification of computer network traffic captured by an IPS framework. They are also highly effective. The main purpose of these techniques was the effective identification of malicious network traffic and this was demonstrated via extensive experimental evaluation (where many experiments were conducted and results are reported in this thesis). In addition, an enhancement of the principal component analysis (PCA) was presented as part of this study. This enhancement proved to outperform the classical PCA on classification of IPS data. Details of the evaluation and experiments are provided in this thesis. One of the classification methods described in this thesis achieved accuracy values of 98.51% and 99.76% on two computer network traffic dataset settings, whereas the Class-balanced Similarity Based Instance Transfer Learning (CB-SBIT) algorithm achieves accuracy values of 93.56% and 96.25% respectively on the same dataset settings. This means the proposed method outperforms the state-of-the-art algorithm. As for the PCA enhancement mentioned above, using its resulting principal components as inputs to classifiers leads to improved accuracy when compared to the classical PCA

Neural network methods for one-to-many multi-valued mapping problems

An investigation of the applicability of neural network-based methods in predicting the values of multiple parameters, given the value of a single parameter within a particular problem domain is presented. In this context, the input parameter may be an important source of variation that is related with a complex mapping function to the remaining sources of variation within a multivariate distribution. The definition of the relationship between the variables of a multivariate distribution and a single source of variation allows the estimation of the values of multiple variables given the value of the single variable, addressing in that way an ill-conditioned one-to-many mapping problem. As part of our investigation, two problem domains are considered: predicting the values of individual stock shares, given the value of the general index, and predicting the grades received by high school pupils, given the grade for a single course or the average grade. With our work, the performance of standard neural network-based methods and in particular multilayer perceptrons (MLPs), radial basis functions (RBFs), mixture density networks (MDNs) and a latent variable method, the general topographic mapping (GTM), is compared. According to the results, MLPs and RBFs outperform MDNs and the GTM for these one-to-many mapping problems

Advanced Methods in Neural Networks-Based Sensitivity Analysis with their Applications in Civil Engineering

Artificial neural networks (ANNs) are powerful tools that are used in various engineering fields. Their characteristics enable them to solve prediction, regression, and classification problems. Nevertheless, the ANN is usually thought of as a black box, in which it is difficult to determine the effect of each explicative variable (input) on the dependent variables (outputs) in any problem. To investigate such effects, sensitivity analysis is usually applied on the optimal pre-trained ANN. Existing sensitivity analysis techniques suffer from drawbacks. Their basis on a single optimal pre-trained ANN model produces instability in parameter sensitivity analysis because of the uncertainty in neural network modeling. To overcome this deficiency, two successful sensitivity analysis paradigms, the neural network committee (NNC)-based sensitivity analysis and the neural network ensemble (NNE)-based parameter sensitivity analysis, are illustrated in this chapter. An NNC is applied in a case study of geotechnical engineering involving strata movement. An NNE is implemented for sensitivity analysis of two classic problems in civil engineering: (i) the fracture failure of notched concrete beams and (ii) the lateral deformation of deep-foundation pits. Results demonstrate good ability to analyze the sensitivity of the most influential parameters, illustrating the underlying mechanisms of such engineering systems

Experimental Study and ANN Dual-Time Scale Perturbation Model of Electrokinetic Properties of Microbiota

[Abstract] The electrokinetic properties of the rumen microbiota are involved in cell surface adhesion and microbial metabolism. An in vitro study was carried out in batch culture to determine the effects of three levels of special surface area (SSA) of biomaterials and four levels of surface tension (ST) of culture medium on electrokinetic properties (Zeta potential, ξ; electrokinetic mobility, μe), fermentation parameters (volatile fatty acids, VFAs), and ST over fermentation processes (ST-a, γ). The obtained results were combined with previously published data (digestibility, D; pH; concentration of ammonia nitrogen, c(NH3-N)) to establish a predictive artificial neural network (ANN) model. Concepts of dual-time series analysis, perturbation theory (PT), and Box-Jenkins Operators were applied for the first time to develop an ANN model to predict the variations of the electrokinetic properties of microbiota. The best dual-time series Radial Basis Functions (RBR) model for ξ of rumen microbiota predicted ξ for >30,000 cases with a correlation coefficient >0.8. This model provided insight into the correlations between electrokinetic property (zeta potential) of rumen microbiota and the perturbations of physical factors (specific surface area and surface tension) of media, digestibility of substrate, and their metabolites (NH3-N, VFAs) in relation to environmental factors.National Natural Science Foundation of China; 31172234National Natural Science Foundation of China; 31260556),Planned Science and Technology Project of Hu-nan Province; 2015NK3041Technology Specialty Fund for Cooperation between Jilin Province and the Chinese Academy of Sciences; 2016SYHZ0022Hunan Provincial Creation Development Project; 2013TF3006Xunta de Galicia; GRC2014/049Xunta de Galicia; R2014/039Ministerio de Economía, Industria y Competitividad; FJCI-2015-2607

Isolating Stock Prices Variation with Neural Networks

In this study we aim to define a mapping function that relates the general index value among a set of shares to the prices of individual shares. In more general terms this is problem of defining the relationship between multivariate data distributions and a specific source of variation within these distributions where the source of variation in question represents a quantity of interest related to a particular problem domain. In this respect we aim to learn a complex mapping function that can be used for mapping different values of the quantity of interest to typical novel samples of the distribution. In our investigation we compare the performance of standard neural network based methods like Multilayer Perceptrons (MLPs) and Radial Basis Functions (RBFs) as well as Mixture Density Networks (MDNs) and a latent variable method, the General Topographic Mapping (GTM). As a reference benchmark of the prediction accuracy we consider a simple method based on the average values over certain intervals of the quantity of interest that we are trying to isolate (the so called Sample Average (SA) method). According to the results, MLPs and RBFs outperform MDNs and the GTM for this one-to-many mapping problem

Structural dynamics-guided hierarchical neural-networks scheme for locating and quantifying damage in beam-type structures

Neural networks-based intelligent identification of structural damage mostly accentuates the capability of neural networks to recognize damage patterns but somewhat makes relatively less noticeable to the effect of structural dynamics in delivering an efficient damage identification scheme as well as damage features. At present, modal frequencies are often selected as inputs to neural networks for predicting location and/or severity of damage; however, this practice lacks solid support by structural dynamics. For that reason, this study investigates the use of structural dynamics-guided neural networks for damage identification. Structural dynamic analysis indicates that there are explicit relations between damage location and the ratio of changes of modal frequencies (RCMFs), and between damage severity and the change in modal frequencies (CMFs), before and after damage. These relations lay the foundation for creating a hierarchical neural-networks scheme to identify damage: using the RCMFs as inputs and damage location as the output to frame neural networks for locating damage; using the CMFs as inputs and damage severity as the output to establish neural networks for quantifying damage. This scheme features the guidance of structural dynamics on choosing damage indices and establishing a neural networks structure. The proposed scheme is numerically verified by identifying the location and severity of cracks in beams, with emphasis on noise robustness, and it is further experimentally validated using a set of steel beams with a through-width transverse crack, showing high accuracy and reliability in damage location and quantification

The Resonate-and-fire Neuron: Time Dependent and Frequency Selective Neurons in Neural Networks

The means through which the nervous system perceives its environment is one of the most fascinating questions in contemporary science. Our endeavors to comprehend the principles of neural science provide an instance of how biological processes may inspire novel methods in mathematical modeling and engineering. The application ofmathematical models towards understanding neural signals and systems represents a vibrant field of research that has spanned over half a century. During this period, multiple approaches to neuronal modeling have been adopted, and each approach is adept at elucidating a specific aspect of nervous system function. Thus while bio-physical models have strived to comprehend the dynamics of actual physical processes occurring within a nerve cell, the phenomenological approach has conceived models that relate the ionic properties of nerve cells to transitions in neural activity. Further-more, the field of neural networks has endeavored to explore how distributed parallel processing systems may become capable of storing memory. Through this project, we strive to explore how some of the insights gained from biophysical neuronal modeling may be incorporated within the field of neural net-works. We specifically study the capabilities of a simple neural model, the Resonate-and-Fire (RAF) neuron, whose derivation is inspired by biophysical neural modeling. While reflecting further biological plausibility, the RAF neuron is also analytically tractable, and thus may be implemented within neural networks. In the following thesis, we provide a brief overview of the different approaches that have been adopted towards comprehending the properties of nerve cells, along with the framework under which our specific neuron model relates to the field of neuronal modeling. Subsequently, we explore some of the time-dependent neurocomputational capabilities of the RAF neuron, and we utilize the model to classify logic gates, and solve the classic XOR problem. Finally we explore how the resonate-and-fire neuron may be implemented within neural networks, and how such a network could be adapted through the temporal backpropagation algorithm