Simplification of rules extracted from neural networks

Artificial neural networks (ANNs) have been proven to be successful general machine learning techniques for, amongst others, pattern recognition and classification. Realworld problems in agriculture (soybean, tea), medicine (cancer, cardiology, mammograms) and finance (credit rating, stock market) are successfully solved using ANNs. ANNs model biological neural systems. A biological neural system consists of neurons interconnected through neural synapses. These neurons serve as information processing units. Synapses carrt information to the neurons, which then processes or responds to the data by sending a signal to the next level of neurons. Information is strengthened or lessened according to the sign ..and magnitude of the weight associated with the connection. An ANN consists of cell-like entities called units (also called artificial neurons) and weighted connections between these units referred to as links. ANNs can be viewed as a directed graph with weighted connections. An unit belongs to one of three groups: input, hidden or output. Input units receive the initial training patterns, which consist of input attributes and the associated target attributes, from the environment. Hidden units do not interact with the environment whereas output units presents the results to the environment. Hidden and output units compute an output ai which is a function f of the sum of its input weights w; multiplied by the output x; of the units j in the preceding layer, together with a bias term fh that acts as a threshold for the unit. The output ai for unit i with n input units is calculated as ai = f("f:,'J= 1 x;w; - 8i ). Training of the ANN is done by adapting the weight values for each unit via a gradient search. Given a set of input-target pairs, the ANN learns the functional relationship between the input and the target. A serious drawback of the neural network approach is the difficulty to determine why a particular conclusion was reached. This is due to the inherit 'black box' nature of the neural network approach. Neural networks rely on 'raw' training data to learn the relationships between the initial inputs and target outputs. Knowledge is encoded in a set of numeric weights and biases. Although this data driven aspect of neural network allows easy adjustments when change of environment or events occur, it is difficult to interpret numeric weights, making it difficult for humans to understand. Concepts represent by symbolic learning algorithms are intuitive and therefore easily understood by humans [Wnek 1994). One approach to understanding the representations formed by neural networks is to extract such symbolic rules from networks. Over the last few years, a number of rule extraction methods have been reported (Craven 1993, Fu 1994). There are some general assumptions that these algorithms adhere to. The first assumption that most rule extraction algorithms make, is that non-input units are either maximally active (activation near 1) or inactive (activation near 0). This Boolean valued activation is approximated by using the standard logistic activation function /(z) = 1/( 1 + e-•z ) and setting s 5.0. The use of the above function parameters guarantees that non-input units always have non-negative activations in the range [0,1). The second underlying premise of rule extraction is that each hidden and output unit implements a symbolic rule. The concept associated with each unit is the consequent of the rule, and certain subsets of the input units represent the antecedent of the rule. Rule extraction algorithms search for those combinations of input values to a particular hidden or output unit that results in it having an optimal (near-one) activation. Here, rule extraction methods exploit a very basic principle of biological neural networks. That is, if the sum of its weighted inputs exceeds a certain threshold, then the biological neuron fires [Fu 1994). This condition is satisfied when the sum of the weighted inputs exceeds the bias, where (E'Jiz,=::l w; > 9i)• It has been shown that most concepts described by humans usally can be expressed as production rules in disjunctive normal form (DNF) notation. Rules expressed in this notation are therefore highly comprehensible and intuitive. In addition, the number of production rules may be reduced and the structure thereof simplified by using propositional logic. A method that extracts production rules in DNF is presented [Viktor 1995). The basic idea of the method is the use of equivalence classes. Similarly weighted links are grouped into a cluster, the assumption being that individual weights do not have unique importance. Clustering considerably reduces the combinatorics of the method as opposed to previously reported approaches. Since the rules are in a logically manipulatable form, significant simplifications in the structure thereof can be obtained, yielding a highly reduced and comprehensible set of rules. Experimental results have shown that the accuracy of the extracted rules compare favourably with the CN2 [Clark 1989] and C4.5 [Quinlan 1993] symbolic rule extraction methods. The extracted rules are highly comprehensible and similar to those extracted by traditional symfiolic methods

Using Neural Networks for Relation Extraction from Biomedical Literature

Using different sources of information to support automated extracting of relations between biomedical concepts contributes to the development of our understanding of biological systems. The primary comprehensive source of these relations is biomedical literature. Several relation extraction approaches have been proposed to identify relations between concepts in biomedical literature, namely, using neural networks algorithms. The use of multichannel architectures composed of multiple data representations, as in deep neural networks, is leading to state-of-the-art results. The right combination of data representations can eventually lead us to even higher evaluation scores in relation extraction tasks. Thus, biomedical ontologies play a fundamental role by providing semantic and ancestry information about an entity. The incorporation of biomedical ontologies has already been proved to enhance previous state-of-the-art results.Comment: Artificial Neural Networks book (Springer) - Chapter 1

An Overview of the Use of Neural Networks for Data Mining Tasks

In the recent years the area of data mining has experienced a considerable demand for technologies that extract knowledge from large and complex data sources. There is a substantial commercial interest as well as research investigations in the area that aim to develop new and improved approaches for extracting information, relationships, and patterns from datasets. Artificial Neural Networks (NN) are popular biologically inspired intelligent methodologies, whose classification, prediction and pattern recognition capabilities have been utilised successfully in many areas, including science, engineering, medicine, business, banking, telecommunication, and many other fields. This paper highlights from a data mining perspective the implementation of NN, using supervised and unsupervised learning, for pattern recognition, classification, prediction and cluster analysis, and focuses the discussion on their usage in bioinformatics and financial data analysis tasks

Biologically Inspired Approaches to Automated Feature Extraction and Target Recognition

Ongoing research at Boston University has produced computational models of biological vision and learning that embody a growing corpus of scientific data and predictions. Vision models perform long-range grouping and figure/ground segmentation, and memory models create attentionally controlled recognition codes that intrinsically cornbine botton-up activation and top-down learned expectations. These two streams of research form the foundation of novel dynamically integrated systems for image understanding. Simulations using multispectral images illustrate road completion across occlusions in a cluttered scene and information fusion from incorrect labels that are simultaneously inconsistent and correct. The CNS Vision and Technology Labs (cns.bu.edulvisionlab and cns.bu.edu/techlab) are further integrating science and technology through analysis, testing, and development of cognitive and neural models for large-scale applications, complemented by software specification and code distribution.Air Force Office of Scientific Research (F40620-01-1-0423); National Geographic-Intelligence Agency (NMA 201-001-1-2016); National Science Foundation (SBE-0354378; BCS-0235298); Office of Naval Research (N00014-01-1-0624); National Geospatial-Intelligence Agency and the National Society of Siegfried Martens (NMA 501-03-1-2030, DGE-0221680); Department of Homeland Security graduate fellowshi