Christensen, Bjørn Klint

This paper describes the implementation of a three-layer feedforward backpropagation neural network. The paper does not explain feedforward, backpropagation or what a neural network is. It is assumed, that the reader knows all this. If not please read chapters 2, 8 and 9 in Parallel Distributed Processing, by David Rummelhart (Rummelhart 1986) for an easy-to-read introduction.What the paper does explain is how a matrix representation of a neural net allows for a very simple implementation.The matrix representation is introduced in (Rummelhart 1986, chapter 9), but only for a two-layer linear network and the feedforward algorithm. This paper develops the idea further to three-layer non-linear networks and the backpropagation algorithm.Figure 1 shows the layout of a three-layer network. There are I input nodes, J hidden nodes and K output nodes all indexed from 0. Bias-node for the hidden nodes is called iI, and bias-node for the output nodes is called hJ

Online Research Database In Technology

                       General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.  • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal   If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.     Downloaded from orbit.dtu.dk on: Dec 18, 2017Matrix representation of a Neural NetworkChristensen, Bjørn KlintPublication date:2003Document VersionPublisher's PDF, also known as Version of recordLink back to DTU OrbitCitation (APA):Christensen, B. K. (2003). Matrix representation of a Neural Network. Technical University of Denmark (DTU). 1      Matrix representation of a Neural Network  by Bjørn Christensen June 2003     Contents 1 INTRODUCTION 2 2 THE WEIGHT MATRIX 3 2.1 Feedforward 4 2.2 Backpropagation 5 2.3 Squared Error 7 3 REFERENCES 7   2  1 Introduction  This paper describes the implementation of a three-layer feedforward backpropagation neural network.  The paper does not explain feedforward, backpropagation or what a neural network is. It is assumed, that the reader knows all this. If not please read chapters 2, 8 and 9 in Parallel Distributed Processing, by David Rummelhart (Rummelhart 1986) for an easy-to-read introduction.  What the paper does explain is how a matrix representation of a neural net allows for a very simple implementation.  The matrix representation is introduced in (Rummelhart 1986, chapter 9), but only for a two-layer linear network and the feedforward algorithm. This paper develops the idea further to three-layer non-linear networks and the backpropagation algorithm.   h0h1hJ=1hJ-1i0i1iI-1iI=1o0o1oK-1input nodes hidden nodes output nodes Figure 1  Figure 1 shows the layout of a three-layer network. There are I input nodes, J hidden nodes and K output nodes all indexed from 0. Bias-node for the hidden nodes is called iI, and bias-node for the output nodes is called hJ.   3  2 The Weight Matrix In the matrix representation the input, hidden and output nodes are represented by three vectors i, h and o respectively. The weights connecting each layer are represented by a matrix. V connects the input layer with the hidden layer, and W connects the hidden layer with the output layer. See Figure 2.   input nodeshidden nodesoutput nodesi0 i1 iI-1 iI=1h0h1hJ-1hJ=1o0o1oK-1v0,0 v0,1 v0,I-1 v0,Iw0,0 w0,1 w0,J-1 w0,Jv1,0 v1,1 v1,I-1 v1,IvJ-1,0 vJ-1,1 vJ-1,I-1 vJ-1,Iw1,0 w1,1 w1,J-1 w1,JwK-1,0 wK-1,1 wK-1,J-1 wK-1,J Figure 2  Definitions: i  - input nodes vector,  dimension = I,  indexed by i[0,I-1] h - hidden nodes vector,  dimension = J,  indexed by j[0,J-1] o - output nodes vector,   dimension = K,  indexed by k[0,K-1]  ib  - input nodes vector incl bias, dimension = I+1,  indexed by i[0,I] hb - hidden nodes vector incl bias, dimension = J+1,  indexed by j[0,J]  V – weight matrix from Ib to h, dimension = J(I+1), Vj,i weight ibi  hj W – weight matrix from hb to o, dimension = K(J+1),Wk,j weight hbj  ok Bold denotes a vector or a matrix in the text.   4 2.1 Feedforward  The feedforward algorithm produces an output vector given an input vector. As described in (Callan 1999, chapter 1), the input to hidden node hj is 1I0ijiiIj,j ViVnet  (1)  The naming convention of Callan is changed to match the definitions in this paper which follows (Rummelhart 1986).   Note that Callan has interchanged the indices in the weight matrix.  The value of hidden node hj is )f(netjj h  (2)  Where f() is the activation function defined as xx-e11)(f  (3)   Using matrix-vector multiplication, the value of all hidden nodes h can be calculated in a single operation )F( ibVh   (4)  Where F() is the vector function that takes f() on all elements of it’s argument. Similarly the output vector is calculated from )F( hbWo   (5)   ib and hb are produced from      1                ,1..0ifor     ii  II ibiib  (6)       1             ,1..0jfor     jj  JJ hbhhb  (7)    5  2.2 Backpropagation The backpropagation algorithm calculates the changes of the weights based on the error between the output o as calculated the by the feedforward algorithm, and the desired target output value t.  Definition: t - target vector,    dimension = K,  indexed by k[0,K-1]  From equation 11 in (Rummelhart 1986, chapter 8), we have the change of each weight: jkkj hηδΔW   (8)  Where  is the learning rate, k is the error signal form output node ok, and hj is the value of node hj. This formula applys to all weights in the network. The error signal from the output layer and the hidden layer are calculated differently. Rummelhart defines the error signal as follows  Error signal from output node k )o)(to(1oδo kkkkk   (9)  Error signal from hidden node j 1-K0kkjkjjj Wδo)h(1hδh  (10)   Definition: o- error signal from output nodes dimension = K,  indexed by k[0,K-1] h- error signal from hidden nodes dimension = J,  indexed by j[0,J-1]   Now we are ready to calculate the weight change W for all weights in a singe operation:  hboWδηhhhhδoδoδoηhδohδohδohδohδohδohδohδohδohδohδohδoηWWWWWWWWWWWWηΔJ1-J101-K10J1-K1-J1-K11-K01-KJ11-J11101J01-J01000J1,-K1-J1,-K1,1-K1,0-KJ1,1-J1,1,11,0J0,1-J0,0,10,0 (11)   So W is  times the outer product of o and hb.  is the operator for the outer product. Similarly for V we get ibhV  δηΔ  (12)    6 We need to express o and h in vector form. Let us define a vector multiplication operator % that multiplies the components of vector x and y as follows: n*n2*21*1%yxyxyxyx  (13)   Calculating o is easy: )()( %% oto1oδo   (14)  Where 1 is a vector of 1’s with same dimension as o.  Calculating h requires a little more work. Let us define a vector S  as: oW δ Ts  (15)  WT is the transpose of W. Expanding W and o gives us: 1-K0kkjk...   ...   ...   ...   Wδo               WδoWδoWδoWδoWδoWδoWδoWδoWδoWδoWδoWδoδoδoδoWWWWWWWWWWWWδoδoδoWWWWWWWWWWWWjswheresssssTJ1-J10J1,-K1-K  J1,1J0,01-J1,-K1-K 1-J1, 11-J0,01,1-K1-K 1,1 10,101,0-K1-K  1,010,001-K10J1,-KJ1,J0,1-J1,-K1-J1,1-J0,1,1-K1,10,11,0-K1,00,01-K10J1,-K1-J1,-K1,1-K1,0-KJ1,1-J1,1,11,0J0,1-J0,0,10,0 (16)   The component sj is operand in the error signal hj. Note that we do not need an error signal from hJ since it is a bias node. Therefore we skip the last component of s, and create a vector s defined as:     1..0jfor  s    jj  Js  (17)  Dimension of s is J. Finally we can write h as: sh)(1hh %%δ   (18)  Here 1 has same dimension as h.   Let us sum up the important equations of the backpropagation algorithm: Error signal from output nodes )()(δ %% oto1oo   (19)  Changes of weights connected to output nodes hboW  δη  (20)  Propagated errors from output nodes oW δ Ts  (21)  Error signal from hidden nodes sh)(1hh %%δ   (22)  Changes of weights connected to hidden nodes ibhV  δηΔ  (23)    7  2.3 Squared Error The squared error is a measure of the correctness of the network. It is calculated for each input pattern p. Rummelhart defines the squared error as: 1K0k2kkp )o(t21E  (24)  In vector notation using the inner product this is o)(to)(t 21Ep  (25)   Summing the squared errors for all patterns, we get the squared error E for an epoch n1ppEE  (26)  Where n is the number of patterns in the training set.   3 References  Rummelhart 1986 Parallel Distributed Programming: Explorations in the Microstructure of Cognition. Vol 1    David E Rummelhart, James L McClelland MIT-press 1986  Callan 1999  The Essence of Nearal Networks    Robert Callan    Prentice Hall 1999 

Matrix representation of a Neural Network

http://orbit.dtu.dk/ws/files/131600547/Matrix_representation_of_a_Neural_Network.pdf

Matrix representation of a Neural Network

Abstract

Similar works

Full text

Available Versions

Online Research Database In Technology