Basic binary relations such as equality and inequality are fundamental to relational data structures. Neural networks should learn such relations and generalise to new unseen data. We show in this study, however, that this generalisation fails with standard feed-forward networks on binary vectors. Even when trained with maximal training data, standard networks do not reliably detect equality.



We introduce differential rectifier (DR) units that we add to the network in different configurations. The DR units create an inductive bias in the networks, so that they do learn to generalise, even from small numbers of examples and we have not found any negative effect of their inclusion in the network. Given the fundamental nature of these relations, we hypothesize that feed-forward neural network learning benefits from inductive bias in other relations as well. Consequently, the further development of suitable inductive biases will be beneficial to many tasks in relational learning with neural networks

Kopparti, R. M.

Weyde, T.

English

arXiv

City Research Online

              City, University of London Institutional RepositoryCitation: Weyde, T. ORCID: 0000-0001-8028-9905 and Kopparti, R. M. (2018). Feed-Forward Neural Networks Need Inductive Bias to Learn Equality Relations. Paper presented at the 32nd Conference on Neural Information Processing Systems (NIPS 2018), 2-8 Dec 2018, Montreal, Canada. This is the accepted version of the paper. This version of the publication may differ from the final published version. Permanent repository link:  https://openaccess.city.ac.uk/id/eprint/23876/Link to published version: Copyright and reuse: City Research Online aims to make research outputs of City, University of London available to a wider audience. Copyright and Moral Rights remain with the author(s) and/or copyright holders. URLs from City Research Online may be freely distributed and linked to.City Research Online:            http://openaccess.city.ac.uk/            publications@city.ac.ukCity Research OnlineFeed-Forward Neural Networks Need Inductive Biasto Learn Equality RelationsTillman WeydeDepartment of Computer ScienceCity, University of LondonLondon, United Kigndomt.e.weyde@city.ac.ukRadha Manisha KoppartiDepartment of Computer ScienceCity, University of LondonLondon, United Kingdomradha.kopparti@city.ac.ukAbstractBasic binary relations such as equality and inequality are fundamental to relationaldata structures. Neural networks should learn such relations and generalise tonew unseen data. We show in this study, however, that this generalisation failswith standard feed-forward networks on binary vectors. Even when trained withmaximal training data, standard networks do not reliably detect equality.We introduce differential rectifier (DR) units that we add to the network in differentconfigurations. The DR units create an inductive bias in the networks, so that theydo learn to generalise, even from small numbers of examples and we have notfound any negative effect of their inclusion in the network. Given the fundamentalnature of these relations, we hypothesize that feed-forward neural network learningbenefits from inductive bias in other relations as well. Consequently, the further de-velopment of suitable inductive biases will be beneficial to many tasks in relationallearning with neural networks.1 IntroductionBasic relations such as equality are fundamental to relational data structures. One goal of applyingneural networks to relational data is that the networks learn to infer these relational structure fromdata. Although equality is typically not learned from data, equality or approximate equality maybe embedded as part of other tasks. The modelling of equality is clearly in the hypothesis space offeed-forward neural networks (FFNNs) [1], but [2, 3] already highlighted that learning of identityrelationships with neural networks may not generalise to unseen data. Therefore, we see learning torecognise equality as relevant from a theoretical and practical perspective.In this study we test whether feed-forward networks learn equality as well as a numeric comparison,thresholded digit sum, and digit reversal of pairs of binary vectors and then generalise this to new datain different settings regarding the task, the amount of data provided, and the depth of the network.We find that the recognition of binary relations is not generalised reliably by feed-forward networks.To address this problem, we introduce an inductive bias with additional predefined network structures,that we call differential rectifier (DR) units. We find in our experiments that DR units induce reliableperfect generalisation for equality and all other tasks except in digit reversal.We see two questions that these results raise: First, which other relations neural networks do not learnand what that means for more complex tasks. Second, what kinds of inductive biases to design andhow to implement them.The remainder of this paper is organised as follows: Section 2 reviews related literature, Section 3introduces the task of learning vector equality and our DR units for inductive bias. Section 4 presentsthe experimental results and in Section 5 follow the conclusions of this paper.32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada.arXiv:1812.01662v1  [cs.LG]  4 Dec 20182 Related workIn relational learning, equality is often not learned from the data, with the exception of the workby [4] who learn to detect equality attributed of objects from images. Learning equality could beinteresting in the context of constraint learning [5] to learn when equality constraints should beregarded as satisfied. Another relevant area is rule learning and application, where soft unificationlike in [6] could be replaced with a learnt model.Since neural networks are currently by far the most popular machine learning method, it seemsof interest whether they can learn equality. There have been a number of theoretical contributionsshowing that feed-forward networks are universal approximators, most generally to our knowledgeby [1]. Presumably because of these results there was relatively little interest in the question whichfunctions neural networks can not learn. One of the few studies in this direction was undertakenin [2] in 1999, where a recurrent neural network failed to distinguish abstract patterns, based onequality relations between sequence elements, although seven-month-old infants showed the abilityto distinguish them after a few minutes of exposure. This was followed by an lively exchange on rulelearning by neural networks and in human language acquisition, where results by [7–9] could notbe reproduced by [10, 11] and [12] disputed claims by [11]. Other approaches, such as [13–15], usedifferent network architectures, problem formulations or evaluation methods.A more specific problem of learning equality relations was posed in [3] by showing that learning ofequality on even numbers does not transfer to odd numbers in binary representation. This relates tothe input neuron for the least significant bit not being set to 1 during training. Recently, [16] addressedthis specific problem with different approaches as an example for extrapolation and inductive biasesfor machine learning in natural language processing. However, they did not address the generalquestion of learning equality with neural networks.If standard neural networks do not generalise equality relations despite the solution being in theirhypothesis space, as we will show for FFNNs below, then the question is how we can enable thelearning of solutions that do generalise. Inductive biases as a solution can be realised in a number ofways and have been of increased interest recently [17, 18].3 Equality relation learningThe studies listed above motivated the approach taken here to study a reduced problem outsidecommon contexts such as image analysis or cognitive modelling: whether feed-forward neuralnetworks trained with back-propagation generally have the ability to learn equality relations andgeneralise to unseen data.The general task is to learn the relation between pairs of binary vectors. This leads to a binaryclassification of the pairs according to the equality or otherwise of its element vectors. We use astandard FFNN as sketched in Figure 1a). This network has 2n input neurons, where n is the vectordimensionality. The hidden layer has 10 neurons with ReLu activation. The output layer has twoneurons representing the two classes (equal/unequal), which use softmax activation. The traininguses the Adam optimiser[19] with cross-entropy loss.The data we train and test the network with is synthetically generated and we vary the type and thedistribution of the data in the experiments below. We are interested in how many training examples areneeded until the network learns to correctly classify pairs of equal vs. unequal vectors. This network,like the following ones have been implemented in Python using PyTorch (http://pytorch.org).Inductive bias creation with DR units In our model, we use differential rectifier (DR) units thatcompare input values by calculating the absolute difference: f(x, y) = |x− y|. We create one DRunit for every vector dimension with weights from the inputs to the DR units fixed at 1, thus learningthe suitable summation weights for the DRs is sufficient for creating a generalisable equality detector.We use two ways of integrating DR units into the neural networks: Early Fusion and Mid Fusion. InEarly Fusion, DR units are concatenated to input units 1b), and in Mid Fusion they are added to thehidden layer 1c). In both cases the existing input and hidden units are unchanged.2a) b) c)Figure 1: Network architectures: a) standard feed-forward network, DR integration with b) earlyfusion and c) mid fusion. The DR units receive their input in both cases from vec 1 and vec 2.Vector Dimensions Plain FFNN Early Fusion Mid Fusionn=2 52% 82% 100%n=3 55% 75% 100%n=5 37% 67% 100%n=10 52% 75% 100%n=30 65% 75% 100%n=100 75% 100% 100%Table 1: Accuracy of the different network types on pairs of vectors of different dimensions. Thejoint train and test data covers all possible equal vector pairs up to 1000, and a random selectionwhere there are more. Only the Mid Fusion architecture reaches reliable equality detection.4 Experiments and ResultsWe performed different sets of experiments using binary vectors for estimating vector equality inrelation to vector dimensionality, data size and dataset structure. We also use two additional tasks totest the effect of DR units in different contexts.Effect of network architecture and vector dimensionality We generate pairs of random binaryvectors with dimensionality n between 2 and 100 as shown in table 1. We use all the possible binaryvectors to generate equal pairs, i.e. 2n pairs, for n < 10 and a random selection of 1000 vectorsotherwise. We also generate the same number of randomly selected unequal vector pairs. Then weuse stratified sampling to split the data 75:25 into train and test set. The network is then trained for20 epochs, which led to convergence in all cases. We run 10 simulations for each configuration. Theaverage results are shown in Table 1.We see that the standard FFNNs never fully generalise, and in many cases barely exceed chancelevel (50%). The early fusion model improves results, but only reaches full performance for 100dimensions. The Mid Fusion reaches perfect test performance in all cases.For the plain FFNN, it looks like there is a trend towards better performance at higher dimensionality,but with the observed variation that may be coincidental. We did not perform an exhaustive gridsearch over all hyperparameters, but tested higher numbers of hidden layers (2,3), and larger hiddenlayers (20,30 neurons) without observing a significant change in the results.Effect of training data size We study here how much the performance depends on the trainingdata size. For this, we vary only the training data size and keep the test set and all other parametersconstant. We use training data sizes of 1% to 50% (in relation to the totally available data as definedabove) and the accuracy achieved in various conditions is plotted in Figure 2. It is worth noting, thatthe Mid Fusion network reaches 100% accuracy from 10% data size on while the FFNN shows onlysmall learning effects.Effect of vector coverage A possible hypothesis for the results of the FFNN is that the coverageof the vectors in the training set plays a role. To share vectors in equal pairs between training and testset would mean to train on the test data, but we created a training set that contains all vectors that3Figure 2: Accuracy of FFNN for 10 dimensional binary vectors after varying the distributions oftraining data from (1%-50%) keeping the testing data fixedType a) b) c) d) e)1) Plain FFNN 50% 52% 75% 77% 50%2) Early Fusion 75% 87% 92% 82% 55%3) Mid Fusion 100% 100% 100% 100% 58%Table 2: Test set accuracy of FFNN withour and with DR units for different vector coverage (a,b,see text for details) and for classification by c) numeric comparison (≥), d) digit sum ≥ 3, and e)inversion of digits.appear in the test set in the unequal pairs for n = 10. The results are shown in column a) of Table 2.We also created a training set where each vector appeared as above, but in both position 1 and 2. Theresults are shown in column b) of table 2. The results in both cases are similar to those without thisadditional coverage in Table 1.Other classification tasks We evaluate here whether the DR units have a negative effect on otherlearning tasks (using n = 3). We evaluated the networks on the classification by comparing the twovectors in the pair as binary numbers with results shown in column c) of Table 2. We also tested atask that is not a comparison of the two vectors in the pair, by calculating the digit sum. We classifyby checking if the digit sum is ≥ 3. In both c) and d) we see, that the performance is actually nothindered but helped by the DR units. We finally tested the task of recognising digit reversal (swappingleast with most significant bits), which DR units are not designed for, as they compare correspondingdigits. As we can see in column e), DR units do not deliver a perfect solution here, but still lead tosomewhat better results than a plain FFNN.5 ConclusionsIn this study we examined the learning behaviour of feed-forward neural networks in vector equalitydetection and observed that the networks do not generalise well to unseen data. We also had similarresults in other tasks like numeric inequality and sum of bits of binary vectors. We thereforeintroduced a simple modification to the network with differential rectifier (DR) units and noticedsubstantial improvements on unseen test data. This improvement is largely independent of vectordimension, data size and other parameters.The question why standard FFNNs do not learn vector equality relations in a generalisable way is arelevant one, and deserves further theoretical and empirical study. It is also important to investigatethe design of further measures for creating and controlling inductive biases in neural network learning,as we find that even relatively simple tasks like generalising equality require them.4References[1] Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforwardnetworks with a nonpolynomial activation function can approximate any function. Neuralnetworks, 6(6):861–867, 1993.[2] G. F. Marcus, S. Vijayan, S.B. Rao, and P.M. Vishton. Rule learning by seven-month-old infants.Science, 283, 5398:77–80, 1999.[3] G. F. Marcus. The algebraic mind: Integrating connectionism and cognitive science. CambridgeMIT Press, 2001.[4] Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, PeterBattaglia, and Tim Lillicrap. A simple neural network module for relational reasoning. InAdvances in neural information processing systems, pages 4967–4976, 2017.[5] Luc De Raedt, Andrea Passerini, and Stefano Teso. Learning constraints from examples. InProceedings in Thirty-Second AAAI Conference on Artificial Intelligence, 2018.[6] Andres Campero, Aldo Pareja, Tim Klinger, Josh Tenenbaum, and Sebastian Riedel. Logicalrule induction and theory learning using neural theorem proving, 2018.[7] Jeffrey Elman. Generalization, rules, and neural networks: A simulation of marcus et. al.https://crl.ucsd.edu/ elman/Papers/MVRVsimulation.html, 1999.[8] Gerry Altmann and Zoltan Dienes. Technical comment on rule learning by seven-month-oldinfants and neural networks. In Science, 284(5416)):875–875, 1999.[9] Thomas R. Shultz and Alan C. Bale. Neural network simulation of infant familiarizationto artificial sentences: Rule-like behavior without explicit rules and variables. Infancy, 2:4,501-536, DOI: 10.1207/S15327078IN020407, 2001.[10] Marlus Vilcu and Robert F Hadley. Generalization in simple recurrent networks. In Proceedingsof the Annual Meeting of the Cognitive Science Society, volume 23, 2001.[11] Marius Vilcu and Robert F Hadley. Two apparent ‘counterexamples’ to marcus: A closer look.Minds and Machines, 15(3-4):359–382, 2005.[12] Thomas R Shultz and Alan C Bale. Neural networks discover a near-identity relation todistinguish simple syntactic forms. Minds and Machines, 16(2):107–139, 2006.[13] Shastri and Chang. A spatiotemporal connectionist model of algebraic rule-learning. Interna-tional Computer Science Institute, pages TR–99–011, 1999.[14] P. Dominey and F. Ramus. Neural network processing of natural language: Isensitivity to serial,temporal and abstract structure of language in the infant. Language and Cognitive Processes,pages 15(1),87–127, 2000.[15] Raquel G. Alhama and Willem Zuidema. Pre-wiring and pre-training: What does a neuralnetwork need to learn truly general identity rules. CoCo at NIPS, 2016.[16] Jeff Mitchell, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Extrapolation in nlp.arXiv:1805.06648, 2018.[17] Jessica B Hamrick, Kelsey R Allen, Victor Bapst, Tina Zhu, Kevin R McKee, Joshua BTenenbaum, and Peter W Battaglia. Relational inductive bias for physical construction inhumans and machines. arXiv preprint arXiv:1806.01203, 2018.[18] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. InAdvances in Neural Information Processing Systems, pages 4077–4087, 2017.[19] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv:1412.6980, 2014.5

Feed-Forward Neural Networks Need Inductive Bias to Learn Equality Relations

https://openaccess.city.ac.uk/id/eprint/23876/1/1812.01662v1.pdf

Feed-Forward Neural Networks Need Inductive Bias to Learn Equality Relations

Abstract

Similar works

Full text

Available Versions

City Research Online