Training neural networks to encode symbols enables combinatorial generalization

Combinatorial generalization - the ability to understand and produce novel combinations of already familiar elements - is considered to be a core capacity of the human mind and a major challenge to neural network models. A significant body of research suggests that conventional neural networks can't solve this problem unless they are endowed with mechanisms specifically engineered for the purpose of representing symbols. In this paper we introduce a novel way of representing symbolic structures in connectionist terms - the vectors approach to representing symbols (VARS), which allows training standard neural architectures to encode symbolic knowledge explicitly at their output layers. In two simulations, we show that neural networks not only can learn to produce VARS representations, but in doing so they achieve combinatorial generalization in their symbolic and non-symbolic output. This adds to other recent work that has shown improved combinatorial generalization under specific training conditions, and raises the question of whether specific mechanisms or training routines are needed to support symbolic processing

Generalization of form in visual pattern classification.

Human observers were trained to criterion in classifying compound Gabor signals with sym- metry relationships, and were then tested with each of 18 blob-only versions of the learning set. General- ization to dark-only and light-only blob versions of the learning signals, as well as to dark-and-light blob versions was found to be excellent, thus implying virtually perfect generalization of the ability to classify mirror-image signals. The hypothesis that the learning signals are internally represented in terms of a 'blob code' with explicit labelling of contrast polarities was tested by predicting observed generalization behaviour in terms of various types of signal representations (pixelwise, Laplacian pyramid, curvature pyramid, ON/OFF, local maxima of Laplacian and curvature operators) and a minimum-distance rule. Most representations could explain generalization for dark-only and light-only blob patterns but not for the high-thresholded versions thereof. This led to the proposal of a structure-oriented blob-code. Whether such a code could be used in conjunction with simple classifiers or should be transformed into a propo- sitional scheme of representation operated upon by a rule-based classification process remains an open question

A Hybrid Neural Network and Virtual Reality System for Spatial Language Processing

This paper describes a neural network model for the study of spatial language. It deals with both geometric and functional variables, which have been shown to play an important role in the comprehension of spatial prepositions. The network is integrated with a virtual reality interface for the direct manipulation of geometric and functional factors. The training uses experimental stimuli and data. Results show that the networks reach low training and generalization errors. Cluster analyses of hidden activation show that stimuli primarily group according to extra-geometrical variables

Categorical invariance and structural complexity in human concept learning

An alternative account of human concept learning based on an invariance measure of the categorical\ud stimulus is proposed. The categorical invariance model (CIM) characterizes the degree of structural\ud complexity of a Boolean category as a function of its inherent degree of invariance and its cardinality or\ud size. To do this we introduce a mathematical framework based on the notion of a Boolean differential\ud operator on Boolean categories that generates the degrees of invariance (i.e., logical manifold) of the\ud category in respect to its dimensions. Using this framework, we propose that the structural complexity\ud of a Boolean category is indirectly proportional to its degree of categorical invariance and directly\ud proportional to its cardinality or size. Consequently, complexity and invariance notions are formally\ud unified to account for concept learning difficulty. Beyond developing the above unifying mathematical\ud framework, the CIM is significant in that: (1) it precisely predicts the key learning difficulty ordering of\ud the SHJ [Shepard, R. N., Hovland, C. L.,&Jenkins, H. M. (1961). Learning and memorization of classifications.\ud Psychological Monographs: General and Applied, 75(13), 1-42] Boolean category types consisting of three\ud binary dimensions and four positive examples; (2) it is, in general, a good quantitative predictor of the\ud degree of learning difficulty of a large class of categories (in particular, the 41 category types studied\ud by Feldman [Feldman, J. (2000). Minimization of Boolean complexity in human concept learning. Nature,\ud 407, 630-633]); (3) it is, in general, a good quantitative predictor of parity effects for this large class of\ud categories; (4) it does all of the above without free parameters; and (5) it is cognitively plausible (e.g.,\ud cognitively tractable)

The Future of Human-Artificial Intelligence Nexus and its Environmental Costs

The environmental costs and energy constraints have become emerging issues for the future development of Machine Learning (ML) and Artificial Intelligence (AI). So far, the discussion on environmental impacts of ML/AI lacks a perspective reaching beyond quantitative measurements of the energy-related research costs. Building on the foundations laid down by Schwartz et al., 2019 in the GreenAI initiative, our argument considers two interlinked phenomena, the gratuitous generalisation capability and the future where ML/AI performs the majority of quantifiable inductive inferences. The gratuitous generalisation capability refers to a discrepancy between the cognitive demands of a task to be accomplished and the performance (accuracy) of a used ML/AI model. If the latter exceeds the former because the model was optimised to achieve the best possible accuracy, it becomes inefficient and its operation harmful to the environment. The future dominated by the non-anthropic induction describes a use of ML/AI so all-pervasive that most of the inductive inferences become furnished by ML/AI generalisations. The paper argues that the present debate deserves an expansion connecting the environmental costs of research and ineffective ML/AI uses (the issue of gratuitous generalisation capability) with the (near) future marked by the all-pervasive Human-Artificial Intelligence Nexus