A General Framework for Robust G-Invariance in G-Equivariant Networks

We introduce a general method for achieving robust group-invariance in group-equivariant convolutional neural networks ($G$-CNNs), which we call the $G$-triple-correlation ($G$-TC) layer. The approach leverages the theory of the triple-correlation on groups, which is the unique, lowest-degree polynomial invariant map that is also complete. Many commonly used invariant maps - such as the max - are incomplete: they remove both group and signal structure. A complete invariant, by contrast, removes only the variation due to the actions of the group, while preserving all information about the structure of the signal. The completeness of the triple correlation endows the $G$-TC layer with strong robustness, which can be observed in its resistance to invariance-based adversarial attacks. In addition, we observe that it yields measurable improvements in classification accuracy over standard Max $G$-Pooling in $G$-CNN architectures. We provide a general and efficient implementation of the method for any discretized group, which requires only a table defining the group's product structure. We demonstrate the benefits of this method for $G$-CNNs defined on both commutative and non-commutative groups - $SO(2)$, $O(2)$, $SO(3)$, and $O(3)$ (discretized as the cyclic $C8$, dihedral $D16$, chiral octahedral $O$ and full octahedral $O_h$ groups) - acting on $\mathbb{R}^2$ and $\mathbb{R}^3$ on both $G$-MNIST and $G$-ModelNet10 datasets

Architectures of Topological Deep Learning: A Survey on Topological Neural Networks

The natural world is full of complex systems characterized by intricate relations between their components: from social interactions between individuals in a social network to electrostatic interactions between atoms in a protein. Topological Deep Learning (TDL) provides a comprehensive framework to process and extract knowledge from data associated with these systems, such as predicting the social community to which an individual belongs or predicting whether a protein can be a reasonable target for drug development. TDL has demonstrated theoretical and practical advantages that hold the promise of breaking ground in the applied sciences and beyond. However, the rapid growth of the TDL literature has also led to a lack of unification in notation and language across Topological Neural Network (TNN) architectures. This presents a real obstacle for building upon existing works and for deploying TNNs to new real-world problems. To address this issue, we provide an accessible introduction to TDL, and compare the recently published TNNs using a unified mathematical and graphical notation. Through an intuitive and critical review of the emerging field of TDL, we extract valuable insights into current challenges and exciting opportunities for future development

Bispectral Neural Networks

We present a neural network architecture, Bispectral Neural Networks (BNNs) for learning representations that are invariant to the actions of compact commutative groups on the space over which a signal is defined. The model incorporates the ansatz of the bispectrum, an analytically defined group invariant that is complete -- that is, it preserves all signal structure while removing only the variation due to group actions. Here, we demonstrate that BNNs are able to simultaneously learn groups, their irreducible representations, and corresponding complete invariant maps purely from the symmetries implicit in data. Further, we demonstrate that the completeness property endows these networks with strong adversarial robustness. This work establishes Bispectral Neural Networks as a powerful computational primitive for robust invariant representation learning

Identifying Interpretable Visual Features in Artificial and Biological Neural Systems

Single neurons in neural networks are often interpretable in that they represent individual, intuitively meaningful features. However, many neurons exhibit $\textit{mixed selectivity}$, i.e., they represent multiple unrelated features. A recent hypothesis proposes that features in deep networks may be represented in $\textit{superposition}$, i.e., on non-orthogonal axes by multiple neurons, since the number of possible interpretable features in natural data is generally larger than the number of neurons in a given network. Accordingly, we should be able to find meaningful directions in activation space that are not aligned with individual neurons. Here, we propose (1) an automated method for quantifying visual interpretability that is validated against a large database of human psychophysics judgments of neuron interpretability, and (2) an approach for finding meaningful directions in network activation space. We leverage these methods to discover directions in convolutional neural networks that are more intuitively meaningful than individual neurons, as we confirm and investigate in a series of analyses. Moreover, we apply the same method to three recent datasets of visual neural responses in the brain and find that our conclusions largely transfer to real neural data, suggesting that superposition might be deployed by the brain. This also provides a link with disentanglement and raises fundamental questions about robust, efficient and factorized representations in both artificial and biological neural systems

Exploring the hierarchical structure of human plans via program generation

Human behavior is inherently hierarchical, resulting from the decomposition of a task into subtasks or an abstract action into concrete actions. However, behavior is typically measured as a sequence of actions, which makes it difficult to infer its hierarchical structure. In this paper, we explore how people form hierarchically-structured plans, using an experimental paradigm that makes hierarchical representations observable: participants create programs that produce sequences of actions in a language with explicit hierarchical structure. This task lets us test two well-established principles of human behavior: utility maximization (i.e. using fewer actions) and minimum description length (MDL; i.e. having a shorter program). We find that humans are sensitive to both metrics, but that both accounts fail to predict a qualitative feature of human-created programs, namely that people prefer programs with reuse over and above the predictions of MDL. We formalize this preference for reuse by extending the MDL account into a generative model over programs, modeling hierarchy choice as the induction of a grammar over actions. Our account can explain the preference for reuse and provides the best prediction of human behavior, going beyond simple accounts of compressibility to highlight a principle that guides hierarchical planning

A Group Theoretic Framework for Neural Computation

How do networks of neurons compute stable representations of continually transforming sense data? This thesis aims to provide a mathematical foundation for this question based in group theory -- the mathematics that describes much of the transformation structure in natural visual signals. The primary contribution of this thesis is a framework that ties group theory to neurophysiologically plausible computational mechanisms by way of generalized Fourier analysis and its roots in group representation theory. I demonstrate mathematically and computationally how group representations may be instantiated in biological neurons to achieve robust invariance and equivariance to signal transforms. I further demonstrate how complete, selective group invariance can be achieved in a third-order generalization of the classical energy model of complex cells, based on the bispectrum. This perspective offers a reframing and generalization of canonical models of receptive fields in visual cortex, explains extra-classical results, and makes novel empirical predictions. I additionally demonstrate the utility of this mathematical framework for building artificial neural networks with improved robustness and invariance and apply these architectures to the problems of learning latent transformation structure from data and group-invariant denoising of signals in associative memory networks

Representations of Entropy and of the Relations Same and Different Early inHuman Development

Animals typically fail 2-item Relational Match to Sample (RMTS), whereas animals from pigeons through primatessucceed at 16-item RMTS. Furthermore, training on the 16-item arrays does not transfer to 2-item arrays in these non-humanspecies. Animal researchers conclude that success on 16-item RMTS reflects a perceptual property of the set, variabilityor entropy, rather than conceptual representations of the relations ‘same’ and ‘different’. Four experiments explore youngchildren’s ability to pass 2-item and 16-item RMTS. Like non-human animals, three- and four-year-olds fail 2-item RMTSwhile passing the16-item task. As with animals, training with 16-item cards does not facilitate success on 2-item RMTS infour- and five-year-olds. These data, as well as data from within the 16-item task, suggests that young children, like non-humananimals, rely on entropy in RMTS tasks. Data from 5 and 6-year-olds suggest a representational change late in the preschoolyears

Representational efficiency outweighs action efficiencyin human program induction

The importance of hierarchically structured representations fortractable planning has long been acknowledged. However, thequestions of how people discover such abstractions and how todefine a set of optimal abstractions remain open. This problemhas been explored in cognitive science in the problem solvingliterature and in computer science in hierarchical reinforce-ment learning. Here, we emphasize an algorithmic perspec-tive on learning hierarchical representations in which the ob-jective is to efficiently encode the structure of the problem, or,equivalently, to learn an algorithm with minimal length. Weintroduce a novel problem-solving paradigm that links prob-lem solving and program induction under the Markov Deci-sion Process (MDP) framework. Using this task, we target thequestion of whether humans discover hierarchical solutions bymaximizing efficiency in number of actions they generate or byminimizing the complexity of the resulting representation andfind evidence for the primacy of representational efficiency

Children’s representation of abstract relations in relational/array match-to-sample tasks

International audienc