MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superposition

With the advent of deep learning, progressively larger neural networks have been designed to solve complex tasks. We take advantage of these capacity-rich models to lower the cost of inference by exploiting computation in superposition. To reduce the computational burden per input, we propose Multiple-Input-Multiple-Output Neural Networks (MIMONets) capable of handling many inputs at once. MIMONets augment various deep neural network architectures with variable binding mechanisms to represent an arbitrary number of inputs in a compositional data structure via fixed-width distributed representations. Accordingly, MIMONets adapt nonlinear neural transformations to process the data structure holistically, leading to a speedup nearly proportional to the number of superposed input items in the data structure. After processing in superposition, an unbinding mechanism recovers each transformed input of interest. MIMONets also provide a dynamic trade-off between accuracy and throughput by an instantaneous on-demand switching between a set of accuracy-throughput operating points, yet within a single set of fixed parameters. We apply the concept of MIMONets to both CNN and Transformer architectures resulting in MIMOConv and MIMOFormer, respectively. Empirical evaluations show that MIMOConv achieves about 2-4 x speedup at an accuracy delta within [+0.68, -3.18]% compared to WideResNet CNNs on CIFAR10 and CIFAR100. Similarly, MIMOFormer can handle 2-4 inputs at once while maintaining a high average accuracy within a [-1.07, -3.43]% delta on the long range arena benchmark. Finally, we provide mathematical bounds on the interference between superposition channels in MIMOFormer. Our code is available at https://github.com/IBM/multiple-input-multiple-output-nets.Comment: accepted in NeurIPS 202

Biologically inspired distributed machine cognition: a new formal approach to hyperparallel computation

The irresistable march toward multiple-core chip technology presents currently intractable pdrogramming challenges. High level mental processes in many animals, and their analogs for social structures, appear similarly massively parallel, and recent mathematical models addressing them may be adaptable to the multi-core programming problem

Institutional Cognition

We generalize a recent mathematical analysis of Bernard Baars' model of human consciousness to explore analogous, but far more complicated, phenomena of institutional cognition. Individual consciousness is limited to a single, tunable, giant component of interacting cogntivie modules, instantiating a Global Workspace. Human institutions, by contrast, seem able to multitask, supporting several such giant components simultaneously, although their behavior remains constrained to a topology generated by cultural context and by the path-dependence inherent to organizational history. Surprisingly, such multitasking, while clearly limiting the phenomenon of inattentional blindness, does not eliminate it. This suggests that organizations (or machines) explicitly designed along these principles, while highly efficient at certain sets of tasks, would still be subject to analogs of the subtle failure patterns explored in Wallace (2005b, 2006). We compare and contrast our results with recent work on collective efficacy and collective consciousness

Generalized inattentional blindness from a Global Workspace perspective

We apply Baars' Global Workspace model of consciousness to inattentional blindness, using the groupoid network method of Stewart et al. to explore modular structures defined by information measures associated with cognitive process. Internal cross-talk breaks the fundamental groupoid symmetry, and, if sufficiently strong, creates, in a highly punctuated manner, a linked, shifting, giant component which instantiates the global workspace of consciousness. Embedding, exterior, information sources act as an external field which breaks the groupoid symmetry in a somewhat different manner, definng the slowly-acting contexts of Baars' theory and providing topological constraints on the manifestations of consciousness. This analysis significantly extends recent mathematical treatments of the global workspace, and identifies a shifting, topologically-determined syntactical and grammatical 'bottleneck' as a tunable rate distortion manifold which constrains what sensory or other signals can be brought to conscious attention, typically in a punctuated manner. Sensations outside the limits of that filter's syntactic 'bandpass' have lower probability of detection, regardless of their structure, accounting for generalized forms of inattentional blindness