Going beyond semantic image segmentation, towards holistic scene understanding, with associative hierarchical random fields

In this thesis we exploit the generality and expressive power of the Associative Hierarchical Random Field (AHRF) graphical model to take its use beyond that of semantic image segmentation, into object-classes, towards a framework for holistic scene understanding. We provide a working definition for the holistic approach to scene understanding, which allows for the integration of existing, disparate, applications into an unifying ensemble. We believe that modelling such an ensemble as an AHRF is both a principled and pragmatic solution. We present a hierarchy that shows several methods for fusing applications together with the AHRF graphical model. Each of the three; feature, potential and energy, layers subsumes its predecessor in generality and together give rise to many options for integration. With applications on street scenes we demonstrate an implementation of each layer. The first layer application joins appearance and geometric features. For our second layer we implement a things and stuff co-junction using higher order AHRF potentials for object detectors, with the goal of answering the classic questions: What? Where? and How many? A holistic approach to recognition-and-reconstruction is realised within our third layer by linking two energy based formulations of both applications. Each application is evaluated qualitatively and quantitatively. In all cases our holistic approach shows improvement over baseline methods

Binding and Normalization of Binary Sparse Distributed Representations by Context-Dependent Thinning

Distributed representations were often criticized as inappropriate for encoding of data with a complex structure. However Plate's Holographic Reduced Representations and Kanerva's Binary Spatter Codes are recent schemes that allow on-the-fly encoding of nested compositional structures by real-valued or dense binary vectors of fixed dimensionality. In this paper we consider procedures of the Context-Dependent Thinning which were developed for representation of complex hierarchical items in the architecture of Associative-Projective Neural Networks. These procedures provide binding of items represented by sparse binary codevectors (with low probability of 1s). Such an encoding is biologically plausible and allows a high storage capacity of distributed associative memory where the codevectors may be stored. In contrast to known binding procedures, Context-Dependent Thinning preserves the same low density (or sparseness) of the bound codevector for varied number of component codevectors. Besides, a bound codevector is not only similar to another one with similar component codevectors (as in other schemes), but it is also similar to the component codevectors themselves. This allows the similarity of structures to be estimated just by the overlap of their codevectors, without retrieval of the component codevectors. This also allows an easy retrieval of the component codevectors. Examples of algorithmic and neural-network implementations of the thinning procedures are considered. We also present representation examples for various types of nested structured data (propositions using role-filler and predicate-arguments representation schemes, trees, directed acyclic graphs) using sparse codevectors of fixed dimension. Such representations may provide a fruitful alternative to the symbolic representations of traditional AI, as well as to the localist and microfeature-based connectionist representations

Non-Convex Multi-species Hopfield models

In this work we introduce a multi-species generalization of the Hopfield model for associative memory, where neurons are divided into groups and both inter-groups and intra-groups pair-wise interactions are considered, with different intensities. Thus, this system contains two of the main ingredients of modern Deep neural network architectures: Hebbian interactions to store patterns of information and multiple layers coding different levels of correlations. The model is completely solvable in the low-load regime with a suitable generalization of the Hamilton-Jacobi technique, despite the Hamiltonian can be a non-definite quadratic form of the magnetizations. The family of multi-species Hopfield model includes, as special cases, the 3-layers Restricted Boltzmann Machine (RBM) with Gaussian hidden layer and the Bidirectional Associative Memory (BAM) model.Comment: This is a pre-print of an article published in J. Stat. Phy

Emergence of Self-Organized Symbol-Based Communication \ud in Artificial Creatures

In this paper, we describe a digital scenario where we simulated the emergence of self-organized symbol-based communication among artificial creatures inhabiting a \ud virtual world of unpredictable predatory events. In our experiment, creatures are autonomous agents that learn symbolic relations in an unsupervised manner, with no explicit feedback, and are able to engage in dynamical and autonomous communicative interactions with other creatures, even simultaneously. In order to synthesize a behavioral ecology and infer the minimum organizational constraints for the design of our creatures, \ud we examined the well-studied case of communication in vervet monkeys. Our results show that the creatures, assuming the role of sign users and learners, behave collectively as a complex adaptive system, where self-organized communicative interactions play a \ud major role in the emergence of symbol-based communication. We also strive in this paper for a careful use of the theoretical concepts involved, including the concepts of symbol and emergence, and we make use of a multi-level model for explaining the emergence of symbols in semiotic systems as a basis for the interpretation of inter-level relationships in the semiotic processes we are studying

Meta-stable states in the hierarchical Dyson model drive parallel processing in the hierarchical Hopfield network

In this paper we introduce and investigate the statistical mechanics of hierarchical neural networks: First, we approach these systems \`a la Mattis, by thinking at the Dyson model as a single-pattern hierarchical neural network and we discuss the stability of different retrievable states as predicted by the related self-consistencies obtained from a mean-field bound and from a bound that bypasses the mean-field limitation. The latter is worked out by properly reabsorbing fluctuations of the magnetization related to higher levels of the hierarchy into effective fields for the lower levels. Remarkably, mixing Amit's ansatz technique (to select candidate retrievable states) with the interpolation procedure (to solve for the free energy of these states) we prove that (due to gauge symmetry) the Dyson model accomplishes both serial and parallel processing. One step forward, we extend this scenario toward multiple stored patterns by implementing the Hebb prescription for learning within the couplings. This results in an Hopfield-like networks constrained on a hierarchical topology, for which, restricting to the low storage regime (where the number of patterns grows at most logarithmical with the amount of neurons), we prove the existence of the thermodynamic limit for the free energy and we give an explicit expression of its mean field bound and of the related improved boun

Parallel processing in immune networks

In this work we adopt a statistical mechanics approach to investigate basic, systemic features exhibited by adaptive immune systems. The lymphocyte network made by B-cells and T-cells is modeled by a bipartite spin-glass, where, following biological prescriptions, links connecting B-cells and T-cells are sparse. Interestingly, the dilution performed on links is shown to make the system able to orchestrate parallel strategies to fight several pathogens at the same time; this multitasking capability constitutes a remarkable, key property of immune systems as multiple antigens are always present within the host. We also define the stochastic process ruling the temporal evolution of lymphocyte activity, and show its relaxation toward an equilibrium measure allowing statistical mechanics investigations. Analytical results are compared with Monte Carlo simulations and signal-to-noise outcomes showing overall excellent agreement. Finally, within our model, a rationale for the experimentally well-evidenced correlation between lymphocytosis and autoimmunity is achieved; this sheds further light on the systemic features exhibited by immune networks.Comment: 21 pages, 9 figures; to appear in Phys. Rev.