Towards Building Deep Networks with Bayesian Factor Graphs

We propose a Multi-Layer Network based on the Bayesian framework of the Factor Graphs in Reduced Normal Form (FGrn) applied to a two-dimensional lattice. The Latent Variable Model (LVM) is the basic building block of a quadtree hierarchy built on top of a bottom layer of random variables that represent pixels of an image, a feature map, or more generally a collection of spatially distributed discrete variables. The multi-layer architecture implements a hierarchical data representation that, via belief propagation, can be used for learning and inference. Typical uses are pattern completion, correction and classification. The FGrn paradigm provides great flexibility and modularity and appears as a promising candidate for building deep networks: the system can be easily extended by introducing new and different (in cardinality and in type) variables. Prior knowledge, or supervised information, can be introduced at different scales. The FGrn paradigm provides a handy way for building all kinds of architectures by interconnecting only three types of units: Single Input Single Output (SISO) blocks, Sources and Replicators. The network is designed like a circuit diagram and the belief messages flow bidirectionally in the whole system. The learning algorithms operate only locally within each block. The framework is demonstrated in this paper in a three-layer structure applied to images extracted from a standard data set.Comment: Submitted for journal publicatio

Dirichlet belief networks for topic structure learning

Recently, considerable research effort has been devoted to developing deep architectures for topic models to learn topic structures. Although several deep models have been proposed to learn better topic proportions of documents, how to leverage the benefits of deep structures for learning word distributions of topics has not yet been rigorously studied. Here we propose a new multi-layer generative process on word distributions of topics, where each layer consists of a set of topics and each topic is drawn from a mixture of the topics of the layer above. As the topics in all layers can be directly interpreted by words, the proposed model is able to discover interpretable topic hierarchies. As a self-contained module, our model can be flexibly adapted to different kinds of topic models to improve their modelling accuracy and interpretability. Extensive experiments on text corpora demonstrate the advantages of the proposed model.Comment: accepted in NIPS 201

Deep Gaussian Processes

In this paper we introduce deep Gaussian process (GP) models. Deep GPs are a deep belief network based on Gaussian process mappings. The data is modeled as the output of a multivariate GP. The inputs to that Gaussian process are then governed by another GP. A single layer model is equivalent to a standard GP or the GP latent variable model (GP-LVM). We perform inference in the model by approximate variational marginalization. This results in a strict lower bound on the marginal likelihood of the model which we use for model selection (number of layers and nodes per layer). Deep belief networks are typically applied to relatively large data sets using stochastic gradient descent for optimization. Our fully Bayesian treatment allows for the application of deep models even when data is scarce. Model selection by our variational bound shows that a five layer hierarchy is justified even when modelling a digit data set containing only 150 examples.Comment: 9 pages, 8 figures. Appearing in Proceedings of the 16th International Conference on Artificial Intelligence and Statistics (AISTATS) 201

Recurrent dirichlet belief networks for interpretable dynamic relational data modelling

The Dirichlet Belief Network (DirBN) has been recently proposed as a promising approach in learning interpretable deep latent representations for objects. In this work, we leverage its interpretable modelling architecture and propose a deep dynamic probabilistic framework - the Recurrent Dirichlet Belief Network (Recurrent-DBN) - to study interpretable hidden structures from dynamic relational data. The proposed Recurrent-DBN has the following merits: (1) it infers interpretable and organised hierarchical latent structures for objects within and across time steps; (2) it enables recurrent long-term temporal dependence modelling, which outperforms the one-order Markov descriptions in most of the dynamic probabilistic frameworks; (3) the computational cost scales to the number of positive links only. In addition, we develop a new inference strategy, which first upward- and-backward propagates latent counts and then downward-and-forward samples variables, to enable efficient Gibbs sampling for the Recurrent-DBN. We apply the Recurrent-DBN to dynamic relational data problems. The extensive experiment results on real-world data validate the advantages of the Recurrent-DBN over the state-of-the-art models in interpretable latent structure discovery and improved link prediction performance

Can Intellectual Processes in the Sciences Also Be Simulated? The Anticipation and Visualization of Possible Future States

Socio-cognitive action reproduces and changes both social and cognitive structures. The analytical distinction between these dimensions of structure provides us with richer models of scientific development. In this study, I assume that (i) social structures organize expectations into belief structures that can be attributed to individuals and communities; (ii) expectations are specified in scholarly literature; and (iii) intellectually the sciences (disciplines, specialties) tend to self-organize as systems of rationalized expectations. Whereas social organizations remain localized, academic writings can circulate, and expectations can be stabilized and globalized using symbolically generalized codes of communication. The intellectual restructuring, however, remains latent as a second-order dynamics that can be accessed by participants only reflexively. Yet, the emerging "horizons of meaning" provide feedback to the historically developing organizations by constraining the possible future states as boundary conditions. I propose to model these possible future states using incursive and hyper-incursive equations from the computation of anticipatory systems. Simulations of these equations enable us to visualize the couplings among the historical--i.e., recursive--progression of social structures along trajectories, the evolutionary--i.e., hyper-incursive--development of systems of expectations at the regime level, and the incursive instantiations of expectations in actions, organizations, and texts.Comment: accepted for publication in Scientometrics (June 2015