A Confident Information First Principle for Parametric Reduction and Model Selection of Boltzmann Machines

Typical dimensionality reduction (DR) methods are data-oriented, focusing on directly reducing the number of random variables (or features) while retaining the maximal variations in the high-dimensional data. Targeting unsupervised situations, this paper aims to address the problem from a novel perspective and considers model-oriented dimensionality reduction in parameter spaces of binary multivariate distributions. Specifically, we propose a general parameter reduction criterion, called Confident-Information-First (CIF) principle, to maximally preserve confident parameters and rule out less confident ones. Formally, the confidence of each parameter can be assessed by its contribution to the expected Fisher information distance within a geometric manifold over the neighbourhood of the underlying real distribution. Then we demonstrate two implementations of CIF in different scenarios. First, when there are no observed samples, we revisit the Boltzmann Machines (BM) from a model selection perspective and theoretically show that both the fully visible BM (VBM) and the BM with hidden units can be derived from the general binary multivariate distribution using the CIF principle. This finding would help us uncover and formalize the essential parts of the target density that BM aims to capture and the non-essential parts that BM should discard. Second, when there exist observed samples, we apply CIF to the model selection for BM, which is in turn made adaptive to the observed samples. The sample-specific CIF is a heuristic method to decide the priority order of parameters, which can improve the search efficiency without degrading the quality of model selection results as shown in a series of density estimation experiments

Representation Learning: A Review and New Perspectives

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, auto-encoders, manifold learning, and deep networks. This motivates longer-term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning

Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes

I argue that data becomes temporarily interesting by itself to some self-improving, but computationally limited, subjective observer once he learns to predict or compress the data in a better way, thus making it subjectively simpler and more beautiful. Curiosity is the desire to create or discover more non-random, non-arbitrary, regular data that is novel and surprising not in the traditional sense of Boltzmann and Shannon but in the sense that it allows for compression progress because its regularity was not yet known. This drive maximizes interestingness, the first derivative of subjective beauty or compressibility, that is, the steepness of the learning curve. It motivates exploring infants, pure mathematicians, composers, artists, dancers, comedians, yourself, and (since 1990) artificial systems.Comment: 35 pages, 3 figures, based on KES 2008 keynote and ALT 2007 / DS 2007 joint invited lectur

Creating Persian-like music using computational intelligence

Dastgāh are modal systems in traditional Persian music. Each Dastgāh consists of a group of melodies called Gushé, classified in twelve groups about a century ago (Farhat, 1990). Prior to that time, musical pieces were transferred through oral tradition. The traditional music productions revolve around the existing Dastgāh, and Gushe pieces. In this thesis computational intelligence tools are employed in creating novel Dastgāh-like music.There are three types of creativity: combinational, exploratory, and transformational (Boden, 2000). In exploratory creativity, a conceptual space is navigated for discovering new forms. Sometimes the exploration results in transformational creativity. This is due to meaningful alterations happening on one or more of the governing dimensions of an item. In combinational creativity new links are established between items not previously connected. Boden stated that all these types of creativity can be implemented using artificial intelligence.Various tools, and techniques are employed, in the research reported in this thesis, for generating Dastgāh-like music. Evolutionary algorithms are responsible for navigating the space of sequences of musical motives. Aesthetical critics are employed for constraining the search space in exploratory (and hopefully transformational) type of creativity. Boltzmann machine models are applied for assimilating some of the mechanisms involved in combinational creativity. The creative processes involved are guided by aesthetical critics, some of which are derived from a traditional Persian music database.In this project, Cellular Automata (CA) are the main pattern generators employed to produce raw creative materials. Various methodologies are suggested for extracting features from CA progressions and mapping them to musical space, and input to audio synthesizers. The evaluation of the results of this thesis are assisted by publishing surveys which targeted both public and professional audiences. The generated audio samples are evaluated regarding their Dastgāh-likeness, and the level of creativity of the systems involved