Representation Learning: A Review and New Perspectives

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, auto-encoders, manifold learning, and deep networks. This motivates longer-term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning

Deep Learning of Representations: Looking Forward

Deep learning research aims at discovering learning algorithms that discover multiple levels of distributed representations, with higher levels representing more abstract concepts. Although the study of deep learning has already led to impressive theoretical results, learning algorithms and breakthrough experiments, several challenges lie ahead. This paper proposes to examine some of these challenges, centering on the questions of scaling deep learning algorithms to much larger models and datasets, reducing optimization difficulties due to ill-conditioning or local minima, designing more efficient and powerful inference and sampling procedures, and learning to disentangle the factors of variation underlying the observed data. It also proposes a few forward-looking research directions aimed at overcoming these challenges

Mode-assisted joint training of deep Boltzmann machines

The deep extension of the restricted Boltzmann machine (RBM), known as the deep Boltzmann machine (DBM), is an expressive family of machine learning models which can serve as compact representations of complex probability distributions. However, jointly training DBMs in the unsupervised setting has proven to be a formidable task. A recent technique we have proposed, called mode-assisted training, has shown great success in improving the unsupervised training of RBMs. Here, we show that the performance gains of the mode-assisted training are even more dramatic for DBMs. In fact, DBMs jointly trained with the mode-assisted algorithm can represent the same data set with orders of magnitude lower number of total parameters compared to state-of-the-art training procedures and even with respect to RBMs, provided a fan-in network topology is also introduced. This substantial saving in number of parameters makes this training method very appealing also for hardware implementations

Memcomputing Artificial Intelligence: Improving Learning Algorithms with Algorithms that Learn

Like sentinels guarding a secret treasure, computationally difficult problems define the edge of what can be accomplished across science and industry. The current computing orthodoxy, based on physical realization of the Turing machine via the von Neumann architecture, has begun to stagnate, as seen in the slowing down of Moore’s ‘law’. Hardware has saturated around its architectural (von Neumann) bottleneck, leading to a growing interest in unconventional computing architectures like quantum computing and neuromorphic computing. In this thesis, we examine a new alternative, the brain-inspired computing paradigm, memcomputing, which computes with and in memory. The dynamical systems induced by digital memcomputing machines (DMMs) are applicable to a large family of optimization problems. DMMs use memory, or learn, from past dynamics to explore the phase space of the underlying problem in a drastically different way than typical algorithmic approaches, opening up many potential benefits from optimization to sampling.Some of the most challenging computational problems (and biggest rewards) come from the attempt to instill intelligent behavior in machines, known as the field of artificial intelligence. The first chapter is a warm up with a hardware model of DMMs applied to numerical inversion problems, then we quickly move to the major theme throughout this work, which is the application of DMMs to computational demanding problems in artificial intelligence. The model we focus on is the Boltzmann machine, which is seen as an ancestor to the more popular feedforward neural networks. It no longer makes headlines not because it lacks capability, but rather our training methods lack the ability to train it, leaving much of its capacity unexplored. In this work we apply memcomputing to the training of Boltzmann machines through the development of mode-assisted training. This is a technique that uses memcomputing to sample the mode of the Boltzmann machine, and uses that to stabilize and improve the weight update algorithms. Mode-assisted training improves the performance of DBNs in a downstream supervised task, the unsupervised learning of RBMs as well as the joint training of deep Boltzmann machines, doing as well or better than networks containing two orders of magnitude more parameters