Learning Group Structure and Disentangled Representations of Dynamical Environments

Discovering the underlying structure of a dynamical environment involves learning representations that are interpretable and disentangled, which is a challenging task. In physics, interpretable representations of our universe and its underlying dynamics are formulated in terms of representations of groups of symmetry transformations. We propose a physics-inspired method, built upon the theory of group representation, that learns a representation of an environment structured around the transformations that generate its evolution. Experimentally, we learn the structure of explicitly symmetric environments without supervision while ensuring the interpretability of the representations. We show that the learned representations allow for accurate long-horizon predictions and further demonstrate a correlation between the quality of predictions and disentanglement in the latent space.Comment: 8 pages + 2 pages of reference

Multirate digital filters, filter banks, polyphase networks, and applications: a tutorial

Multirate digital filters and filter banks find application in communications, speech processing, image compression, antenna systems, analog voice privacy systems, and in the digital audio industry. During the last several years there has been substantial progress in multirate system research. This includes design of decimation and interpolation filters, analysis/synthesis filter banks (also called quadrature mirror filters, or QMFJ, and the development of new sampling theorems. First, the basic concepts and building blocks in multirate digital signal processing (DSPJ, including the digital polyphase representation, are reviewed. Next, recent progress as reported by several authors in this area is discussed. Several applications are described, including the following: subband coding of waveforms, voice privacy systems, integral and fractional sampling rate conversion (such as in digital audio), digital crossover networks, and multirate coding of narrow-band filter coefficients. The M-band QMF bank is discussed in considerable detail, including an analysis of various errors and imperfections. Recent techniques for perfect signal reconstruction in such systems are reviewed. The connection between QMF banks and other related topics, such as block digital filtering and periodically time-varying systems, based on a pseudo-circulant matrix framework, is covered. Unconventional applications of the polyphase concept are discussed

Constraint-Based Regularization of Neural Networks

We propose a method for efficiently incorporating constraints into a stochastic gradient Langevin framework for the training of deep neural networks. Constraints allow direct control of the parameter space of the model. Appropriately designed, they reduce the vanishing/exploding gradient problem, control weight magnitudes and stabilize deep neural networks and thus improve the robustness of training algorithms and the generalization capabilities of the trained neural network. We present examples of constrained training methods motivated by orthogonality preservation for weight matrices and explicit weight normalizations. We describe the methods in the overdamped formulation of Langevin dynamics and the underdamped form, in which momenta help to improve sampling efficiency. The methods are explored in test examples in image classification and natural language processing.Comment: T. Vlaar won best student paper award at OPT202