Bayesian Methods in Tensor Analysis

Tensors, also known as multidimensional arrays, are useful data structures in machine learning and statistics. In recent years, Bayesian methods have emerged as a popular direction for analyzing tensor-valued data since they provide a convenient way to introduce sparsity into the model and conduct uncertainty quantification. In this article, we provide an overview of frequentist and Bayesian methods for solving tensor completion and regression problems, with a focus on Bayesian methods. We review common Bayesian tensor approaches including model formulation, prior assignment, posterior computation, and theoretical properties. We also discuss potential future directions in this field.Comment: 32 pages, 8 figures, 2 table

Machine Learning and System Identification for Estimation in Physical Systems

In this thesis, we draw inspiration from both classical system identification and modern machine learning in order to solve estimation problems for real-world, physical systems. The main approach to estimation and learning adopted is optimization based. Concepts such as regularization will be utilized for encoding of prior knowledge and basis-function expansions will be used to add nonlinear modeling power while keeping data requirements practical.The thesis covers a wide range of applications, many inspired by applications within robotics, but also extending outside this already wide field.Usage of the proposed methods and algorithms are in many cases illustrated in the real-world applications that motivated the research.Topics covered include dynamics modeling and estimation, model-based reinforcement learning, spectral estimation, friction modeling and state estimation and calibration in robotic machining.In the work on modeling and identification of dynamics, we develop regularization strategies that allow us to incorporate prior domain knowledge into flexible, overparameterized models. We make use of classical control theory to gain insight into training and regularization while using tools from modern deep learning. A particular focus of the work is to allow use of modern methods in scenarios where gathering data is associated with a high cost.In the robotics-inspired parts of the thesis, we develop methods that are practically motivated and make sure that they are implementable also outside the research setting. We demonstrate this by performing experiments in realistic settings and providing open-source implementations of all proposed methods and algorithms

Accurate and reliable probabilistic modeling with high-dimensional data

Machine learning studies algorithms for learning from data. Probabilistic modeling and reasoning define a principled framework for machine learning, where probability theory is used to represent and manipulate knowledge. In this thesis we focus on two fundamental tasks in probabilistic machine learning: probabilistic prediction and density estimation. We study reliability of probabilistic predictive models, propose flexible models for density estimation, and propose a novel training regime for densities with low-dimensional structure. Neural networks demonstrate state-of-the-art performance in many different prediction tasks. At the same time, modern neural networks trained by maximum likelihood have poorly calibrated predictive uncertainties and suffer from adversarial examples. We hypothesize that careful probabilistic treatment of neural networks would make them better calibrated and more robust. However, Bayesian neural networks have to rely on uninformative priors and crude approximations, which makes it difficult to test this hypothesis. In this thesis we take a step back and study adversarial robustness of a simple, linear model, demonstrating that it no longer suffers from calibration errors on adversarial points when the approximate inference method is accurate and the prior is chosen carefully. Classic density estimation methods do not scale to complex, high-dimensional data like natural images. Normalizing flows model the target density as an invertible transformation of a simple base density, and demonstrate good results in high-dimensional density estimation tasks. State-of-the-art normalizing flow architectures rely on parametrizations of univariate invertible functions. Simple additive/affine parametrizations are often used, stacking many layers to express complex transformations. In this thesis we propose novel parametrizations based on cubic and rational-quadratic splines. The proposed flows demonstrate improved parameter-efficiency and advance state-of-the-art on several density estimation benchmarks. The manifold hypothesis says that the data are likely to lie on a lower-dimensional manifold. This assumption is built into many machine learning models, but using it with density models like normalizing flows is difficult: the standard likelihood-based training objective becomes ill-defined. Injective normalizing flows can be implemented, but their training objective is no longer tractable, requiring approximations or heuristic alternatives. In this thesis we propose a novel training objective that uses nested dropout to align the latent space of a normalizing flow, allowing us to extract a sequence of manifold densities from the trained model. Our experiments demonstrate that the manifolds fit by the method match the data well

Robust Monotonic Optimization Framework for Multicell MISO Systems

The performance of multiuser systems is both difficult to measure fairly and to optimize. Most resource allocation problems are non-convex and NP-hard, even under simplifying assumptions such as perfect channel knowledge, homogeneous channel properties among users, and simple power constraints. We establish a general optimization framework that systematically solves these problems to global optimality. The proposed branch-reduce-and-bound (BRB) algorithm handles general multicell downlink systems with single-antenna users, multiantenna transmitters, arbitrary quadratic power constraints, and robustness to channel uncertainty. A robust fairness-profile optimization (RFO) problem is solved at each iteration, which is a quasi-convex problem and a novel generalization of max-min fairness. The BRB algorithm is computationally costly, but it shows better convergence than the previously proposed outer polyblock approximation algorithm. Our framework is suitable for computing benchmarks in general multicell systems with or without channel uncertainty. We illustrate this by deriving and evaluating a zero-forcing solution to the general problem.Comment: Published in IEEE Transactions on Signal Processing, 16 pages, 9 figures, 2 table