Unsupervised Representation Learning with Correlations

Unsupervised representation learning algorithms have been playing important roles in machine learning and related fields. However, due to optimization intractability or lack of consideration in given data correlation structures, some unsupervised representation learning algorithms still cannot well discover the inherent features from the data, under certain circumstances. This thesis extends these algorithms, and improves over the above issues by taking data correlations into consideration. We study three different aspects of improvements on unsupervised representation learning algorithms by utilizing correlation information, via the following three tasks respectively: 1. Using estimated correlations between data points to provide smart optimization initializations, for multi-way matching (Chapter 2). In this work, we define a correlation score between pairs of data points as metrics for correlations, and initialize all the permutation matrices along a maximum spanning tree of the undirected graph with these metrics as the weights. 2. Faster optimization by utilizing the correlations in the observations, for variational inference (Chapter 3). We construct a positive definite matrix from the negative Hessian of the log-likelihood part of the objective that can capture the influence of the observation correlations on the parameter vector. We then use the inverse of this matrix to rescale the gradient. 3. Utilizing additional side-information on data correlation structures to explicitly learn correlations between data points, for extensions of Variational Auto-Encoders (VAEs) (Chapters 4 and 5). Consider the case where we know a correlation graph G of the data points. Instead of placing an i.i.d. prior as in the most common setting, we adopt correlated priors and/or correlated variational distributions on the latent variables through utilizing the graph G. Empirical results on these tasks show the success of the proposed methods in improving the performances of unsupervised representation learning algorithms. We compare our methods with multiple recent advanced algorithms on various tasks, on both synthetic and real datasets. We also provide theoretical analysis for some of the proposed methods, showing their advantages under certain situations. The proposed methods have wide ranges of applications. For examples, image compression (via smart initializations for multi-way matching), link prediction (by VAEs with correlations), etc

Contract-Based Design: Theories and Applications

Most things we know only exist in relation to one another. Their states are strongly coupled due to dependencies that arise from such relations. For a system designer, acknowledging the presence of these dependencies is as crucial to guaranteeing performance as studying them. As the roles played by technology in fields such as transportation, healthcare, and finance continue to be more profound and diverse, modern engineering systems have grown to be more reliant on the integration of technologies across multiple disciplines and their requirements. The need to ensure proper division of labor, integration of system modules, and attribution of legal responsibility calls for a more methodological look into co-design considerations. Originally conceived in computer programming, contract-based reasoning is a design approach whose promise of a formal compositional paradigm is receiving attention from a broader engineering community. Our work is dedicated to narrowing the gap between the theory and application of this yet nascent framework. In the first half of this dissertation, we introduce a model interface contract theory for input/output automata with guards and a formalization of the directive-response architecture using assume-guarantee contracts and show how these may be used to guide the formal design of a traffic intersection and an automated valet parking system respectively. Next, we address a major drawback of assume-guarantee contracts, i.e., the problem of a void contract due to antecedent failure. Our proposed solution is a reactive version of assume-guarantee contracts that enables direct specification at the assumption and guarantee level along with a novel synthesis algorithm that exposes the effects of failures on the contract structure. This is then used to help optimize, adapt, and robustify our design against an uncertain environment. In light of ongoing development of autonomous driving technologies and its potential impact on the safety of future transportation, the second half of this work is dedicated to the application of the design-by-contract framework to the distributed control of autonomous vehicles. We start by defining and proving properties of "assume-guarantee profiles," our proposed approach to transparent distributed multi-agent decision making and behavior prediction. Next, we provide a local conflict resolution algorithm in the context of a quasi-simultaneous game which guarantees safety and liveness to the composition of autonomous vehicle systems in this game. Finally, to facilitate the extension of these frameworks to real-life urban driving settings, we also supply an effective method to predict agent behavior that utilizes recent advances in machine learning research.</p