Towards efficient and robust reinforcement learning via synthetic environments and offline data

Over the past decade, Deep Reinforcement Learning (RL) has driven many advances in sequential decision-making, including remarkable applications in superhuman Go-playing, robotic control, and automated algorithm discovery. However, despite these successes, deep RL is also notoriously sample-inefficient, usually generalizes poorly to settings beyond the original environment, and can be unstable during training. Moreover, the conventional RL setting still relies on exploring and learning tabula-rasa in new environments and does not make use of pre-existing data. This thesis investigates two promising directions to address these challenges. First, we explore the use of synthetic data and environments in order to broaden an agent's experience. Second, we propose principled techniques to leverage pre-existing datasets, thereby reducing or replacing the need for costly online data collection. The first part of the thesis focuses on the generation of synthetic data and environments to train RL agents. While there is a rich history in model-based RL of leveraging a learned dynamics model to improve sample efficiency, these methods are usually restricted to single-task settings. To overcome this limitation, we propose Augmented World Models, a novel approach designed for offline-to-online transfer where the test dynamics may differ from the training data. Our method augments a learned dynamics model with simple transformations that seek to capture potential changes in the physical properties of a robot, leading to more robust policies. Additionally, we train the agent with the sampled augmentation as context for test-time inference, significantly improving zero-shot generalization to novel dynamics. Going beyond commonly used forward dynamics models, we propose an alternative paradigm, Synthetic Experience Replay, which uses generative modeling to directly model and upsample the agent's training data distribution. Leveraging recent advances in diffusion generative models, our approach outperforms and is composable with standard data augmentation, and is particularly effective in low-data regimes. Furthermore, our method opens the door for certain RL agents to train stably with much larger networks than before. In the second part of the thesis, we explore a complementary direction to data efficiency where we can leverage pre-existing data. While adjacent fields of machine learning, such as computer vision and natural language processing, have made significant progress in scaling data and model size, traditional RL algorithms can find it difficult to incorporate additional data due to the need for on-policy data. We begin by investigating a principled method for incorporating expert demonstrations to accelerate online RL, KL-regularization to a behavioral prior, and identify a pathology stemming from the behavioral prior having uncalibrated uncertainties. We show that standard parameterizations of the behavioral reference policy can lead to unstable training dynamics, and propose a solution, Non-Parametric Prior Actor–Critic, that represents the new state-of-the-art in locomotion and dexterous manipulation tasks. Furthermore, we make advances in offline reinforcement learning, with which agents can be trained without any online data collection at all. In this domain, we elucidate the design space of offline model-based RL algorithms and highlight where prior methods use suboptimal heuristics and choices for hyperparameters. By rigorously searching through this space, we show that we can vastly improve standard algorithms and provide insights into which design choices are most important. Finally, we make progress towards extending offline RL to pixel-based environments by presenting Vision Datasets for Deep Data-Driven RL, the first comprehensive and publicly available evaluation suite for this field, alongside simple model-based and model-free baselines for assessing future progress in this domain. In conclusion, this thesis represents explorations toward making RL algorithms more efficient and readily deployable in the real world. Further progress along these directions may bring us closer to the ultimate goal of more generally capable agents, that are able to both generate appropriate learning environments for themselves and bootstrap learning from vast quantities of pre-existing data

Determination of Cap Model Parameters using Numerical Optimization Method for Powder Compaction

Many advantages are inherent to the successful powder metallurgy (P/M) process especially in high volume manufacturing. The strength/density distribution of the compacted product is crucial to overall success. The finite element analysis (FEA) method has become an effective way to numerically simulate strength/density distribution in a P/M compact. The modified Drucker-Prager cap (DPC) model has been shown to be a suitable constitutive relationship for metal powder compaction simulation. The calibration of the modified DPC model involves a procedure known as a triaxial compression test. Equipment for completing a triaxial compression test on metal powders is neither readily available nor standardized in the P/M industry. A robust calibration procedure that requires only simple experimental tests would increase the usability of the simulation procedure. This research created a universal cost/time-effective calibration method to accurately determine all parameters of a modified DPC model by using a combination of numerical simulation methods, numerical optimization methods and common material testing techniques. The use of the triaxial compression test is eliminated and the new method relies only upon conventional compaction equipment, standard geometry and readily available metallographic techniques. The DPC parameters were determined by applying the proposed method on ferrous powders. The predicted DPC parameters were verified on a compressed product with complex geometry

Modulated Unit-Norm Tight Frames for Compressed Sensing

In this paper, we propose a compressed sensing (CS) framework that consists of three parts: a unit-norm tight frame (UTF), a random diagonal matrix and a column-wise orthonormal matrix. We prove that this structure satisfies the restricted isometry property (RIP) with high probability if the number of measurements $m = O(s \log^2s \log^2n)$ for $s$-sparse signals of length $n$ and if the column-wise orthonormal matrix is bounded. Some existing structured sensing models can be studied under this framework, which then gives tighter bounds on the required number of measurements to satisfy the RIP. More importantly, we propose several structured sensing models by appealing to this unified framework, such as a general sensing model with arbitrary/determinisic subsamplers, a fast and efficient block compressed sensing scheme, and structured sensing matrices with deterministic phase modulations, all of which can lead to improvements on practical applications. In particular, one of the constructions is applied to simplify the transceiver design of CS-based channel estimation for orthogonal frequency division multiplexing (OFDM) systems.Comment: submitted to IEEE Transactions on Signal Processin

Epitaxial Growth and Intrinsic Magnetic Properties of Magnetic Thin Films on Semiconductor Materials

Spin electronics, or spintronics, is an emergent interdisciplinary area whereby the spin degree of freedom in electronic devices is employed. One of the most important topics of spintronics is to develop magnetic/semiconductor hybrid materials for the next generation spin devices such as Spin-FET. This dissertation investigates the magnetic properties of magnetic thin films deposited on semiconductor substrate. Fe and Ni films have been deposited onto GaAs substrate with different substrate treatment. Both single and poly crystalline samples were developed during the process, and the magnetic properties were compared to explore the differences. The origin of a uniaxial magnetic anisotropy (UMA) from the crystal symmetry of the bcc Fe, observed in Fe/GaAs (100) has been investigated. Another major work for this thesis is the Artemis TR-ARPES (time resolved – angular resolved photoemission spectroscopy) project in Rutherford Appleton Laboratory. Successful MBE growth was achieved for the first time during the Artemis TR-ARPES project, followed by static and time-resolved MOKE, and time-resolved photoemission measurements. Preliminary time-resolved photoemission spectroscopy measurements has been performed on the single crystal Fe samples, which is the world first result in term of the study of the time-dependent valence band structures using direct photoemission technique from a metallic material. The research finally extended to the CoFeB film on GaAs substrate. The discovery of uniaxial magnetic anisotropy of this amorphous film make it one of the hottest topic in research. XMCD measurements were conducted and the results went through a series of calculations gives an indication of the relation between magnetic moment to the anisotropy. TEM inspections offers a clear look at the interface of the samples

An approach to collaborative assembly design modification and assembly planning

Ph.DDOCTOR OF PHILOSOPH