Regularization in Machine Learning (ML) is a central technique with great practical significance, whose motivations depend on the learning setting. For example, the popular entropic regularization scheme for Reinforcement Learning (RL) is used to aid in disambiguating optimal policies. On the other hand, Generative Adversarial Networks (GANs) employ regularization for computational purposes, avoiding instability in training. Despite the widespread use and multi-faceted motivation of regularization, extensive evidence has suggested that regularization is a crucial method towards empirical success. Therefore, it is natural that a formal study of regularization would present results that aid in closing the gap between theory and practice. In this thesis, we study a range of different learning problems from the unifying perspective of regularization and uncover various results that contribute to our understanding of machine learning methods. First, we focus on generative modelling and discover a primal-dual relationship between two pioneering methods in the literature of generative modelling, namely Generative Adversarial Networks (GANs) and Autoencoders. The discovery not only explicates a bridge between existing results but proves to be helpful in algorithmic guidance. The study on generative models is then extended to build a boosting-based model that can generate samples compliant with local differential privacy. We then focus on machine learning robustness, where one is interested in understanding the susceptibility of a model in the face of adversarial threats. We show that regularization is intimately connected to distributional robustness, which subsumes existing results and extends them to a great deal of generality, including applications to the unsupervised learning setting. We continue this narrative to the RL setting and similarly expose the robustifying benefits of using regularization, which sheds light on the widely-used entropy-regularized schemes, amongst others. In summary, this thesis's study of regularization contributes substantially within the literature of generative modelling, machine learning robustness, and RL while touching upon additional domains such as privacy and boosting
