Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flows

Achieving a balance between image quality (precision) and diversity (recall) is a significant challenge in the domain of generative models. Current state-of-the-art models primarily rely on optimizing heuristics, such as the Fr\'echet Inception Distance. While recent developments have introduced principled methods for evaluating precision and recall, they have yet to be successfully integrated into the training of generative models. Our main contribution is a novel training method for generative models, such as Generative Adversarial Networks and Normalizing Flows, which explicitly optimizes a user-defined trade-off between precision and recall. More precisely, we show that achieving a specified precision-recall trade-off corresponds to minimizing a unique $f$-divergence from a family we call the \mbox{\em PR-divergences}. Conversely, any $f$-divergence can be written as a linear combination of PR-divergences and corresponds to a weighted precision-recall trade-off. Through comprehensive evaluations, we show that our approach improves the performance of existing state-of-the-art models like BigGAN in terms of either precision or recall when tested on datasets such as ImageNet

An Analytic Approach to the Structure and Composition of General Learning Problems

Gowers presents, in his 2000 essay "The Two Cultures of Mathematics", two kinds of mathematicians he calls the theory-builders and problem-solvers. Of course both kinds of research are important; theory building may directly lead to solutions to problems, and by studying individual problems one uncovers the general structures of problems themselves. However, referencing a remark of Atiyah, Gowers observes that because so much research is produced, the results that can be ``organised coherently and explained economically'' will be the ones that last. Unlike mathematics, the field of machine learning abounds in problem-solvers --- this is wonderful as it leads to a large number of problems being solved --- but it is with regard to the point of Gowers that we are motivated to develop an appropriately general analytic framework to study machine learning problems themselves. To do this we first locate and develop the appropriate analytic objects to study. Chapter 2 recalls some concepts and definitions from the theory of topological vector spaces. In particular, the families of radiant and co-radiant sets and dualities. In Chapter 4 we will need generalisations of a variety of existing results on these families, and these are presented in Chapter 3. Classically a machine learning problem involves four quantities: an outcome space, a family of predictions (or model), a loss function, and a probability distribution. If the loss function is sufficiently general we can combine it with the set of predictions to form a set of real functions, which under very general assumptions, turns out to be closed, convex, and in particular, co-radiant. With the machinery of the previous two chapters in place, in Chapter 4 we lay out the foundations for an analytic theory of the classical machine learning problem, including a general analysis of link functions, by which we may rewrite almost any loss function as a scoring rule; a discussion of scoring rules and their properisation; and using the co-radiant results from Chapter 3 in particular, a theory of prediction aggregation. Chapters 5 and 6 develop results inspired by and related to adversarial learning. Chapter 5 develops a theory of boosted density estimation with strong convergence guarantees, where density updates are computed by training a classifier, and Chapter 6 uses the theory of optimal transport to formulate a robust Bayes minimisation problem, in which we develop a universal theory of regularisation and deliver new strong results for the problem of adversarial learning