Learning Exponential Families in High-Dimensions: Strong Convexity and Sparsity

The versatility of exponential families, along with their attendant convexity properties, make them a popular and effective statistical model. A central issue is learning these models in high-dimensions when the optimal parameter vector is sparse. This work characterizes a certain strong convexity property of general exponential families, which allows their generalization ability to be quantified. In particular, we show how this property can be used to analyze generic exponential families under L1 regularization

Abstract

Strong restricted-orientation convexity is a generalization of standard convexity. We explore the properties of strongly convex sets in multidimensional Euclidean space and identify major properties of standard convex sets that also hold for strong convexity. We characterize strongly convex flats and halfspaces, and establish the strong convexity of the affine hull of a strongly convex set. We then show that, for every point in the boundary of a strongly convex set, there is a supporting strongly convex hyperplane through it. Finally, we show that a closed set with nonempty interior is strongly convex if and only if it is the intersection of strongly convex halfspaces; we state a condition under which this result extends to sets with empty interior