We consider the problem of cardinality penalized optimization of a convex function
over the probability simplex with additional convex constraints. The classical
ℓ_1 regularizer fails to promote sparsity on the probability simplex since ℓ_1 norm
on the probability simplex is trivially constant. We propose a direct relaxation of
the minimum cardinality problem and show that it can be efficiently solved using
convex programming. As a first application we consider recovering a sparse probability
measure given moment constraints, in which our formulation becomes linear
programming, hence can be solved very efficiently. A sufficient condition for
exact recovery of the minimum cardinality solution is derived for arbitrary affine
constraints. We then develop a penalized version for the noisy setting which can
be solved using second order cone programs. The proposed method outperforms
known rescaling heuristics based on ℓ_1 norm. As a second application we consider
convex clustering using a sparse Gaussian mixture and compare our results with
the well known soft k-means algorithm
