Let p be an unknown and arbitrary probability distribution over [0,1). We
consider the problem of {\em density estimation}, in which a learning algorithm
is given i.i.d. draws from p and must (with high probability) output a
hypothesis distribution that is close to p. The main contribution of this
paper is a highly efficient density estimation algorithm for learning using a
variable-width histogram, i.e., a hypothesis distribution with a piecewise
constant probability density function.
In more detail, for any k and ϵ, we give an algorithm that makes
O~(k/ϵ2) draws from p, runs in O~(k/ϵ2)
time, and outputs a hypothesis distribution h that is piecewise constant with
O(klog2(1/ϵ)) pieces. With high probability the hypothesis h
satisfies dTV(p,h)≤C⋅optk(p)+ϵ,
where dTV denotes the total variation distance (statistical
distance), C is a universal constant, and optk(p) is the smallest
total variation distance between p and any k-piecewise constant
distribution. The sample size and running time of our algorithm are optimal up
to logarithmic factors. The "approximation factor" C in our result is
inherent in the problem, as we prove that no algorithm with sample size bounded
in terms of k and ϵ can achieve C<2 regardless of what kind of
hypothesis distribution it uses.Comment: conference version appears in NIPS 201