1 research outputs found
Proper Learning, Helly Number, and an Optimal SVM Bound
The classical PAC sample complexity bounds are stated for any Empirical Risk
Minimizer (ERM) and contain an extra logarithmic factor
which is known to be necessary for ERM in general. It has been recently shown
by Hanneke (2016) that the optimal sample complexity of PAC learning for any VC
class C is achieved by a particular improper learning algorithm, which outputs
a specific majority-vote of hypotheses in C. This leaves the question of when
this bound can be achieved by proper learning algorithms, which are restricted
to always output a hypothesis from C.
In this paper we aim to characterize the classes for which the optimal sample
complexity can be achieved by a proper learning algorithm. We identify that
these classes can be characterized by the dual Helly number, which is a
combinatorial parameter that arises in discrete geometry and abstract
convexity. In particular, under general conditions on C, we show that the dual
Helly number is bounded if and only if there is a proper learner that obtains
the optimal joint dependence on and .
As further implications of our techniques we resolve a long-standing open
problem posed by Vapnik and Chervonenkis (1974) on the performance of the
Support Vector Machine by proving that the sample complexity of SVM in the
realizable case is ,
where is the dimension. This gives the first optimal PAC bound for
Halfspaces achieved by a proper learning algorithm, and moreover is
computationally efficient