17,417 research outputs found
Statistical Mechanics of Soft Margin Classifiers
We study the typical learning properties of the recently introduced Soft
Margin Classifiers (SMCs), learning realizable and unrealizable tasks, with the
tools of Statistical Mechanics. We derive analytically the behaviour of the
learning curves in the regime of very large training sets. We obtain
exponential and power laws for the decay of the generalization error towards
the asymptotic value, depending on the task and on general characteristics of
the distribution of stabilities of the patterns to be learned. The optimal
learning curves of the SMCs, which give the minimal generalization error, are
obtained by tuning the coefficient controlling the trade-off between the error
and the regularization terms in the cost function. If the task is realizable by
the SMC, the optimal performance is better than that of a hard margin Support
Vector Machine and is very close to that of a Bayesian classifier.Comment: 26 pages, 12 figures, submitted to Physical Review
Learning curves for Soft Margin Classifiers
Typical learning curves for Soft Margin Classifiers (SMCs) learning both
realizable and unrealizable tasks are determined using the tools of Statistical
Mechanics. We derive the analytical behaviour of the learning curves in the
regimes of small and large training sets. The generalization errors present
different decay laws towards the asymptotic values as a function of the
training set size, depending on general geometrical characteristics of the rule
to be learned. Optimal generalization curves are deduced through a fine tuning
of the hyperparameter controlling the trade-off between the error and the
regularization terms in the cost function. Even if the task is realizable, the
optimal performance of the SMC is better than that of a hard margin Support
Vector Machine (SVM) learning the same rule, and is very close to that of the
Bayesian classifier.Comment: 26 pages, 10 figure
- …