Typical learning curves for Soft Margin Classifiers (SMCs) learning both
realizable and unrealizable tasks are determined using the tools of Statistical
Mechanics. We derive the analytical behaviour of the learning curves in the
regimes of small and large training sets. The generalization errors present
different decay laws towards the asymptotic values as a function of the
training set size, depending on general geometrical characteristics of the rule
to be learned. Optimal generalization curves are deduced through a fine tuning
of the hyperparameter controlling the trade-off between the error and the
regularization terms in the cost function. Even if the task is realizable, the
optimal performance of the SMC is better than that of a hard margin Support
Vector Machine (SVM) learning the same rule, and is very close to that of the
Bayesian classifier.Comment: 26 pages, 10 figure