9 research outputs found
A computationally and cognitively plausible model of supervised and unsupervised learning
Author version made available in accordance with the publisher's policy. "The final publication is available at link.springer.com”The issue of chance correction has been discussed for many decades in the context of
statistics, psychology and machine learning, with multiple measures being shown to
have desirable properties, including various definitions of Kappa or Correlation, and
the psychologically validated ΔP measures. In this paper, we discuss the relationships
between these measures, showing that they form part of a single family of measures,
and that using an appropriate measure can positively impact learning
Adabook and Multibook: adaptive boosting with chance correction
There has been considerable interest in boosting and bagging, including the combination of the adaptive
techniques of AdaBoost with the random selection with replacement techniques of Bagging. At the same
time there has been a revisiting of the way we evaluate, with chance-corrected measures like Kappa,
Informedness, Correlation or ROC AUC being advocated. This leads to the question of whether learning
algorithms can do better by optimizing an appropriate chance corrected measure. Indeed, it is possible for a
weak learner to optimize Accuracy to the detriment of the more reaslistic chance-corrected measures, and
when this happens the booster can give up too early. This phenomenon is known to occur with conventional
Accuracy-based AdaBoost, and the MultiBoost algorithm has been developed to overcome such problems
using restart techniques based on bagging. This paper thus complements the theoretical work showing the
necessity of using chance-corrected measures for evaluation, with empirical work showing how use of a
chance-corrected measure can improve boosting. We show that the early surrender problem occurs in
MultiBoost too, in multiclass situations, so that chance-corrected AdaBook and Multibook can beat standard
Multiboost or AdaBoost, and we further identify which chance-corrected measures to use when
The problem with Kappa
It is becoming clear that traditional
evaluation measures used in
Computational Linguistics (including
Error Rates, Accuracy, Recall, Precision
and F-measure) are of limited value for
unbiased evaluation of systems, and are
not meaningful for comparison of
algorithms unless both the dataset and
algorithm parameters are strictly
controlled for skew (Prevalence and
Bias). The use of techniques originally
designed for other purposes, in particular
Receiver Operating Characteristics Area
Under Curve, plus variants of Kappa,
have been proposed to fill the void.
This paper aims to clear up some of the
confusion relating to evaluation, by
demonstrating that the usefulness of each
evaluation method is highly dependent on
the assumptions made about the
distributions of the dataset and the
underlying populations. The behaviour of
a number of evaluation measures is
compared under common assumptions.
Deploying a system in a context which
has the opposite skew from its validation
set can be expected to approximately
negate Fleiss Kappa and halve Cohen
Kappa but leave Powers Kappa
unchanged. For most performance
evaluation purposes, the latter is thus
most appropriate, whilst for comparison
of behaviour, Matthews Correlation is
recommended