3,730 research outputs found
A unifying view for performance measures in multi-class prediction
In the last few years, many different performance measures have been
introduced to overcome the weakness of the most natural metric, the Accuracy.
Among them, Matthews Correlation Coefficient has recently gained popularity
among researchers not only in machine learning but also in several application
fields such as bioinformatics. Nonetheless, further novel functions are being
proposed in literature. We show that Confusion Entropy, a recently introduced
classifier performance measure for multi-class problems, has a strong
(monotone) relation with the multi-class generalization of a classical metric,
the Matthews Correlation Coefficient. Computational evidence in support of the
claim is provided, together with an outline of the theoretical explanation
Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to the Grading of Astrocytoma Tissues
We use partial class memberships in soft classification to model uncertain
labelling and mixtures of classes. Partial class memberships are not restricted
to predictions, but may also occur in reference labels (ground truth, gold
standard diagnosis) for training and validation data.
Classifier performance is usually expressed as fractions of the confusion
matrix, such as sensitivity, specificity, negative and positive predictive
values. We extend this concept to soft classification and discuss the bias and
variance properties of the extended performance measures. Ambiguity in
reference labels translates to differences between best-case, expected and
worst-case performance. We show a second set of measures comparing expected and
ideal performance which is closely related to regression performance, namely
the root mean squared error RMSE and the mean absolute error MAE.
All calculations apply to classical crisp classification as well as to soft
classification (partial class memberships and/or one-class classifiers). The
proposed performance measures allow to test classifiers with actual borderline
cases. In addition, hardening of e.g. posterior probabilities into class labels
is not necessary, avoiding the corresponding information loss and increase in
variance.
We implement the proposed performance measures in the R package
"softclassval", which is available from CRAN and at
http://softclassval.r-forge.r-project.org.
Our reasoning as well as the importance of partial memberships for
chemometric classification is illustrated by a real-word application:
astrocytoma brain tumor tissue grading (80 patients, 37000 spectra) for finding
surgical excision borders. As borderline cases are the actual target of the
analytical technique, samples which are diagnosed to be borderline cases must
be included in the validation.Comment: The manuscript is accepted for publication in Chemometrics and
Intelligent Laboratory Systems. Supplementary figures and tables are at the
end of the pd
ICLabel: An automated electroencephalographic independent component classifier, dataset, and website
The electroencephalogram (EEG) provides a non-invasive, minimally
restrictive, and relatively low cost measure of mesoscale brain dynamics with
high temporal resolution. Although signals recorded in parallel by multiple,
near-adjacent EEG scalp electrode channels are highly-correlated and combine
signals from many different sources, biological and non-biological, independent
component analysis (ICA) has been shown to isolate the various source generator
processes underlying those recordings. Independent components (IC) found by ICA
decomposition can be manually inspected, selected, and interpreted, but doing
so requires both time and practice as ICs have no particular order or intrinsic
interpretations and therefore require further study of their properties.
Alternatively, sufficiently-accurate automated IC classifiers can be used to
classify ICs into broad source categories, speeding the analysis of EEG studies
with many subjects and enabling the use of ICA decomposition in near-real-time
applications. While many such classifiers have been proposed recently, this
work presents the ICLabel project comprised of (1) an IC dataset containing
spatiotemporal measures for over 200,000 ICs from more than 6,000 EEG
recordings, (2) a website for collecting crowdsourced IC labels and educating
EEG researchers and practitioners about IC interpretation, and (3) the
automated ICLabel classifier. The classifier improves upon existing methods in
two ways: by improving the accuracy of the computed label estimates and by
enhancing its computational efficiency. The ICLabel classifier outperforms or
performs comparably to the previous best publicly available method for all
measured IC categories while computing those labels ten times faster than that
classifier as shown in a rigorous comparison against all other publicly
available EEG IC classifiers.Comment: Intended for NeuroImage. Updated from version one with minor
editorial and figure change
Learning From Noisy Singly-labeled Data
Supervised learning depends on annotated examples, which are taken to be the
\emph{ground truth}. But these labels often come from noisy crowdsourcing
platforms, like Amazon Mechanical Turk. Practitioners typically collect
multiple labels per example and aggregate the results to mitigate noise (the
classic crowdsourcing problem). Given a fixed annotation budget and unlimited
unlabeled data, redundant annotation comes at the expense of fewer labeled
examples. This raises two fundamental questions: (1) How can we best learn from
noisy workers? (2) How should we allocate our labeling budget to maximize the
performance of a classifier? We propose a new algorithm for jointly modeling
labels and worker quality from noisy crowd-sourced data. The alternating
minimization proceeds in rounds, estimating worker quality from disagreement
with the current model and then updating the model by optimizing a loss
function that accounts for the current estimate of worker quality. Unlike
previous approaches, even with only one annotation per example, our algorithm
can estimate worker quality. We establish a generalization error bound for
models learned with our algorithm and establish theoretically that it's better
to label many examples once (vs less multiply) when worker quality is above a
threshold. Experiments conducted on both ImageNet (with simulated noisy
workers) and MS-COCO (using the real crowdsourced labels) confirm our
algorithm's benefits.Comment: 18 pages, 3 figure
Object-Based Image Classification of Summer Crop with Machine Learning Methods
The strategic management of agricultural lands involves crop field monitoring each year. Crop discrimination via remote sensing is a complex task, especially if different crops have a similar spectral response and cropping pattern. In such cases, crop identification could be improved by combining object-based image analysis and advanced machine learning methods. In this investigation, we evaluated the C4.5 decision tree, logistic regression (LR), support vector machine (SVM) and multilayer perceptron (MLP) neural network methods, both as single classifiers and combined in a hierarchical classification, for the mapping of nine major summer crops (both woody and herbaceous) from ASTER satellite images captured in two different dates. Each method was built with different combinations of spectral and textural features obtained after the segmentation of the remote images in an object-based framework. As single classifiers, MLP and SVM obtained maximum overall accuracy of 88%, slightly higher than LR (86%) and notably higher than C4.5 (79%). The SVM+SVM classifier (best method) improved these results to 89%. In most cases, the hierarchical classifiers considerably increased the accuracy of the most poorly classified class (minimum sensitivity). The SVM+SVM method offered a significant improvement in classification accuracy for all of the studied crops compared to the conventional decision tree classifier, ranging between 4% for safflower and 29% for corn, which suggests the application of object-based image analysis and advanced machine learning methods in complex crop classification tasks.This research was partly financed by the TIN2011-22794 project of the Spanish Ministerial
Commission of Science and Technology (MICYT), FEDER funds, the P2011-TIC-7508 project of the
âJunta de AndalucĂaâ (Spain) and the Kearney Foundation of Soil Science (USA). The research of
Peña was co-financed by the Fulbright-MEC postdoctoral program, financed by the Spanish Ministry
for Science and Innovation, and by the JAEDoc Program, supported by CSIC and FEDER funds.
ASTER data were available to us through a NASA EOS scientific investigator affiliation.We acknowledge support by the CSIC Open Access Publication Initiative through its Unit of Information Resources for Research (URICI).Peer Reviewe
Object-Based Image Classification of Summer Crops with Machine Learning Methods
The strategic management of agricultural lands involves crop field monitoring each year. Crop discrimination via remote sensing is a complex task, especially if different crops have a similar spectral response and cropping pattern. In such cases, crop identification could be improved by combining object-based image analysis and advanced machine learning methods. In this investigation, we evaluated the C4.5 decision tree, logistic regression (LR), support vector machine (SVM) and multilayer perceptron (MLP) neural network methods, both as single classifiers and combined in a hierarchical classification, for the mapping of nine major summer crops (both woody and herbaceous) from ASTER satellite images captured in two different dates. Each method was built with different combinations of spectral and textural features obtained after the segmentation of the remote images in an object-based framework. As single classifiers, MLP and SVM obtained maximum overall accuracy of 88%, slightly higher than LR (86%) and notably higher than C4.5 (79%). The SVM+SVM classifier (best method) improved these results to 89%. In most cases, the hierarchical classifiers considerably increased the accuracy of the most poorly classified class (minimum sensitivity). The SVM+SVM method offered a
significant improvement in classification accuracy for all of the studied crops compared to
the conventional decision tree classifier, ranging between 4% for safflower and 29% for
corn, which suggests the application of object-based image analysis and advanced machine
learning methods in complex crop classification task
- âŠ