4,728 research outputs found
Quantitative Analysis of Saliency Models
Previous saliency detection research required the reader to evaluate
performance qualitatively, based on renderings of saliency maps on a few
shapes. This qualitative approach meant it was unclear which saliency models
were better, or how well they compared to human perception. This paper provides
a quantitative evaluation framework that addresses this issue. In the first
quantitative analysis of 3D computational saliency models, we evaluate four
computational saliency models and two baseline models against ground-truth
saliency collected in previous work.Comment: 10 page
An empirical evaluation of imbalanced data strategies from a practitioner's point of view
This research tested the following well known strategies to deal with binary
imbalanced data on 82 different real life data sets (sampled to imbalance rates
of 5%, 3%, 1%, and 0.1%): class weight, SMOTE, Underbagging, and a baseline
(just the base classifier). As base classifiers we used SVM with RBF kernel,
random forests, and gradient boosting machines and we measured the quality of
the resulting classifier using 6 different metrics (Area under the curve,
Accuracy, F-measure, G-mean, Matthew's correlation coefficient and Balanced
accuracy). The best strategy strongly depends on the metric used to measure the
quality of the classifier. For AUC and accuracy class weight and the baseline
perform better; for F-measure and MCC, SMOTE performs better; and for G-mean
and balanced accuracy, underbagging
Towards a Reliable Comparison and Evaluation of Network Intrusion Detection Systems Based on Machine Learning Approaches
Presently, we are living in a hyper-connected world where millions of heterogeneous devices are continuously sharing information in different application contexts for wellness, improving communications, digital businesses, etc. However, the bigger the number of devices and connections are, the higher the risk of security threats in this scenario. To counteract against malicious behaviours and preserve essential security services, Network Intrusion Detection Systems (NIDSs) are the most widely used defence line in communications networks. Nevertheless, there is no standard methodology to evaluate and fairly compare NIDSs. Most of the proposals elude mentioning crucial steps regarding NIDSs validation that make their comparison hard or even impossible. This work firstly includes a comprehensive study of recent NIDSs based on machine learning approaches, concluding that almost all of them do not accomplish with what authors of this paper consider mandatory steps for a reliable comparison and evaluation of NIDSs. Secondly, a structured methodology is proposed and assessed on the UGR'16 dataset to test its suitability for addressing network attack detection problems. The guideline and steps recommended will definitively help the research community to fairly assess NIDSs, although the definitive framework is not a trivial task and, therefore, some extra effort should still be made to improve its understandability and usability further
A unifying view for performance measures in multi-class prediction
In the last few years, many different performance measures have been
introduced to overcome the weakness of the most natural metric, the Accuracy.
Among them, Matthews Correlation Coefficient has recently gained popularity
among researchers not only in machine learning but also in several application
fields such as bioinformatics. Nonetheless, further novel functions are being
proposed in literature. We show that Confusion Entropy, a recently introduced
classifier performance measure for multi-class problems, has a strong
(monotone) relation with the multi-class generalization of a classical metric,
the Matthews Correlation Coefficient. Computational evidence in support of the
claim is provided, together with an outline of the theoretical explanation
Training recurrent neural networks robust to incomplete data: application to Alzheimer's disease progression modeling
Disease progression modeling (DPM) using longitudinal data is a challenging
machine learning task. Existing DPM algorithms neglect temporal dependencies
among measurements, make parametric assumptions about biomarker trajectories,
do not model multiple biomarkers jointly, and need an alignment of subjects'
trajectories. In this paper, recurrent neural networks (RNNs) are utilized to
address these issues. However, in many cases, longitudinal cohorts contain
incomplete data, which hinders the application of standard RNNs and requires a
pre-processing step such as imputation of the missing values. Instead, we
propose a generalized training rule for the most widely used RNN architecture,
long short-term memory (LSTM) networks, that can handle both missing predictor
and target values. The proposed LSTM algorithm is applied to model the
progression of Alzheimer's disease (AD) using six volumetric magnetic resonance
imaging (MRI) biomarkers, i.e., volumes of ventricles, hippocampus, whole
brain, fusiform, middle temporal gyrus, and entorhinal cortex, and it is
compared to standard LSTM networks with data imputation and a parametric,
regression-based DPM method. The results show that the proposed algorithm
achieves a significantly lower mean absolute error (MAE) than the alternatives
with p < 0.05 using Wilcoxon signed rank test in predicting values of almost
all of the MRI biomarkers. Moreover, a linear discriminant analysis (LDA)
classifier applied to the predicted biomarker values produces a significantly
larger AUC of 0.90 vs. at most 0.84 with p < 0.001 using McNemar's test for
clinical diagnosis of AD. Inspection of MAE curves as a function of the amount
of missing data reveals that the proposed LSTM algorithm achieves the best
performance up until more than 74% missing values. Finally, it is illustrated
how the method can successfully be applied to data with varying time intervals.Comment: arXiv admin note: substantial text overlap with arXiv:1808.0550
Robust training of recurrent neural networks to handle missing data for disease progression modeling
Disease progression modeling (DPM) using longitudinal data is a challenging
task in machine learning for healthcare that can provide clinicians with better
tools for diagnosis and monitoring of disease. Existing DPM algorithms neglect
temporal dependencies among measurements and make parametric assumptions about
biomarker trajectories. In addition, they do not model multiple biomarkers
jointly and need to align subjects' trajectories. In this paper, recurrent
neural networks (RNNs) are utilized to address these issues. However, in many
cases, longitudinal cohorts contain incomplete data, which hinders the
application of standard RNNs and requires a pre-processing step such as
imputation of the missing values. We, therefore, propose a generalized training
rule for the most widely used RNN architecture, long short-term memory (LSTM)
networks, that can handle missing values in both target and predictor
variables. This algorithm is applied for modeling the progression of
Alzheimer's disease (AD) using magnetic resonance imaging (MRI) biomarkers. The
results show that the proposed LSTM algorithm achieves a lower mean absolute
error for prediction of measurements across all considered MRI biomarkers
compared to using standard LSTM networks with data imputation or using a
regression-based DPM method. Moreover, applying linear discriminant analysis to
the biomarkers' values predicted by the proposed algorithm results in a larger
area under the receiver operating characteristic curve (AUC) for clinical
diagnosis of AD compared to the same alternatives, and the AUC is comparable to
state-of-the-art AUCs from a recent cross-sectional medical image
classification challenge. This paper shows that built-in handling of missing
values in LSTM network training paves the way for application of RNNs in
disease progression modeling.Comment: 9 pages, 1 figure, MIDL conferenc
- …