k-fold cross-validation can significantly over-estimate true
classification accuracy in common EEG-based passive BCI
experimental designs: an empirical investigation
In passive brain computer interface (BCI) studies, a common approach is to collect
data from mental states of interest during relatively long trials and divide these trials
into shorter “epochs” to serve as individual samples in classification. While it is
known that using k-fold cross-validation (CV) in this scenario can result in unreliable
estimates of mental state separability (due to autocorrelation in the samples derived
from the same trial), k-fold CV is still commonly used and reported in passive BCI
studies. What is not known is the extent to which k-fold CV misrepresents true mental
state separability. This makes it difficult to interpret the results of studies that use
it. Furthermore, if the seriousness of the problem were clearly known, perhaps more
researchers would be aware that they should avoid it. In this work, a novel experiment
explored how the degree of correlation among samples within a class affects EEG-based
mental state classification accuracy estimated by k-fold CV. Results were compared
to a ground-truth (GT) accuracy and to “block-wise” CV, an alternative to k-fold
which is purported to alleviate the autocorrelation issues. Factors such as the degree
of true class separability and the feature set and classifier used were also explored.
The results show that, under some conditions, k-fold CV inflated the GT classification
accuracy by up to 25%. It is our recommendation that the number of samples derived
from the same trial should be reduced whenever possible in single-subject analysis,
and that both the k-fold and block-wise CV results are reported
Is data on this page outdated, violates copyrights or anything else? Report the problem now and we will take corresponding actions after reviewing your request.