Over-optimism in bioinformatics: an illustration

In statistical bioinformatics research, different optimization mechanisms potentially lead to "over-optimism" in published papers. The present empirical study illustrates these mechanisms through a concrete example from an active research field. The investigated sources of over-optimism include the optimization of the data sets, of the settings, of the competing methods and, most importantly, of the method’s characteristics. We consider a "promising" new classification algorithm that turns out to yield disappointing results in terms of error rate, namely linear discriminant analysis incorporating prior knowledge on gene functional groups through an appropriate shrinkage of the within-group covariance matrix. We quantitatively demonstrate that this disappointing method can artificially seem superior to existing approaches if we "fish for significance”. We conclude that, if the improvement of a quantitative criterion such as the error rate is the main contribution of a paper, the superiority of new algorithms should be validated using "fresh" validation data sets

fMRI activation detection with EEG priors

The purpose of brain mapping techniques is to advance the understanding of the relationship between structure and function in the human brain in so-called activation studies. In this work, an advanced statistical model for combining functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) recordings is developed to fuse complementary information about the location of neuronal activity. More precisely, a new Bayesian method is proposed for enhancing fMRI activation detection by the use of EEG-based spatial prior information in stimulus based experimental paradigms. I.e., we model and analyse stimulus influence by a spatial Bayesian variable selection scheme, and extend existing high-dimensional regression methods by incorporating prior information on binary selection indicators via a latent probit regression with either a spatially-varying or constant EEG effect. Spatially-varying effects are regularized by intrinsic Markov random field priors. Inference is based on a full Bayesian Markov Chain Monte Carlo (MCMC) approach. Whether the proposed algorithm is able to increase the sensitivity of mere fMRI models is examined in both a real-world application and a simulation study. We observed, that carefully selected EEG--prior information additionally increases sensitivity in activation regions that have been distorted by a low signal-to-noise ratio

A Bayesian Variable Selection Approach Yields Improved Detection of Brain Activation From Complex-Valued fMRI

Voxel functional magnetic resonance imaging (fMRI) time courses are complex-valued signals giving rise to magnitude and phase data. Nevertheless, most studies use only the magnitude signals and thus discard half of the data that could potentially contain important information. Methods that make use of complex-valued fMRI (CV-fMRI) data have been shown to lead to superior power in detecting active voxels when compared to magnitude-only methods, particularly for small signal-to-noise ratios (SNRs). We present a new Bayesian variable selection approach for detecting brain activation at the voxel level from CV-fMRI data. We develop models with complex-valued spike-and-slab priors on the activation parameters that are able to combine the magnitude and phase information. We present a complex-valued EM variable selection algorithm that leads to fast detection at the voxel level in CV-fMRI slices and also consider full posterior inference via Markov chain Monte Carlo (MCMC). Model performance is illustrated through extensive simulation studies, including the analysis of physically based simulated CV-fMRI slices. Finally, we use the complex-valued Bayesian approach to detect active voxels in human CV-fMRI from a healthy individual who performed unilateral finger tapping in a designed experiment. The proposed approach leads to improved detection of activation in the expected motor-related brain regions and produces fewer false positive results than other methods for CV-fMRI. Supplementary materials for this article are available online