68 research outputs found
ROC and AUC with a Binary Predictor: a Potentially Misleading Metric
In analysis of binary outcomes, the receiver operator characteristic (ROC)
curve is heavily used to show the performance of a model or algorithm. The ROC
curve is informative about the performance over a series of thresholds and can
be summarized by the area under the curve (AUC), a single number. When a
predictor is categorical, the ROC curve has one less than number of categories
as potential thresholds; when the predictor is binary there is only one
threshold. As the AUC may be used in decision-making processes on determining
the best model, it important to discuss how it agrees with the intuition from
the ROC curve. We discuss how the interpolation of the curve between thresholds
with binary predictors can largely change the AUC. Overall, we show using a
linear interpolation from the ROC curve with binary predictors corresponds to
the estimated AUC, which is most commonly done in software, which we believe
can lead to misleading results. We compare R, Python, Stata, and SAS software
implementations. We recommend using reporting the interpolation used and
discuss the merit of using the step function interpolator, also referred to as
the "pessimistic" approach by Fawcett (2006).Comment: 16 pages, 3 figures, cod
ciftiTools: A package for reading, writing, visualizing and manipulating CIFTI files in R
Surface- and grayordinate-based analysis of MR data has well-recognized
advantages, including improved whole-cortex visualization, the ability to
perform surface smoothing to avoid issues associated with volumetric smoothing,
improved inter-subject alignment, and reduced dimensionality. The CIFTI
grayordinate file format introduced by the Human Connectome Project further
advances grayordinate-based analysis by combining gray matter data from the
left and right cortical hemispheres with gray matter data from the subcortex
and cerebellum into a single file. Analyses performed in grayordinate space are
well-suited to leverage information shared across the brain and across subjects
through both traditional analysis techniques and more advanced statistical
methods, including Bayesian methods. The R statistical environment facilitates
use of advanced statistical techniques, yet little support for grayordinates
analysis has been previously available in R. Indeed, few comprehensive
programmatic tools for working with CIFTI files have been available in any
language. Here, we present the ciftiTools R package, which provides a unified
environment for reading, writing, visualizing, and manipulating CIFTI files and
related data formats. We illustrate ciftiTools' convenient and user-friendly
suite of tools for working with grayordinates and surface geometry data in R,
and we describe how ciftiTools is being utilized to advance the statistical
analysis of grayordinate-based functional MRI data.Comment: 41 pages, 6 figure
Relating multi-sequence longitudinal intensity profiles and clinical covariates in new multiple sclerosis lesions
Structural magnetic resonance imaging (MRI) can be used to detect lesions in
the brains of multiple sclerosis (MS) patients. The formation of these lesions
is a complex process involving inflammation, tissue damage, and tissue repair,
all of which are visible on MRI. Here we characterize the lesion formation
process on longitudinal, multi-sequence structural MRI from 34 MS patients and
relate the longitudinal changes we observe within lesions to therapeutic
interventions. In this article, we first outline a pipeline to extract voxel
level, multi-sequence longitudinal profiles from four MRI sequences within
lesion tissue. We then propose two models to relate clinical covariates to the
longitudinal profiles. The first model is a principal component analysis (PCA)
regression model, which collapses the information from all four profiles into a
scalar value. We find that the score on the first PC identifies areas of slow,
long-term intensity changes within the lesion at a voxel level, as validated by
two experienced clinicians, a neuroradiologist and a neurologist. On a quality
scale of 1 to 4 (4 being the highest) the neuroradiologist gave the score on
the first PC a median rating of 4 (95% CI: [4,4]), and the neurologist gave it
a median rating of 3 (95% CI: [3,3]). In the PCA regression model, we find that
treatment with disease modifying therapies (p-value < 0.01), steroids (p-value
< 0.01), and being closer to the boundary of abnormal signal intensity (p-value
< 0.01) are associated with a return of a voxel to intensity values closer to
that of normal-appearing tissue. The second model is a function-on-scalar
regression, which allows for assessment of the individual time points at which
the covariates are associated with the profiles. In the function-on-scalar
regression both age and distance to the boundary were found to have a
statistically significant association with the profiles
ANALYTIC PROGRAMMING WITH fMRI DATA: A QUICK-START GUIDE FOR STATISTICIANS USING R
Functional magnetic resonance imaging (fMRI) is a thriving field that plays an important role in medical imaging analysis, biological and neuroscience research and practice. This manuscript gives a didactic introduction to the statistical analysis of fMRI data using the R project along with the relevant R code. The goal is to give tatisticians who would like to pursue research in this area a quick start for programming with fMRI data along with the available data visualization tools
Removing inter-subject technical variability in magnetic resonance imaging studies
Magnetic resonance imaging (MRI) intensities are acquired in arbitrary units, making scans non-comparable across sites and between subjects. Intensity normalization is a first step for the improvement of comparability of the images across subjects. However, we show that unwanted inter-scan variability associated with imaging site, scanner effect and other technical artifacts is still present after standard intensity normalization in large multi-site neuroimaging studies. We propose RAVEL (Removal of Artificial Voxel Effect by Linear regression), a tool to remove residual technical variability after intensity normalization. As proposed by SVA and RUV [Leek and Storey, 2007, 2008, Gagnon-Bartsch and Speed, 2012], two batch effect correction tools largely used in genomics, we decompose the voxel intensities of images registered to a template into a biological component and an unwanted variation component. The unwanted variation component is estimated from a control region obtained from the cerebrospinal fluid (CSF), where intensities are known to be unassociated with disease status and other clinical covariates. We perform a singular value decomposition (SVD) of the control voxels to estimate factors of unwanted variation. We then estimate the unwanted factors using linear regression for every voxel of the brain and take the residuals as the RAVEL-corrected intensities. We assess the performance of RAVEL using T1-weighted (T1-w) images from more than 900 subjects with Alzheimer’s disease (AD) and mild cognitive impairment (MCI), as well as healthy controls from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. We compare RAVEL to intensity-normalization-only methods, histogram matching, and White Stripe. We show that RAVEL performs best at improving the replicability of the brain regions that are empirically found to be most associated with AD, and that these regions are significantly more present in structures impacted by AD (hippocampus, amygdala, parahippocampal gyrus, enthorinal area and fornix stria terminals). In addition, we show that the RAVEL-corrected intensities have the best performance in distinguishing between MCI subjects and healthy subjects by using the mean hippocampal intensity (AUC=67%), a marked improvement compared to results from intensity normalization alone (AUC=63% and 59% for histogram matching and White Stripe, respectively). RAVEL is generalizable to many imaging modalities, and shows promise for longitudinal studies. Additionally, because the choice of the control region is left to the user, RAVEL can be applied in studies of many brain disorders
Thrombolytic removal of intraventricular haemorrhage in treatment of severe stroke: results of the randomised, multicentre, multiregion, placebo-controlled CLEAR III trial
Background:
Intraventricular haemorrhage is a subtype of intracerebral haemorrhage, with 50% mortality and serious disability for survivors. We aimed to test whether attempting to remove intraventricular haemorrhage with alteplase versus saline irrigation improved functional outcome.
Methods:
In this randomised, double-blinded, placebo-controlled, multiregional trial (CLEAR III), participants with a routinely placed extraventricular drain, in the intensive care unit with stable, non-traumatic intracerebral haemorrhage volume less than 30 mL, intraventricular haemorrhage obstructing the 3rd or 4th ventricles, and no underlying pathology were adaptively randomly assigned (1:1), via a web-based system to receive up to 12 doses, 8 h apart of 1 mg of alteplase or 0·9% saline via the extraventricular drain. The treating physician, clinical research staff, and participants were masked to treatment assignment. CT scans were obtained every 24 h throughout dosing. The primary efficacy outcome was good functional outcome, defined as a modified Rankin Scale score (mRS) of 3 or less at 180 days per central adjudication by blinded evaluators. This study is registered with ClinicalTrials.gov, NCT00784134.
Findings:
Between Sept 18, 2009, and Jan 13, 2015, 500 patients were randomised: 249 to the alteplase group and 251 to the saline group. 180-day follow-up data were available for analysis from 246 of 249 participants in the alteplase group and 245 of 251 participants in the placebo group. The primary efficacy outcome was similar in each group (good outcome in alteplase group 48% vs saline 45%; risk ratio [RR] 1·06 [95% CI 0·88–1·28; p=0·554]). A difference of 3·5% (RR 1·08 [95% CI 0·90–1·29], p=0·420) was found after adjustment for intraventricular haemorrhage size and thalamic intracerebral haemorrhage. At 180 days, the treatment group had lower case fatality (46 [18%] vs saline 73 [29%], hazard ratio 0·60 [95% CI 0·41–0·86], p=0·006), but a greater proportion with mRS 5 (42 [17%] vs 21 [9%]; RR 1·99 [95% CI 1·22–3·26], p=0·007). Ventriculitis (17 [7%] alteplase vs 31 [12%] saline; RR 0·55 [95% CI 0·31–0·97], p=0·048) and serious adverse events (114 [46%] alteplase vs 151 [60%] saline; RR 0·76 [95% CI 0·64–0·90], p=0·002) were less frequent with alteplase treatment. Symptomatic bleeding (six [2%] in the alteplase group vs five [2%] in the saline group; RR 1·21 [95% CI 0·37–3·91], p=0·771) was similar.
Interpretation:
In patients with intraventricular haemorrhage and a routine extraventricular drain, irrigation with alteplase did not substantially improve functional outcomes at the mRS 3 cutoff compared with irrigation with saline. Protocol-based use of alteplase with extraventricular drain seems safe. Future investigation is needed to determine whether a greater frequency of complete intraventricular haemorrhage removal via alteplase produces gains in functional status
ISLES 2015 - A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI
Ischemic stroke is the most common cerebrovascular disease, and its diagnosis, treatment, and study relies on non-invasive imaging. Algorithms for stroke lesion segmentation from magnetic resonance imaging (MRI) volumes are intensely researched, but the reported results are largely incomparable due to different datasets and evaluation schemes. We approached this urgent problem of comparability with the Ischemic Stroke Lesion Segmentation (ISLES) challenge organized in conjunction with the MICCAI 2015 conference. In this paper we propose a common evaluation framework, describe the publicly available datasets, and present the results of the two sub-challenges: Sub-Acute Stroke Lesion Segmentation (SISS) and Stroke Perfusion Estimation (SPES). A total of 16 research groups participated with a wide range of state-of-the-art automatic segmentation algorithms. A thorough analysis of the obtained data enables a critical evaluation of the current state-of-the-art, recommendations for further developments, and the identification of remaining challenges. The segmentation of acute perfusion lesions addressed in SPES was found to be feasible. However, algorithms applied to sub-acute lesion segmentation in SISS still lack accuracy. Overall, no algorithmic characteristic of any method was found to perform superior to the others. Instead, the characteristics of stroke lesion appearances, their evolution, and the observed challenges should be studied in detail. The annotated ISLES image datasets continue to be publicly available through an online evaluation system to serve as an ongoing benchmarking resource (www.isles-challenge.org).Peer reviewe
- …