Evaluation of clinical prediction models (part 1):from development to external validation

Evaluating the performance of a clinical prediction model is crucial to establish its predictive accuracy in the populations and settings intended for use. In this article, the first in a three part series, Collins and colleagues describe the importance of a meaningful evaluation using internal, internal-external, and external validation, as well as exploring heterogeneity, fairness, and generalisability in model performance

Different perceptions of the burden of upper GI endoscopy: an empirical study in three patient groups

Background: Few studies have evaluated patients' perceived burden of cancer surveillance tests. Cancer screening and surveillance, however, require a large number of patients to undergo potentially burdensome tests with only some experiencing health gains from it. We investigated the determinants of patients' reported burden of upper gastrointestinal (GI) endoscopy by comparing data from three patient groups. Patients and methods: A total of 476 patients were included: 180 patients under regular surveillance for Barrett esophagus (BE), a premalignant disorder; 214 patients with non-specific upper GI symptoms (NS), and 82 patients recently diagnosed with upper GI cancer (CA). We assessed pain, discomfort and overall burden experienced during endoscopy, symptoms in the week afterwards and psychological distress over time (Hospital Anxiety and Depression scale and Impact of Event Scale). Results: Two-thirds (66%) of patients reported discomfort and overall burden of upper GI endoscopy. Only 23% reported any pain. BE patients reported significantly less discomfort, pain and overall burden than the other patients: those with NS reported more discomfort, CA patients more pain, and both more overall burden. These differences could be statistically explained by the number of previous endoscopies and whether sedation was provided or not, but not by patient characteristics. Conclusion: The perception of upper GI endoscopy varies by patient group, due to potential adaptation after multiple endoscopies and aspects of th

Factors Influencing the Statistical Power of Complex Data Analysis Protocols for Molecular Signature Development from Microarray Data

Critical to the development of molecular signatures from microarray and other high-throughput data is testing the statistical significance of the produced signature in order to ensure its statistical reproducibility. While current best practices emphasize sufficiently powered univariate tests of differential expression, little is known about the factors that affect the statistical power of complex multivariate analysis protocols for high-dimensional molecular signature development.We show that choices of specific components of the analysis (i.e., error metric, classifier, error estimator and event balancing) have large and compounding effects on statistical power. The effects are demonstrated empirically by an analysis of 7 of the largest microarray cancer outcome prediction datasets and supplementary simulations, and by contrasting them to prior analyses of the same data.THE FINDINGS OF THE PRESENT STUDY HAVE TWO IMPORTANT PRACTICAL IMPLICATIONS: First, high-throughput studies by avoiding under-powered data analysis protocols, can achieve substantial economies in sample required to demonstrate statistical significance of predictive signal. Factors that affect power are identified and studied. Much less sample than previously thought may be sufficient for exploratory studies as long as these factors are taken into consideration when designing and executing the analysis. Second, previous highly-cited claims that microarray assays may not be able to predict disease outcomes better than chance are shown by our experiments to be due to under-powered data analysis combined with inappropriate statistical tests

Criteria for evaluation of novel markers of cardiovascular risk: A scientific statement from the American Heart Association

There is increasing interest in utilizing novel markers of cardiovascular disease risk, and consequently, there is a need to assess the value of their use. This scientific statement reviews current concepts of risk evaluation and proposes standards for the critical appraisal of risk assessment methods. An adequate evaluation of a novel risk marker requires a sound research design, a representative at-risk population, and an adequate number of outcome events. Studies of a novel marker should report the degree to which it adds to the prognostic information provided by standard risk markers. No single statistical measure provides all the information needed to assess a novel marker, so measures of both discrimination and accuracy should be reported. The clinical value of a marker should be assessed by its effect on patient management and outcomes. In general, a novel risk marker should be evaluated in several phases, including initial proof of concept, prospective validation in independent populations, documentation of incremental information when added to standard risk markers, assessment of effects on patient management and outcomes, and ultimately, cost-effectiveness

Thermal Perceptual Thresholds are typical in Autism Spectrum Disorder but Strongly Related to Intra-individual Response Variability

Individuals with autism spectrum disorder (ASD) are often reported to exhibit an apparent indifference to pain or temperature. Leading models suggest that this behavior is the result of elevated perceptual thresholds for thermal stimuli, but data to support these assertions are inconclusive. An alternative proposal suggests that the sensory features of ASD arise from increased intra-individual perceptual variability. In this study, we measured method-of-limits warm and cool detection thresholds in 142 individuals (83 with ASD, 59 with typical development [TD], aged 7–54 years), testing relationships with diagnostic group, demographics, and clinical measures. We also investigated the relationship between detection thresholds and a novel measure of intra-individual (trial-to-trial) threshold variability, a putative index of “perceptual noise.” This investigation found no differences in thermal detection thresholds between individuals with ASD and typical controls, despite large differences between groups in sensory reactivity questionnaires and modest group differences in intra-individual variability. Lower performance IQ, male sex, and higher intra-individual variability in threshold estimates were the most significant predictors of elevated detection thresholds. Although no psychophysical measure was significantly correlated with questionnaire measures of sensory hyporeactivity, large intra-individual variability may partially explain the elevated psychophysical thresholds seen in a subset of the ASD population

Rare coding variants in PLCG2, ABI3, and TREM2 implicate microglial-mediated innate immunity in Alzheimer's disease

We identified rare coding variants associated with Alzheimer’s disease (AD) in a 3-stage case-control study of 85,133 subjects. In stage 1, 34,174 samples were genotyped using a whole-exome microarray. In stage 2, we tested associated variants (P<1×10-4) in 35,962 independent samples using de novo genotyping and imputed genotypes. In stage 3, an additional 14,997 samples were used to test the most significant stage 2 associations (P<5×10-8) using imputed genotypes. We observed 3 novel genome-wide significant (GWS) AD associated non-synonymous variants; a protective variant in PLCG2 (rs72824905/p.P522R, P=5.38×10-10, OR=0.68, MAFcases=0.0059, MAFcontrols=0.0093), a risk variant in ABI3 (rs616338/p.S209F, P=4.56×10-10, OR=1.43, MAFcases=0.011, MAFcontrols=0.008), and a novel GWS variant in TREM2 (rs143332484/p.R62H, P=1.55×10-14, OR=1.67, MAFcases=0.0143, MAFcontrols=0.0089), a known AD susceptibility gene. These protein-coding changes are in genes highly expressed in microglia and highlight an immune-related protein-protein interaction network enriched for previously identified AD risk genes. These genetic findings provide additional evidence that the microglia-mediated innate immune response contributes directly to AD development

Genetically elevated high-density lipoprotein cholesterol through the cholesteryl ester transfer protein gene does not associate with risk of Alzheimer's disease

Introduction: There is conflicting evidence whether high-density lipoprotein cholesterol (HDL-C) is a risk factor for Alzheimer's disease (AD) and dementia. Genetic variation in the cholesteryl ester transfer protein (CETP) locus is associated with altered HDL-C. We aimed to assess AD risk by genetically predicted HDL-C. Methods: Ten single nucleotide polymorphisms within the CETP locus predicting HDL-C were applied to the International Genomics of Alzheimer's Project (IGAP) exome chip stage 1 results in up 16,097 late onset AD cases and 18,077 cognitively normal elderly controls. We performed instrumental variables analysis using inverse variance weighting, weighted median, and MR-Egger. Results: Based on 10 single nucleotide polymorphisms distinctly predicting HDL-C in the CETP locus, we found that HDL-C was not associated with risk of AD (P > .7). Discussion: Our study does not support the role of HDL-C on risk of AD through HDL-C altered by CETP. This study does not rule out other mechanisms by which HDL-C affects risk of AD

A novel Alzheimer disease locus located near the gene encoding tau protein

This is the author accepted manuscript. The final version is available from the publisher via the DOI in this recordAPOE ε4, the most significant genetic risk factor for Alzheimer disease (AD), may mask effects of other loci. We re-analyzed genome-wide association study (GWAS) data from the International Genomics of Alzheimer's Project (IGAP) Consortium in APOE ε4+ (10 352 cases and 9207 controls) and APOE ε4- (7184 cases and 26 968 controls) subgroups as well as in the total sample testing for interaction between a single-nucleotide polymorphism (SNP) and APOE ε4 status. Suggestive associations (P<1 × 10-4) in stage 1 were evaluated in an independent sample (stage 2) containing 4203 subjects (APOE ε4+: 1250 cases and 536 controls; APOE ε4-: 718 cases and 1699 controls). Among APOE ε4- subjects, novel genome-wide significant (GWS) association was observed with 17 SNPs (all between KANSL1 and LRRC37A on chromosome 17 near MAPT) in a meta-analysis of the stage 1 and stage 2 data sets (best SNP, rs2732703, P=5·8 × 10-9). Conditional analysis revealed that rs2732703 accounted for association signals in the entire 100-kilobase region that includes MAPT. Except for previously identified AD loci showing stronger association in APOE ε4+ subjects (CR1 and CLU) or APOE ε4- subjects (MS4A6A/MS4A4A/MS4A6E), no other SNPs were significantly associated with AD in a specific APOE genotype subgroup. In addition, the finding in the stage 1 sample that AD risk is significantly influenced by the interaction of APOE with rs1595014 in TMEM106B (P=1·6 × 10-7) is noteworthy, because TMEM106B variants have previously been associated with risk of frontotemporal dementia. Expression quantitative trait locus analysis revealed that rs113986870, one of the GWS SNPs near rs2732703, is significantly associated with four KANSL1 probes that target transcription of the first translated exon and an untranslated exon in hippocampus (P≤1.3 × 10-8), frontal cortex (P≤1.3 × 10-9) and temporal cortex (P≤1.2 × 10-11). Rs113986870 is also strongly associated with a MAPT probe that targets transcription of alternatively spliced exon 3 in frontal cortex (P=9.2 × 10-6) and temporal cortex (P=2.6 × 10-6). Our APOE-stratified GWAS is the first to show GWS association for AD with SNPs in the chromosome 17q21.31 region. Replication of this finding in independent samples is needed to verify that SNPs in this region have significantly stronger effects on AD risk in persons lacking APOE ε4 compared with persons carrying this allele, and if this is found to hold, further examination of this region and studies aimed at deciphering the mechanism(s) are warranted