Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis

Background: The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq’s Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the United States Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Community Directive (ICD) 203 – and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products.Methods: Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments.Results: Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability.Conclusion: Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards

Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies

New genetic loci link adipose and insulin biology to body fat distribution.

Body fat distribution is a heritable trait and a well-established predictor of adverse metabolic outcomes, independent of overall adiposity. To increase our understanding of the genetic basis of body fat distribution and its molecular links to cardiometabolic traits, here we conduct genome-wide association meta-analyses of traits related to waist and hip circumferences in up to 224,459 individuals. We identify 49 loci (33 new) associated with waist-to-hip ratio adjusted for body mass index (BMI), and an additional 19 loci newly associated with related waist and hip circumference measures (P < 5 × 10(-8)). In total, 20 of the 49 waist-to-hip ratio adjusted for BMI loci show significant sexual dimorphism, 19 of which display a stronger effect in women. The identified loci were enriched for genes expressed in adipose tissue and for putative regulatory elements in adipocytes. Pathway analyses implicated adipogenesis, angiogenesis, transcriptional regulation and insulin resistance as processes affecting fat distribution, providing insight into potential pathophysiological mechanisms

The low statistical power of psychological research: Causes, consequences and potential remedies

© 2020 Felix Singleton ThornThis dissertation examines two major issues in psychological research: formal sample size planning and reporting biases. It is organized into three main parts. The first part examines the history of formal sample size planning and reporting biases in the psychology research literature, outlining the history of the dominant approach to statistical analysis (Chapter 2), demonstrating the implications of low statistical power and reporting biases on research literatures (Chapter 3), and examining the history of statistical power analysis as represented in the psychology research literature (Chapter 4). The second part of this dissertation examines psychologists’ research and publication practices. Chapter 5 presents a meta-analysis of previous power surveys and finds that the average statistical power of psychology research at Cohen’s small and medium effect size benchmarks was lower than typical goal levels and that this value remained approximately constant from the 1960s to 2014. Chapter 6 presents an analysis of more than 130,000 effect size estimates from over 9,000 articles published in 5 APA journals from 1985 to 2013 and finds that the average effect size reported in this body of psychological research decreased over time. Together Chapters 5 and 6 suggest that the average statistical power of psychological research remained stable or may even have decreased over time. In order to investigate why this is the case, Chapter 7 presents the results of a survey of researchers from across fields of psychological research about their research planning practices. This survey highlights the most important barriers that prevent researchers from using formal sample size planning during the design phase of their research and shows that while most researchers believe statistical power is important for their research purposes, practical constraints act to limit achieved sample sizes in most studies. The final part of this thesis examines the implications of low statistical power and reporting biases on scientific research and provides suggestions on how research planning methods could be improved. Bringing together all of the previous large-scale replication projects that have been conducted in the behavioral sciences, Chapter 8 shows that effect sizes in replication studies are, on average, considerably lower than those reported in original studies, and quantifies the substantial heterogeneity in this value across replication projects. Finally, Chapter 9 examines sample size planning efforts reported in recent Psychological Science articles and uses this to illustrate a guide to effect size selection for formal sample size planning. Together, this dissertation shows that low statistical power and reporting biases remain serious problems for the behavioral sciences research literature. Contrasting the long history of efforts to improve the statistical power of psychology research with the lack of change in the average power of research from 1962 to 2014, I argue that new methods of avoiding the negative impact of low statistical power and reporting biases are necessary. Several recent publication and methodological developments, namely (a) preregistration, (b) pre-prints and data repositories, (c) the registered reports publication format and (d) the increasing use of large scale collaborative research projects, provide possible mechanisms with which to reduce the negative impact of low statistical power and reporting biases on the published scientific literature

Approaches to effect size selection for power analysis

Researchers routinely have to decide upon the sample size they include in their research. When formal sample size planning is used it is important to understand that the approach to sample size selection (e.g., AIPE or power analysis) as well as the method used to develop the alternative hypothesis (i.e., the effect sizes and parameter estimates used in power analysis) has important implications for the appropriate interpretation of the results. This paper presents the results of analysis of the sample size planning approach used in 121 empirical research articles published in the November 2017 to August 2018 issues of Psychological Science, and uses the results of this analysis to illustrate a guide to sample size planning under the most common methods of sample size determination (power analysis, Accuracy in Parameter Estimation, Statistical Assurance, and Bayesian sample size determination)

Meta-analysis of Power Surveys of Psychological Research

This project is a systematic review and meta-analysis of the previous studies of statistical power in psychology