Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses

Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis that can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling but also from decisions regarding the quantification of the measured behavior. In this study, we gave the same speech-production data set to 46 teams of researchers and asked them to answer the same research question, resulting in substantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further found little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise, or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system, and calibrate their (un)certainty in their conclusions

COPDGene® 2019: Redefining the Diagnosis of Chronic Obstructive Pulmonary Disease

Background:Chronic obstructive pulmonary disease (COPD) remains a major cause of morbidity and mortality. Present-day diagnostic criteria are largely based solely on spirometric criteria. Accumulating evidence has identified a substantial number of individuals without spirometric evidence of COPD who suffer from respiratory symptoms and/or increased morbidity and mortality. There is a clear need for an expanded definition of COPD that is linked to physiologic, structural (computed tomography [CT]) and clinical evidence of disease. Using data from the COPD Genetic Epidemiology study (COPDGene®), we hypothesized that an integrated approach that includes environmental exposure, clinical symptoms, chest CT imaging and spirometry better defines disease and captures the likelihood of progression of respiratory obstruction and mortality. Methods:Four key disease characteristics - environmental exposure (cigarette smoking), clinical symptoms (dyspnea and/or chronic bronchitis), chest CT imaging abnormalities (emphysema, gas trapping and/or airway wall thickening), and abnormal spirometry - were evaluated in a group of 8784 current and former smokers who were participants in COPDGene® Phase 1. Using these 4 disease characteristics, 8 categories of participants were identified and evaluated for odds of spirometric disease progression (FEV1 > 350 ml loss over 5 years), and the hazard ratio for all-cause mortality was examined. Results:Using smokers without symptoms, CT imaging abnormalities or airflow obstruction as the reference population, individuals were classified as Possible COPD, Probable COPD and Definite COPD. Current Global initiative for obstructive Lung Disease (GOLD) criteria would diagnose 4062 (46%) of the 8784 study participants with COPD. The proposed COPDGene® 2019 diagnostic criteria would add an additional 3144 participants. Under the new criteria, 82% of the 8784 study participants would be diagnosed with Possible, Probable or Definite COPD. These COPD groups showed increased risk of disease progression and mortality. Mortality increased in patients as the number of their COPD characteristics increased, with a maximum hazard ratio for all cause-mortality of 5.18 (95% confidence interval [CI]: 4.15-6.48) in those with all 4 disease characteristics. Conclusions:A substantial portion of smokers with respiratory symptoms and imaging abnormalities do not manifest spirometric obstruction as defined by population normals. These individuals are at significant risk of death and spirometric disease progression. We propose to redefine the diagnosis of COPD through an integrated approach using environmental exposure, clinical symptoms, CT imaging and spirometric criteria. These expanded criteria offer the potential to stimulate both current and future interventions that could slow or halt disease progression in patients before disability or irreversible lung structural changes develop