Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small “many analyst” study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future

Towards an understanding of retouch flakes: A use-wear blind test on knapped stone microdebitage

The retouching and resharpening of lithic tools during their production and maintenance leads to the production of large numbers of small flakes and chips known as microdebitage. Standard analytical approaches to this material involves the mapping of microartefact densities to identify activity areas, and the creation of techno-typologies to characterise the form of retouch flakes from different types of tools. Whilst use-wear analysis is a common approach to the analysis of tools, it has been applied much less commonly to microdebitage. This paper contends that the use-wear analysis of microdebitage holds great potential for identifying activity areas on archaeological sites, representing a relatively unexplored analytical resource within microartefact assemblages. In order to test the range of factors that affect the identification of use-wear traces on small retouch flakes, a blind test consisting of 40 retouch flakes was conducted. The results show that wear traces can be identified with comparable levels of accuracy to those reported for historic blind tests of standard lithic tools suggesting that the use-wear analysis of retouch flakes can be a useful analytical tool in understanding site function, and in increasing sample sizes in cases where assemblages contain few tools.BC Grant Number: 328642 Funder: European Commission FP7 Marie Curie Actions Intra-European Fellowship programme - Grant Number: AH/N007506/1 Funder: Arts and Humanities Research Council (UK) - Grant Number: HAR2016-75201-P Funder: Ministry of Science and Innovation, Spain - Grant Number: AH/L503939/1 Funder: Arts and Humanities Research Council (UK) through the South West and Wales Doctoral TrainingPeer reviewe

Clinical impact of NPM1-mutant molecular persistence after chemotherapy for acute myeloid leukemia

Abstract Monitoring of NPM1 mutant (NPM1mut) measurable residual disease (MRD) in acute myeloid leukemia (AML) has an established role in patients who are treated with intensive chemotherapy. The European LeukemiaNet has defined molecular persistence at low copy number (MP-LCN) as an MRD transcript level <1% to 2% with a <1-log change between any 2 positive samples collected after the end of treatment (EOT). Because the clinical impact of MP-LCN is unknown, we sought to characterize outcomes in patients with persistent NPM1mut MRD after EOT and identify factors associated with disease progression. Consecutive patients with newly diagnosed NPM1mut AML who received ≥2 cycles of intensive chemotherapy were included if bone marrow was NPM1mut MRD positive at the EOT, and they were not transplanted in first complete remission. One hundred patients were followed for a median of 23.5 months; 42% remained free of progression at 1 year, either spontaneously achieving complete molecular remission (CRMRD−; 30%) or retaining a low-level NPM1mut transcript (12% for ≥12 months and 9% at last follow-up). Forty percent met the criteria for MP-LCN. Preemptive salvage therapy significantly prolonged relapse-free survival. Risk factors associated with disease progression were concurrent FLT3-internal tandem duplication at diagnosis and suboptimal MRD response (NPM1mut reduction <4.4-log) at EOT

Impacts of Climate Change on the Global Invasion Potential of the African Clawed Frog Xenopus laevis

International audienceBy altering or eliminating delicate ecological relationships, non-indigenous species are con- sidered a major threat to biodiversity, as well as a driver of environmental change. Global cli- mate change affects ecosystems and ecological communities, leading to changes in the phenology, geographic ranges, or population abundance of several species. Thus, predicting the impacts of global climate change on the current and future distribution of invasive species is an important subject in macroecological studies. The African clawed frog (Xenopus laevis), native to South Africa, possesses a strong invasion potential and populations have become established in numerous countries across four continents. The global invasion potential of X. laevis was assessed using correlative species distribution models (SDMs). SDMs were com- puted based on a comprehensive set of occurrence records covering South Africa, North America, South America and Europe and a set of nine environmental predictors. Models were built using both a maximum entropy model and an ensemble approach integrating eight algo- rithms. The future occurrence probabilities for X. laevis were subsequently computed using bioclimatic variables for 2070 following four different IPCC scenarios. Despite minor differ- ences between the statistical approaches, both SDMs predict the future potential distribution of X. laevis, on a global scale, to decrease across all climate change scenarios. On a conti- nental scale, both SDMs predict decreasing potential distributions in the species’ native range in South Africa, as well as in the invaded areas in North and South America, and in Australia where the species has not been introduced. In contrast, both SDMs predict the potential range size to expand in Europe. Our results suggest that all probability classes will be equally affected by climate change. New regional conditions may promote new invasions or the spread of established invasive populations, especially in France and Great Britain

Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Gould E, Fraser H, Parker T, et al. Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology. 2023.Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different (mostly social science) fields, and has found substantial variability among results, despite analysts having the same data and research question. We implemented an analogous study in ecology and evolutionary biology, fields in which there have been no empirical exploration of the variation in effect sizes or model predictions generated by the analytical decisions of different researchers. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment), and the project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future

Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.</p