Quantifying Support for the Null Hypothesis in Psychology: An Empirical Investigation

In the traditional statistical framework, nonsignificant results leave researchers in a state of suspended disbelief. In this study, we examined, empirically, the treatment and evidential impact of nonsignificant results. Our specific goals were twofold: to explore how psychologists interpret and communicate nonsignificant results and to assess how much these results constitute evidence in favor of the null hypothesis. First, we examined all nonsignificant findings mentioned in the abstracts of the 2015 volumes of Psychonomic Bulletin & Review, Journal of Experimental Psychology: General, and Psychological Science (N = 137). In 72% of these cases, nonsignificant results were misinterpreted, in that the authors inferred that the effect was absent. Second, a Bayes factor reanalysis revealed that fewer than 5% of the nonsignificant findings provided strong evidence (i.e., BF01 > 10) in favor of the null hypothesis over the alternative hypothesis. We recommend that researchers expand their statistical tool kit in order to correctly interpret nonsignificant results and to be able to evaluate the evidence for and against the null hypothesis

A Consensus-Based Transparency Checklist

We present a consensus-based checklist to improve and document the transparency of research reports in social and behavioural research. An accompanying online application allows users to complete the form and generate a report that they can submit with their manuscript or post to a public repository

Author Correction: A consensus-based transparency checklist.

An amendment to this paper has been published and can be accessed via a link at the top of the paper

Consensus-based guidance for conducting and reporting multi-analyst studies

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research

Science Forum: Consensus-based guidance for conducting and reporting multi-analyst studies

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research

Consensus-based guidance for conducting and reporting multi-analyst studies

International audienceAny large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research

A consensus-based transparency checklist

We present a consensus-based checklist to improve and document the transparency of research reports in social and behavioural research. An accompanying online application allows users to complete the form and generate a report that they can submit with their manuscript or post to a public repository

A Worldwide Test of the Predictive Validity of Ideal Partner Preference-Matching

©American Psychological Association, [2024]. This paper is not the copy of record and may not exactly replicate the authoritative document published in the APA journal. The final article is available, upon publication, at: [ARTICLE DOI]”Ideal partner preferences (i.e., ratings of the desirability of attributes like attractiveness or intelligence) are the source of numerous foundational findings in the interdisciplinary literature on human mating. Recently, research on the predictive validity of ideal partner preference-matching (i.e., do people positively evaluate partners who match versus mismatch their ideals?) has become mired in several problems. First, articles exhibit discrepant analytic and reporting practices. Second, different findings emerge across laboratories worldwide, perhaps because they sample different relationship contexts and/or populations. This registered report—partnered with the Psychological Science Accelerator—uses a highly powered design (N=10,358) across 43 countries and 22 languages to estimate preference-matching effect sizes. The most rigorous tests revealed significant preference-matching effects in the whole sample and for partnered and single participants separately. The “corrected pattern metric” that collapses across 35 traits revealed a zero-order effect of β=.19 and an effect of β=.11 when included alongside a normative preference-matching metric. Specific traits in the “level metric” (interaction) tests revealed very small (average β=.04) effects. Effect sizes were similar for partnered participants who reported ideals before entering a relationship, and there was no consistent evidence that individual differences moderated any effects. Comparisons between stated and revealed preferences shed light on gender differences and similarities: For attractiveness, men’s and (especially) women’s stated preferences underestimated revealed preferences (i.e., they thought attractiveness was less important than it actually was). For earning potential, men’s stated preferences underestimated—and women’s stated preferences overestimated—revealed preferences. Implications for the literature on human mating are discussed.Unfunde

Many Labs 5:Testing pre-data collection peer review as an intervention to increase replicability

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3?9; median total sample = 1,279.5, range = 276?3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (?r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols (r = .05) was similar to that of the RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than that of the original studies (r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00?.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19?.50)