Manipulating the alpha level cannot cure significance testing

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p = 0.05 to p = 0.005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of 0.05, 0.01, 0.005, or anything else, is not acceptable

A many-analysts approach to the relation between religiosity and well-being

The relation between religiosity and well-being is one of the most researched topics in the psychology of religion, yet the directionality and robustness of the effect remains debated. Here, we adopted a many-analysts approach to assess the robustness of this relation based on a new cross-cultural dataset (N=10,535 participants from 24 countries). We recruited 120 analysis teams to investigate (1) whether religious people self-report higher well-being, and (2) whether the relation between religiosity and self-reported well-being depends on perceived cultural norms of religion (i.e., whether it is considered normal and desirable to be religious in a given country). In a two-stage procedure, the teams first created an analysis plan and then executed their planned analysis on the data. For the first research question, all but 3 teams reported positive effect sizes with credible/confidence intervals excluding zero (median reported β=0.120). For the second research question, this was the case for 65% of the teams (median reported β=0.039). While most teams applied (multilevel) linear regression models, there was considerable variability in the choice of items used to construct the independent variables, the dependent variable, and the included covariates

A Many-analysts Approach to the Relation Between Religiosity and Well-being

The relation between religiosity and well-being is one of the most researched topics in the psychology of religion, yet the directionality and robustness of the effect remains debated. Here, we adopted a many-analysts approach to assess the robustness of this relation based on a new cross-cultural dataset (N = 10, 535 participants from 24 countries). We recruited 120 analysis teams to investigate (1) whether religious people self-report higher well-being, and (2) whether the relation between religiosity and self-reported well-being depends on perceived cultural norms of religion (i.e., whether it is considered normal and desirable to be religious in a given country). In a two-stage procedure, the teams first created an analysis plan and then executed their planned analysis on the data. For the first research question, all but 3 teams reported positive effect sizes with credible/confidence intervals excluding zero (median reported β = 0.120). For the second research question, this was the case for 65% of the teams (median reported β = 0.039). While most teams applied (multilevel) linear regression models, there was considerable variability in the choice of items used to construct the independent variables, the dependent variable, and the included covariates

Crowdsourcing hypothesis tests: Making transparent how design choices shape research results

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer fiveoriginal research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams renderedstatistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.</div

Multidimensional signals and analytic flexibility: Estimating degrees of freedom in human speech analyses

Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis which can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling, but also from decisions regarding the quantification of the measured behavior. In the present study, we gave the same speech production data set to 46 teams of researchers and asked them to answer the same research question, resulting insubstantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further find little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system and calibrate their (un)certainty in their conclusions

Understanding rumination as a form of inner speech : probing the role of motor processes

Rumination is known to be a predominantly verbal process and has been proposed to be considered as such as a dysfunctional form of inner speech (i.e., the silent production of words in one’s mind). On the other hand, research on the psychophysiology of inner speech revealed that the neural processes involved in overt and covert speech tend to be very similar. This is coherent with the idea that some forms of inner speech could be considered as a kind of simulation of overt speech, in the same way as imagined actions can be considered as the result of a simulation of the corresponding overt action (e.g., walking and imagined walking). In other words, the motor simulation hypothesis suggests that the speech motor system should be involved as well during inner speech production. The corollary hypothesis might be drawn, according to which the production of inner speech (and rumination) should be disrupted by a disruption of the speech motor system. We conducted a series of five studies aiming to probe the role of the speech motor system in rumination. Overall, our results highlight that although verbal rumination may be considered as a form of inner speech, it might not specifically involve the speech motor system. More precisely, we argue that rumination might be considered as a particularly strongly condensed form of inner speech that does not systematically involve fully specified articulatory features. We discuss these findings in relation to the habit-goal framework of depressive rumination and we discuss the implications of these findings for theories of inner speech production.Les ruminations mentales sont majoritairement exprimées sous forme verbale et il a été proposé de les considérer, par conséquent, comme une forme dysfonctionnelle de parole intérieure (i.e., production mentale de parole). D’autre part, les recherches sur la psychophysiologie de la parole intérieure ont révélé que les processus neuronaux impliqués dans la parole à voix haute et dans la parole intérieure sont similaires. Ces observations sont cohérentes avec l’idée que certaines formes de parole intérieure pourraient être considérées comme une forme de simulation de la parole à voix haute, de la même manière que certaines actions imaginées peuvent être considérées comme le résultat d’une simulation de l’action correspondante (par exemple, marcher et s’imaginer en train de marcher). En d’autres termes, l’hypothèse de la simulation motrice suggère que le système moteur de la parole devrait également être impliqué lors de la production de parole intérieure. L’hypothèse corollaire peut être formulée, selon laquelle la production de parole intérieure (et de ruminations) devrait être perturbée par une perturbation du système moteur de la parole. Nous avons mené une série de cinq études visant à sonder le rôle du système moteur de la parole dans les ruminations. Dans l’ensemble, nos résultats soulignent que, bien que la rumination verbale puisse être considérée comme une forme de parole intérieure, elle ne semble pas recruter spécifiquement le système moteur de la parole. Plus précisément, nous soutenons que la rumination peut être considérée comme une forme de parole intérieure particulièrement condensée, qui s’exprimerait sous la forme d’une représentation phonologique, et dont les traits articulatoires ne seraient pas complètements spécifiés. Nous faisons le lien entre ces résultats et l’hypothèse théorique du cadre “habitude-but” de la rumination dépressive et nous discutons de leurs implications pour les théories de la production de parole intérieure

Comprendre les ruminations mentales comme une forme de parole intérieure : examiner le rôle des processus moteurs

Les ruminations mentales sont majoritairement exprimées sous forme verbale et il a été proposé de les considérer, par conséquent, comme une forme dysfonctionnelle de parole intérieure (i.e., production mentale de parole). D’autre part, les recherches sur la psychophysiologie de la parole intérieure ont révélé que les processus neuronaux impliqués dans la parole à voix haute et dans la parole intérieure sont similaires. Ces observations sont cohérentes avec l’idée que certaines formes de parole intérieure pourraient être considérées comme une forme de simulation de la parole à voix haute, de la même manière que certaines actions imaginées peuvent être considérées comme le résultat d’une simulation de l’action correspondante (par exemple, marcher et s’imaginer en train de marcher). En d’autres termes, l’hypothèse de la simulation motrice suggère que le système moteur de la parole devrait également être impliqué lors de la production de parole intérieure. L’hypothèse corollaire peut être formulée, selon laquelle la production de parole intérieure (et de ruminations) devrait être perturbée par une perturbation du système moteur de la parole. Nous avons mené une série de cinq études visant à sonder le rôle du système moteur de la parole dans les ruminations. Dans l’ensemble, nos résultats soulignent que, bien que la rumination verbale puisse être considérée comme une forme de parole intérieure, elle ne semble pas recruter spécifiquement le système moteur de la parole. Plus précisément, nous soutenons que la rumination peut être considérée comme une forme de parole intérieure particulièrement condensée, qui s’exprimerait sous la forme d’une représentation phonologique, et dont les traits articulatoires ne seraient pas complètements spécifiés. Nous faisons le lien entre ces résultats et l’hypothèse théorique du cadre “habitude-but” de la rumination dépressive et nous discutons de leurs implications pour les théories de la production de parole intérieure.Rumination is known to be a predominantly verbal process and has been proposed to be considered as such as a dysfunctional form of inner speech (i.e., the silent production of words in one’s mind). On the other hand, research on the psychophysiology of inner speech revealed that the neural processes involved in overt and covert speech tend to be very similar. This is coherent with the idea that some forms of inner speech could be considered as a kind of simulation of overt speech, in the same way as imagined actions can be considered as the result of a simulation of the corresponding overt action (e.g., walking and imagined walking). In other words, the motor simulation hypothesis suggests that the speech motor system should be involved as well during inner speech production. The corollary hypothesis might be drawn, according to which the production of inner speech (and rumination) should be disrupted by a disruption of the speech motor system. We conducted a series of five studies aiming to probe the role of the speech motor system in rumination. Overall, our results highlight that although verbal rumination may be considered as a form of inner speech, it might not specifically involve the speech motor system. More precisely, we argue that rumination might be considered as a particularly strongly condensed form of inner speech that does not systematically involve fully specified articulatory features. We discuss these findings in relation to the habit-goal framework of depressive rumination and we discuss the implications of these findings for theories of inner speech production

A fully automated, transparent, reproducible, and blind protocol for sequential analyses

International audienceDespite many cultural, methodological, and technical improvements, one of the major obstacle to results reproducibility remains the pervasive low statistical power. In response to this problem, a lot of attention has recently been drawn to sequential analyses. This type of procedure has been shown to be more efficient (to require less observations and therefore less resources) than classical fixed-N procedures. However, these procedures are submitted to both intrapersonal and interpersonal biases during data collection and data analysis. In this tutorial, we explain how automation can be used to prevent these biases. We show how to synchronise open and free experiment software programs with the Open Science Framework and how to automate sequential data analyses in R. This tutorial is intended to researchers with beginner experience with R but no previous experience with sequential analyses is required