Manipulating the alpha level cannot cure significance testing

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p = 0.05 to p = 0.005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of 0.05, 0.01, 0.005, or anything else, is not acceptable

Brain Processing of Complex Geometric Forms in a Visual Memory Task Increases P2 Amplitude

We study the cognitive processing of visual working memory in three different conditions of memory load and configuration change. Altering this features has been shown to alter the brain’s processing in memory tasks. Most studies dealing with this issue have used the verbal-phonological modality. We use complex geometric polygons to assess visual working memory in a modified change detection task. Three different types of backgrounds were used to manipulate memory loading and 18 complex geometric polygons to manipulate stimuli configuration. The goal of our study was to test whether the memory load and configuration affect the correct-recall ratios. We expected that increasing visual items loading and changing configuration of items would induce differences in working memory performance. Brain activity related to the task was assessed through event-related potentials (ERP), during the test phase of each trial. Our results showed that visual items loading and changing of item configuration affect working memory on test phase on ERP component P2, but does not affect performance. However frontal related ERP component—P3—was minimally affected by visual memory loading or configuration changing, supporting that working memory is related to a filtering processing in posterior brain regions

Manipulating the Alpha Level Cannot Cure Significance Testing

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p = 0.05 to p = 0.005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of 0.05, 0.01, 0.005, or anything else, is not acceptable