Selection bias, vote counting, and money-priming effects: A comment on Rohrer, Pashler, and Harris (2015) and Vohs (2015)

When a series of studies fails to replicate a well-documented effect, researchers might be tempted to use a "vote counting" approach to decide whether the effect is reliable-that is, simply comparing the number of successful and unsuccessful replications. Vohs's (2015) response to the absence of money priming effects reported by Rohrer, Pashler, and Harris (2015) provides an example of this approach. Unfortunately, vote counting is a poor strategy to assess the reliability of psychological findings because it neglects the impact of selection bias and questionable research practices. In the present comment, we show that a range of meta-analytic tools indicate irregularities in the money priming literature discussed by Rohrer et al. and Vohs, which all point to the conclusion that these effects are distorted by selection bias, reporting biases, or p-hacking. This could help to explain why money-priming effects have proven unreliable in a number of direct replication attempts in which biases have been minimized through preregistration or transparent reporting. Our major conclusion is that the simple proportion of significant findings is a poor guide to the reliability of research and that preregistered replications are an essential means to assess the reliability of money-priming effects. (PsycINFO Database Recor

Publication bias and low power in field studies on goal priming

Research on goal priming asks whether the subtle activation of an achievement goal can improve task performance. Studies in this domain employ a range of priming methods, such as surreptitiously displaying a photograph of an athlete winning a race, and a range of dependent variables including measures of creativity and workplace performance. Chen, Latham, Piccolo and Itzchakov (Chen et al. 2021 J. Appl. Psychol. 70, 216-253) recently undertook a meta-analysis of this research and reported positive overall effects in both laboratory and field studies, with field studies yielding a moderate-to-large effect that was significantly larger than that obtained in laboratory experiments. We highlight a number of issues with Chen et al.'s selection of field studies and then report a new meta-analysis (k = 13, N = 683) that corrects these. The new meta-analysis reveals suggestive evidence of publication bias and low power in goal priming field studies. We conclude that the available evidence falls short of demonstrating goal priming effects in the workplace, and offer proposals for how future research can provide stronger tests

Mapping the Reliability Multiverse of Contextual Cuing

Cronbach (1957) famously noted the divergence between the experimental and psychometric traditions in psychology and called for a unification, but many domains of cognitive experimental psychology continue to pay minimal heed to basic psychometric principles. The present article considers the lack of attention devoted to the reliability of measures extracted in a popular visual search task for studying putatively unconscious mental processes, contextual cuing, and the inferential fallacies that this neglect can cause. Two experiments (total N = 200) demonstrated that the reliability of contextual cuing and awareness measures can be increased by three manipulations designed to increase between-participant variability in search performance. At the same time, the data were subjected to a multiverse analysis, which found that specific data preprocessing pipelines result in more reliable estimates. Nevertheless, the reliability estimates remained too low for drawing firm conclusions from standard statistical techniques. Interpreting results from analyses based on individual differences, such as the typical low correlations between implicit and explicit measures, will be challenging so long as the underlying measures have poor reliability

Do Incidental Environmental Anchors Bias Consumers’ Price Estimations?

It is well-established that decision makers bias their estimates of unknown quantities in the direction of a salient numerical anchor. Some standard anchoring paradigms have been shown to yield pervasive biases, such as Tversky and Kahneman’s (1974) classic 2-step task which includes a comparative question followed by an estimation question. In contrast there is much less evidence for the claim that incidental environmental anchors can produce assimilative effects on judgments, such as the amount people are willing to pay for a meal being greater at a restaurant called Studio 97 compared to one called Studio 17. Three studies are reported in which the basic incidental environmental anchoring method of Critcher and Gilovich (2008) is employed to measure consumer price estimations. No statistically significant evidence of incidental anchoring was obtained. In contrast, robust standard anchoring effects were found. The results suggest that anchoring is limited to situations which require explicit thinking about the anchor

Correlation analysis to investigate unconscious mental processes: A critical appraisal and mini-tutorial.

As a method to investigate the scope of unconscious mental processes, researchers frequently obtain concurrent measures of task performance and stimulus awareness across participants. Even though both measures might be significantly greater than zero, the correlation between them might not, encouraging the inference that an unconscious process drives task performance. We highlight the pitfalls of this null-correlation approach and provide a mini-tutorial on ways to avoid them. As reference, we use a recent study by Salvador et al. (2018) reporting a non-significant correlation between the extent to which memory was suppressed by a Think/No-Think cue and an index of cue awareness. In the Null Hypothesis Significance Testing (NHST) framework, it is inappropriate to interpret failure to reject the null hypothesis (i.e., correlation = 0) as evidence for the null. Furthermore, psychological measures are often unreliable, which can dramatically attenuate the size of observed correlations. A Bayesian approach can circumvent both problems and compare the extent to which the data provide evidence for the null versus the alternative hypothesis (i.e., correlation > 0), while considering the usually low reliabilities of the variables. Applied to Salvador et al.'s data, this approach indicates no to moderate support for the claimed unconscious nature of participants' memory-suppression performance-depending on the model of the alternative hypothesis. Hence, more reliable data are needed. When analyzing correlational data, we recommend researchers to employ the Bayesian methods developed here (and made freely available as R scripts), rather than standard NHST methods, to take account of unreliability

Rethinking Attentional Habits

Attentional habits acquired by visual statistical learning cause enduring biases toward specific locations. These habits, driven by recent search history, are thought to be independent of both goal-directed and stimulus-driven attentional mechanisms. This theoretical claim is based on three characteristics that these habits apparently exhibit, that is, they are inflexible, implicit, and efficient. We review methodological limitations in previous studies and briefly describe recent results that challenge this new framework. We conclude that it might be premature to assume that attentional habits are based on a special search history process that differs from the two traditionally recognized attentional mechanisms

Is probabilistic cuing of visual search an inflexible attentional habit? A meta-analytic review

In studies on probabilistic cuing of visual search, participants search for a target among several distractors and report some feature of the target. In a biased stage the target appears more frequently in one specific area of the search display. Eventually, participants become faster at finding the target in that rich region compared to the sparse region. In some experiments, this stage is followed by an unbiased stage, where the target is evenly located across all regions of the display. Despite this change in the spatial distribution of targets, search speed usually remains faster when the target is located in the previously rich region. The persistence of the bias even when it is no longer advantageous has been taken as evidence that this phenomenon is an attentional habit. The aim of this meta-analysis was to test whether the magnitude of probabilistic cuing decreases from the biased to the unbiased stage. A meta-analysis of 42 studies confirmed that probabilistic cuing during the unbiased stage was roughly half the size of cuing during the biased stage, and this decrease persisted even after correcting for publication bias. Thus, the evidence supporting the claim that probabilistic cuing is an attentional habit might not be as compelling as previously thought

Raising awareness about measurement error in research on unconscious mental processes

Experimental psychologists often neglect the poor psychometric properties of the dependent measures collected in their studies. In particular, a low reliability of measures can have dramatic consequences for the interpretation of key findings in some of the most popular experimental paradigms, especially when strong inferences are drawn from the absence of statistically significant correlations. In research on unconscious cognition, for instance, it is commonly argued that the lack of a correlation between task performance and measures of awareness or explicit recollection of the target stimuli provides strong support for the conclusion that the cognitive processes underlying performance must be unconscious. Using contextual cuing of visual search as a case study, we show that given the low reliability of the dependent measures collected in these studies, it is usually impossible to draw any firm conclusion about the unconscious character of this effect from correlational analyses. Furthermore, both a psychometric meta-analysis of the available evidence and a cognitive-modeling approach suggest that, in fact, we should expect to see very low correlations between performance and awareness at the empirical level, even if both constructs are perfectly related at the latent level. Convincing evidence for the unconscious character of contextual cuing and other effects will most likely demand richer and larger data sets, coupled with more powerful analytic approaches

Selection bias, vote counting, and money-priming effects: A comment on Rohrer, Pashler, and Harris (2015) and Vohs (2015)

When a series of studies fails to replicate a well-documented effect, researchers might be tempted to use a "vote counting" approach to decide whether the effect is reliable-that is, simply comparing the number of successful and unsuccessful replications. Vohs's (2015) response to the absence of money priming effects reported by Rohrer, Pashler, and Harris (2015) provides an example of this approach. Unfortunately, vote counting is a poor strategy to assess the reliability of psychological findings because it neglects the impact of selection bias and questionable research practices. In the present comment, we show that a range of meta-analytic tools indicate irregularities in the money priming literature discussed by Rohrer et al. and Vohs, which all point to the conclusion that these effects are distorted by selection bias, reporting biases, or p-hacking. This could help to explain why money-priming effects have proven unreliable in a number of direct replication attempts in which biases have been minimized through preregistration or transparent reporting. Our major conclusion is that the simple proportion of significant findings is a poor guide to the reliability of research and that preregistered replications are an essential means to assess the reliability of money-priming effects. (PsycINFO Database Recor

Pre-exposure of repeated search configurations facilitates subsequent contextual cuing of visual search

Contextual cuing is the enhancement of visual search when the configuration of distractors has been experienced previously. It has been suggested that contextual cuing relies on associative learning between the distractor locations and the target position. Four experiments examined the effect of pre-exposing configurations of consistent distractors on subsequent contextual cuing. The findings demonstrate a facilitation of subsequent cuing for pre-exposed configurations compared to novel configurations that have not been pre-exposed. This suggests that learning of repeated visual search patterns involves acquisition of not just distractor-target associations, but also associations between distractors within the search context, an effect that is not captured by the Brady and Chun (2007) connectionist model of contextual cuing. A new connectionist model of contextual cuing is proposed that learns associations between repeated distractor stimuli, enabling it to predict an effect of pre-exposure on contextual cuing