The Process of Replication Target Selection in Psychology: What to Consider?

Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well justified, systematic, and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications, and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target

The general fault in our fault lines

Pervading global narratives suggest that political polarization is increasing, yet the accuracy of such group meta-perceptions has been drawn into question. A recent US study suggests that these beliefs are inaccurate and drive polarized beliefs about out-groups. However, it also found that informing people of inaccuracies reduces those negative beliefs. In this work, we explore whether these results generalize to other countries. To achieve this, we replicate two of the original experiments with 10,207 participants across 26 countries. We focus on local group divisions, which we refer to as fault lines. We find broad generalizability for both inaccurate meta-perceptions and reduced negative motive attribution through a simple disclosure intervention. We conclude that inaccurate and negative group meta-perceptions are exhibited in myriad contexts and that informing individuals of their misperceptions can yield positive benefits for intergroup relations. Such generalizability highlights a robust phenomenon with implications for political discourse worldwide

Registered Replication Report on Fischer, Castel, Dodd, and Pratt (2003)

The attentional spatial-numerical association of response codes (Att-SNARC) effect (Fischer, Castel, Dodd, & Pratt, 2003)—the finding that participants are quicker to detect left-side targets when the targets are preceded by small numbers and quicker to detect right-side targets when they are preceded by large numbers—has been used as evidence for embodied number representations and to support strong claims about the link between number and space (e.g., a mental number line). We attempted to replicate Experiment 2 of Fischer et al. by collecting data from 1,105 participants at 17 labs. Across all 1,105 participants and four interstimulus-interval conditions, the proportion of times the effect we observed was positive (i.e., directionally consistent with the original effect) was 50. Further, the effects we observed both within and across labs were minuscule and incompatible with those observed by Fischer et al. Given this, we conclude that we failed to replicate the effect reported by Fischer et al. In addition, our analysis of several participant-level moderators (finger-counting habits, reading and writing direction, handedness, and mathematics fluency and mathematics anxiety) revealed no substantial moderating effects. Our results indicate that the Att-SNARC effect cannot be used as evidence to support strong claims about the link between number and space

A Novel Mutation in LEPRE1 That Eliminates Only the KDEL ER- Retrieval Sequence Causes Non-Lethal Osteogenesis Imperfecta

Prolyl 3-hydroxylase 1 (P3H1), encoded by the LEPRE1 gene, forms a molecular complex with cartilage-associated protein (CRTAP) and cyclophilin B (encoded by PPIB) in the endoplasmic reticulum (ER). This complex is responsible for one step in collagen post-translational modification, the prolyl 3-hydroxylation of specific proline residues, specifically α1(I) Pro986. P3H1 provides the enzymatic activity of the complex and has a Lys-Asp-Glu-Leu (KDEL) ER-retrieval sequence at the carboxyl terminus. Loss of function mutations in LEPRE1 lead to the Pro986 residue remaining unmodified and lead to slow folding and excessive helical post-translational modification of type I collagen, which is seen in both dominant and recessive osteogenesis imperfecta (OI). Here, we present the case of siblings with non-lethal OI due to novel compound heterozygous mutations in LEPRE1 (c.484delG and c.2155dupC). The results of RNA analysis and real-time PCR suggest that mRNA with c.2155dupC escapes from nonsense-mediated RNA decay. Without the KDEL ER- retrieval sequence, the product of the c.2155dupC variant cannot be retained in the ER. This is the first report of a mutation in LEPRE1 that eliminates only the KDEL ER-retrieval sequence, whereas other functional domains remain intact. Our study shows, for the first time, that the KDEL ER- retrieval sequence is essential for P3H1 functionality and that a defect in KDEL is sufficient for disease onset

Investigating variation in replicability

Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect

Insights on the Evolution of Prolyl 3-Hydroxylation Sites from Comparative Analysis of Chicken and Xenopus Fibrillar Collagens

Recessive mutations that prevent 3-hydroxyproline formation in type I collagen have been shown to cause forms of osteogenesis imperfecta. In mammals, all A-clade collagen chains with a GPP sequence at the A1 site (P986), except α1(III), have 3Hyp at residue P986. Available avian, amphibian and reptilian type III collagen sequences from the genomic database (Ensembl) all differ in sequence motif from mammals at the A1 site. This suggests a potential evolutionary distinction in prolyl 3-hydroxylation between mammals and earlier vertebrates. Using peptide mass spectrometry, we confirmed that this 3Hyp site is fully occupied in α1(III) from an amphibian, Xenopus laevis, as it is in chicken. A thorough characterization of all predicted 3Hyp sites in collagen types I, II, III and V from chicken and xenopus revealed further differences in the pattern of occupancy of the A3 site (P707). In mammals only α2(I) and α2(V) chains had any 3Hyp at the A3 site, whereas in chicken all α-chains except α1(III) had A3 at least partially 3-hydroxylated. The A3 site was also partially 3-hydroxylated in xenopus α1(I). Minor differences in covalent cross-linking between chicken, xenopus and mammal type I and III collagens were also found as a potential index of evolving functional differences. The function of 3Hyp is still unknown but observed differences in site occupancy during vertebrate evolution are likely to give important clues

Registered replication report on Fischer, Castel, Dodd, and Pratt (2003)

The attentional spatial-numerical association of response codes (Att-SNARC) effect (Fischer, Castel, Dodd, & Pratt, 2003)—the finding that participants are quicker to detect left-side targets when the targets are preceded by small numbers and quicker to detect right-side targets when they are preceded by large numbers—has been used as evidence for embodied number representations and to support strong claims about the link between number and space (e.g., a mental number line). We attempted to replicate Experiment 2 of Fischer et al. by collecting data from 1,105 participants at 17 labs. Across all 1,105 participants and four interstimulus-interval conditions, the proportion of times the effect we observed was positive (i.e., directionally consistent with the original effect) was .50. Further, the effects we observed both within and across labs were minuscule and incompatible with those observed by Fischer et al. Given this, we conclude that we failed to replicate the effect reported by Fischer et al. In addition, our analysis of several participant-level moderators (finger-counting habits, reading and writing direction, handedness, and mathematics fluency and mathematics anxiety) revealed no substantial moderating effects. Our results indicate that the Att-SNARC effect cannot be used as evidence to support strong claims about the link between number and space

Registered Replication Report on Fischer, Castel, Dodd, and Pratt (2003)

The attentional spatial-numerical association of response codes (Att-SNARC) effect (Fischer, Castel, Dodd, & Pratt, 2003)—the finding that participants are quicker to detect left-side targets when the targets are preceded by small numbers and quicker to detect right-side targets when they are preceded by large numbers—has been used as evidence for embodied number representations and to support strong claims about the link between number and space (e.g., a mental number line). We attempted to replicate Experiment 2 of Fischer et al. by collecting data from 1,105 participants at 17 labs. Across all 1,105 participants and four interstimulus-interval conditions, the proportion of times the effect we observed was positive (i.e., directionally consistent with the original effect) was .50. Further, the effects we observed both within and across labs were minuscule and incompatible with those observed by Fischer et al. Given this, we conclude that we failed to replicate the effect reported by Fischer et al. In addition, our analysis of several participant-level moderators (finger-counting habits, reading and writing direction, handedness, and mathematics fluency and mathematics anxiety) revealed no substantial moderating effects. Our results indicate that the Att-SNARC effect cannot be used as evidence to support strong claims about the link between number and space

The process of replication target selection in psychology: what to consider?

Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target. The resulting checklist can be used for transparently communicating the rationale for selecting studies for replication

A synthesis of evidence for policy from behavioural science during COVID-19

Scientific evidence regularly guides policy decisions1, with behavioural science increasingly part of this process2. In April 2020, an influential paper3 proposed 19 policy recommendations (‘claims’) detailing how evidence from behavioural science could contribute to efforts to reduce impacts and end the COVID-19 pandemic. Here we assess 747 pandemic-related research articles that empirically investigated those claims. We report the scale of evidence and whether evidence supports them to indicate applicability for policymaking. Two independent teams, involving 72 reviewers, found evidence for 18 of 19 claims, with both teams finding evidence supporting 16 (89%) of those 18 claims. The strongest evidence supported claims that anticipated culture, polarization and misinformation would be associated with policy effectiveness. Claims suggesting trusted leaders and positive social norms increased adherence to behavioural interventions also had strong empirical support, as did appealing to social consensus or bipartisan agreement. Targeted language in messaging yielded mixed effects and there were no effects for highlighting individual benefits or protecting others. No available evidence existed to assess any distinct differences in effects between using the terms ‘physical distancing’ and ‘social distancing’. Analysis of 463 papers containing data showed generally large samples; 418 involved human participants with a mean of 16,848 (median of 1,699). That statistical power underscored improved suitability of behavioural science research for informing policy decisions. Furthermore, by implementing a standardized approach to evidence selection and synthesis, we amplify broader implications for advancing scientific evidence in policy formulation and prioritization