The Process of Replication Target Selection in Psychology: What to Consider?

Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well justified, systematic, and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications, and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target

CRTAP Is Required for Prolyl 3- Hydroxylation and Mutations Cause Recessive Osteogenesis Imperfecta

n/

Deciding what to replicate: a decision model for replication study selection under resource and knowledge constraints

Robust scientific knowledge is contingent upon replication of original findings. However, replicating researchers are constrained by resources, and will almost always have to choose one replication effort to focus on from a set of potential candidates. To select a candidate efficiently in these cases, we need methods for deciding which out of all candidates considered would be the most useful to replicate, given some overall goal researchers wish to achieve. In this article we assume that the overall goal researchers wish to achieve is to maximize the utility gained by conducting the replication study. We then propose a general rule for study selection in replication research based on the replication value of the set of claims considered for replication. The replication value of a claim is defined as the maximum expected utility we could gain by conducting a replication of the claim, and is a function of (a) the value of being certain about the claim, and (b) uncertainty about the claim based on current evidence. We formalize this definition in terms of a causal decision model, utilizing concepts from decision theory and causal graph modeling. We discuss the validity of using replication value as a measure of expected utility gain, and we suggest approaches for deriving quantitative estimates of replication value. Our goal in this article is not to define concrete guidelines for study selection, but to provide the necessary theoretical foundations on which such concrete guidelines could be built.Translational Abstract Replication-redoing a study using the same procedures-is an important part of checking the robustness of claims in the psychological literature. The practice of replicating original studies has been woefully devalued for many years, but this is now changing. Recent calls for improving the quality of research in psychology has generated a surge of interest in funding, conducting, and publishing replication studies. Because many studies have never been replicated, and researchers have limited time and money to perform replication studies, researchers must decide which studies are the most important to replicate. This way scientists learn the most, given limited resources. In this article, we lay out what it means to think about what is the most important thing to replicate, and we propose a general decision rule for picking a study to replicate. That rule depends on a concept we call replication value. Replication value is a function of the importance of the study, and how uncertain we are about the findings. In this article we explain how researchers can think precisely about the value of replication studies. We then discuss when and how it makes sense to use replication value as a measure of how valuable a replication study would be, and we discuss factors that funders, journals, or scientists could consider when determining how valuable a replication study is.Multivariate analysis of psychological dat

The general fault in our fault lines

Pervading global narratives suggest that political polarization is increasing, yet the accuracy of such group meta-perceptions has been drawn into question. A recent US study suggests that these beliefs are inaccurate and drive polarized beliefs about out-groups. However, it also found that informing people of inaccuracies reduces those negative beliefs. In this work, we explore whether these results generalize to other countries. To achieve this, we replicate two of the original experiments with 10,207 participants across 26 countries. We focus on local group divisions, which we refer to as fault lines. We find broad generalizability for both inaccurate meta-perceptions and reduced negative motive attribution through a simple disclosure intervention. We conclude that inaccurate and negative group meta-perceptions are exhibited in myriad contexts and that informing individuals of their misperceptions can yield positive benefits for intergroup relations. Such generalizability highlights a robust phenomenon with implications for political discourse worldwide

Insights on the Evolution of Prolyl 3-Hydroxylation Sites from Comparative Analysis of Chicken and Xenopus Fibrillar Collagens

Recessive mutations that prevent 3-hydroxyproline formation in type I collagen have been shown to cause forms of osteogenesis imperfecta. In mammals, all A-clade collagen chains with a GPP sequence at the A1 site (P986), except α1(III), have 3Hyp at residue P986. Available avian, amphibian and reptilian type III collagen sequences from the genomic database (Ensembl) all differ in sequence motif from mammals at the A1 site. This suggests a potential evolutionary distinction in prolyl 3-hydroxylation between mammals and earlier vertebrates. Using peptide mass spectrometry, we confirmed that this 3Hyp site is fully occupied in α1(III) from an amphibian, Xenopus laevis, as it is in chicken. A thorough characterization of all predicted 3Hyp sites in collagen types I, II, III and V from chicken and xenopus revealed further differences in the pattern of occupancy of the A3 site (P707). In mammals only α2(I) and α2(V) chains had any 3Hyp at the A3 site, whereas in chicken all α-chains except α1(III) had A3 at least partially 3-hydroxylated. The A3 site was also partially 3-hydroxylated in xenopus α1(I). Minor differences in covalent cross-linking between chicken, xenopus and mammal type I and III collagens were also found as a potential index of evolving functional differences. The function of 3Hyp is still unknown but observed differences in site occupancy during vertebrate evolution are likely to give important clues

Registered Replication Report on Fischer, Castel, Dodd, and Pratt (2003)

The attentional spatial-numerical association of response codes (Att-SNARC) effect (Fischer, Castel, Dodd, & Pratt, 2003)—the finding that participants are quicker to detect left-side targets when the targets are preceded by small numbers and quicker to detect right-side targets when they are preceded by large numbers—has been used as evidence for embodied number representations and to support strong claims about the link between number and space (e.g., a mental number line). We attempted to replicate Experiment 2 of Fischer et al. by collecting data from 1,105 participants at 17 labs. Across all 1,105 participants and four interstimulus-interval conditions, the proportion of times the effect we observed was positive (i.e., directionally consistent with the original effect) was 50. Further, the effects we observed both within and across labs were minuscule and incompatible with those observed by Fischer et al. Given this, we conclude that we failed to replicate the effect reported by Fischer et al. In addition, our analysis of several participant-level moderators (finger-counting habits, reading and writing direction, handedness, and mathematics fluency and mathematics anxiety) revealed no substantial moderating effects. Our results indicate that the Att-SNARC effect cannot be used as evidence to support strong claims about the link between number and space

A Novel Mutation in LEPRE1 That Eliminates Only the KDEL ER- Retrieval Sequence Causes Non-Lethal Osteogenesis Imperfecta

Prolyl 3-hydroxylase 1 (P3H1), encoded by the LEPRE1 gene, forms a molecular complex with cartilage-associated protein (CRTAP) and cyclophilin B (encoded by PPIB) in the endoplasmic reticulum (ER). This complex is responsible for one step in collagen post-translational modification, the prolyl 3-hydroxylation of specific proline residues, specifically α1(I) Pro986. P3H1 provides the enzymatic activity of the complex and has a Lys-Asp-Glu-Leu (KDEL) ER-retrieval sequence at the carboxyl terminus. Loss of function mutations in LEPRE1 lead to the Pro986 residue remaining unmodified and lead to slow folding and excessive helical post-translational modification of type I collagen, which is seen in both dominant and recessive osteogenesis imperfecta (OI). Here, we present the case of siblings with non-lethal OI due to novel compound heterozygous mutations in LEPRE1 (c.484delG and c.2155dupC). The results of RNA analysis and real-time PCR suggest that mRNA with c.2155dupC escapes from nonsense-mediated RNA decay. Without the KDEL ER- retrieval sequence, the product of the c.2155dupC variant cannot be retained in the ER. This is the first report of a mutation in LEPRE1 that eliminates only the KDEL ER-retrieval sequence, whereas other functional domains remain intact. Our study shows, for the first time, that the KDEL ER- retrieval sequence is essential for P3H1 functionality and that a defect in KDEL is sufficient for disease onset

Registered replication report on Fischer, Castel, Dodd, and Pratt (2003)

The attentional spatial-numerical association of response codes (Att-SNARC) effect (Fischer, Castel, Dodd, & Pratt, 2003)—the finding that participants are quicker to detect left-side targets when the targets are preceded by small numbers and quicker to detect right-side targets when they are preceded by large numbers—has been used as evidence for embodied number representations and to support strong claims about the link between number and space (e.g., a mental number line). We attempted to replicate Experiment 2 of Fischer et al. by collecting data from 1,105 participants at 17 labs. Across all 1,105 participants and four interstimulus-interval conditions, the proportion of times the effect we observed was positive (i.e., directionally consistent with the original effect) was .50. Further, the effects we observed both within and across labs were minuscule and incompatible with those observed by Fischer et al. Given this, we conclude that we failed to replicate the effect reported by Fischer et al. In addition, our analysis of several participant-level moderators (finger-counting habits, reading and writing direction, handedness, and mathematics fluency and mathematics anxiety) revealed no substantial moderating effects. Our results indicate that the Att-SNARC effect cannot be used as evidence to support strong claims about the link between number and space

Registered Replication Report on Fischer, Castel, Dodd, and Pratt (2003)

The attentional spatial-numerical association of response codes (Att-SNARC) effect (Fischer, Castel, Dodd, & Pratt, 2003)—the finding that participants are quicker to detect left-side targets when the targets are preceded by small numbers and quicker to detect right-side targets when they are preceded by large numbers—has been used as evidence for embodied number representations and to support strong claims about the link between number and space (e.g., a mental number line). We attempted to replicate Experiment 2 of Fischer et al. by collecting data from 1,105 participants at 17 labs. Across all 1,105 participants and four interstimulus-interval conditions, the proportion of times the effect we observed was positive (i.e., directionally consistent with the original effect) was .50. Further, the effects we observed both within and across labs were minuscule and incompatible with those observed by Fischer et al. Given this, we conclude that we failed to replicate the effect reported by Fischer et al. In addition, our analysis of several participant-level moderators (finger-counting habits, reading and writing direction, handedness, and mathematics fluency and mathematics anxiety) revealed no substantial moderating effects. Our results indicate that the Att-SNARC effect cannot be used as evidence to support strong claims about the link between number and space

Investigating variation in replicability

Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect