Test validity defined as d-connection between target and measured attribute:Expanding the causal definition of Borsboom et al. (2004)

This article suggests a modification to the conception of test validity put forward by Borsboom, Mellenberghand van Heerden (2004). According to the original definition, a test is only valid if test outcomes are causedby variation in the target attribute. According to the d-connection definition of test validity, a test is validfor measuring an attribute if (a) the attribute exists, and (b) variation in the attribute is d-connected tovariation in the measurement outcomes. In other words, a test is valid whenever test outcomes inform useither about whathashappened to the target attribute in the past, or about whatwillhappen to the targetattribute in the future. Thus, the d-connection definition expands the number of scenarios in which a test canbe considered valid. Defining test validity as d-connection between target and measured attribute situatesthe validity concept squarely within the structural causal modeling framework of Pearl (2009)

Test validity defined as d-connection between target and measured attribute:Expanding the causal definition of Borsboom et al. (2004)

This article suggests a modification to the conception of test validity put forward by Borsboom, Mellenberghand van Heerden (2004). According to the original definition, a test is only valid if test outcomes are causedby variation in the target attribute. According to the d-connection definition of test validity, a test is validfor measuring an attribute if (a) the attribute exists, and (b) variation in the attribute is d-connected tovariation in the measurement outcomes. In other words, a test is valid whenever test outcomes inform useither about whathashappened to the target attribute in the past, or about whatwillhappen to the targetattribute in the future. Thus, the d-connection definition expands the number of scenarios in which a test canbe considered valid. Defining test validity as d-connection between target and measured attribute situatesthe validity concept squarely within the structural causal modeling framework of Pearl (2009)

Deciding what to replicate:A decision model for replication study selection under resource and knowledge constraints

Robust scientific knowledge is contingent upon replication of original findings. However,researchers who conduct replication studies face a dicult problem; there are many morestudies in need of replication than there are funds available for replicating. To select studies forreplication eciently, we need to understand which studies arethe mostin need of replication.In other words, we need to understand which replication eorts have the highest expected utility.In this article we propose a general rule for study selection in replication research based onthereplication valueof the claims considered for replication. Thereplication valueof a claimis defined as the maximum expected utility we could gain by conducting a replication of theclaim, and is a function of (1) the value of being certain about the claim, and (2) uncertaintyabout the claim based on current evidence. We formalize this definition in terms of a causaldecision model, utilizing concepts from decision theory and causal graph modeling. We discussthe validity of usingreplication valueas a measure of expected utility gain, and we suggestapproaches for deriving quantitative estimates ofreplication value

Deciding what to replicate: a formal definition of “replication value” and a decision model for replication study selection

Multivariate analysis of psychological dat

The Process of Replication Target Selection in Psychology: What to Consider?

Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well justified, systematic, and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications, and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target

The process of replication target selection in psychology: what to consider?

Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target. The resulting checklist can be used for transparently communicating the rationale for selecting studies for replication