Comments on the original article by Matthew Makel (see record 2014-06823-002) which raises an important concern regarding selfcorrection in science. While the notion that science should be self-correcting, and is failing, is not new, recent events in both social and biological sciences have led to a renewed call to develop approaches that would allow for self-correction ( Economist, 2013). Makel addresses a number of issues that hinder self-correction through replications and suggests a number of solutions. In this commentary, we as the editors of Psychology of Aesthetics, Creativity, and the Arts (PACA), would like to offer our perspective. As editors, we are in a unique position to influence the field and the ability to self-correct through published replications. Replications, however, are not a one-size fits all

Reiter-Palmon, Roni

Tinio, Pablo P. L.

English

The University of Nebraska, Omaha

University of Nebraska at Omaha DigitalCommons@UNO Psychology Faculty Publications Department of Psychology 2-1-2014 Pipeline Revision: A Call to Change Roni Reiter-Palmon Pablo P. L. Tinio Follow this and additional works at: https://digitalcommons.unomaha.edu/psychfacpub  Part of the Psychology Commons COMMENT Pipeline Revisions: A Call to Change Roni Reiter-Palmon University of Nebraska–Omaha    Pablo P. L. Tinio Queens College of the City University of New York  Makel (this issue) raises an important concern regarding self-correction in science. While the notion that science should be self-correcting, and is failing, is not new, recent events in both social and biological sciences have led to a renewed call to develop approaches that would allow for self-correction (Economist, 2013). Makel addresses a number of issues that hinder self-correction through replications and suggests a number of solutions. In this commentary, we as the editors of Psychology of Aesthetics, Creativity, and the Arts (PACA), would like to offer our perspective. One of the important impediments cited in relation to replications is the inability to publish such replications. As Makel noted, editors and reviewers have much influence over this issue. As editors, we are in a unique position to influence the field and the ability to self-correct through published replications.  Replications, however, are not a one-size fits all. Even when we discuss replications, there is a variety of ways in which replications can take place. First, of course, are the direct replications—these studies try to mirror exactly the original study and replicate the findings from that study. Conceptual replications intend to replicate the findings to a different sample, or perhaps using a different instrument, allowing for greater generalization. However, if such a replication fails (i.e., the findings of the replication do not mirror the original study), it is not clear whether the original findings should be questioned or whether the findings just cannot be generalized. The final type of replication is a “replication and extension.” In this type of study, previous findings are replicated, either exactly using the same procedure and methodology or “in spirit”; however, the study also includes new variables, extending the findings or adding something new. Of course, in a replication and extension, the quality of the extension is ultimately tied to the quality of the replication. While all three types of studies may suffer from the pitfalls that typically exist in reviewing for publication, the first, the direct replication, is the most vulnerable to rejection by reviewers and editors alike. The reasons cited for such a rejection typically involve the lack of novelty, uniqueness, or providing a new contribution to the filed.  These types of assessments are commonly asked in the review process. We, the editors of PACA, ask reviewers to evaluate the “significance of the article.” While the question itself does not directly suggest that only new information should be rated highly, the common understanding is that if an article does not provide new information, it might not provide a significant contribution. If you think about manuscripts you wrote and were rejected, or manuscripts that you have reviewed, how often were these rejected because they did not offer anything new, or what they offered in the case of conceptual replications, or replications and extensions, was not “new enough”?  The reasons for focusing on the new findings, those that we perceive move the field forward, are important to understand, as we cannot offer solutions without first understanding the root cause. From the perspective of the editors, our job is to ensure that the articles we publish are of high quality, but in addition to that, we must also focus on indicators that have come to signal high quality. Impact factor values have become the standard for quality. In many cases, departments are encouraging faculty members to focus on publishing in journals with high impact factors, by including these values in tenure and promotion decisions. As editors, we are therefore excited when the impact factor of our journal goes up, or worry when it goes down. With that comes another issue: what is more likely to be cited (and therefore increase the impact factor of the journal); the new finding or the replication? Therefore, it is not surprising that editors and reviewers favor the novel finding over the replication.  However, if we, as a scientific field, decide that it is important to publish replications, we should provide the right context in which to do so. As we have started to discuss the issue of publishing replications during the APA conference and in preparation for this special section, we realized that using the guidelines that we currently use are not particularly useful for the evaluation of replications. The current focus on significance, importance, or in other journals, on contribution to the filed, moving the field forward, or novelty of the findings, is not a good or useful metric for the evaluation of replications. As a result, we need to modify the guidance we give to reviewers.  Discussions with colleagues who review (not only for PACA) suggest that the issue of whether a study adds anything new is an important aspect in their decision to recommend acceptance or rejection. Even if the specific questions do not include this issue, how novel or unique the contribution is will come up in the review letter. If we are to publish replications, we must design reviewer guidelines that fit the task and that would allow us to accept the manuscripts for publication. If we want to practice what we preach, we need to create a situation that is favorable to the publication of replications, and one important aspect is the guidance we provide to reviewers. Specifically, for the purpose of evaluating replications, we must move away from novelty, new contribution, and significance. Not only that, we must provide guidelines that ensure that these types of considerations are explicitly excluded. In fact, the whole point of a replication study is to repeat what has been done before! A more appropriate metric may be how closely the replication study actually replicates the original study design—including sample, procedures, materials, and analyses. In addition, we must evaluate what is considered a successful or close enough replication. If there are minor variations in the methodology or design, is it now different or will still be considered a direct replication? Methodological rigor may need to be sacrificed in favor of exact replication. While we typically do not publish null effects, in the case of replications, null effects, which contradict the original findings, are of interest. Finally, replications should be identified as such, so reviewers are aware that they need to apply different guidelines and those guidelines should be provided and explicitly stated.  It is even likely that the different kinds of replication (direct, conceptual, or with extension) will require somewhat different guidelines. For example, when evaluating a conceptual replication, we would want to evaluate how different is the new sample or population. Therefore, it is important to determine what defines a sufficient difference to indicate that this is a conceptual replication and not a direct replication.  We are aware that these changes are just initial steps toward promoting self-correction through published replications, small first steps that challenge ideas and practices that are entrenched in the culture of scientific psychology. As editors of PACA, we are also aware of the opportunity that this new focus affords us to engage the field in a lively dialogue; replication, after all, involves a point-counterpoint exchange between people that could lead to a resolution or further debate. With this special section, we hope to start this discussion and be able to provide guidance about our expectations—as editors and as a field—to reviewers and to those who will attempt to conduct and publish those replications. We believe that this guidance is of critical importance, and we are working to develop more specific guidelines that will encourage the publication of replications in PACA. One of our goals for the next year is to communicate these guidelines to PACA reviewers and to put them into practice during the various stages of the peer review process. While not yet complete, at the core, these guidelines would focus on how close the replication is to the original article in terms of sample, procedure, measures, and so on, and whether any differences are meaningful in making the replication more difficult to generalize. We are also in the process of determining specific topics and issues that are in need of, and amenable to, the kinds of replication that Makel describes. With that in mind, we plan to poll the readership of PACA as well as Division 10 membership on what they view as the most important findings that should be replicated.1 Finally—and what an opportune moment to make this announcement— because of the significant increase in the number of submissions to PACA during the past years, we will increase our page number from 400 to 500. This gives us the additional space needed to accommodate the publication of replications. We are committed to making this happen. Stay tuned!  1 We thank an anonymous reviewer for this suggestion.  Reference  Economist. (Oct 19, 2013). Trouble at the lab. Retrieved from http://www .economist.com/news/briefing/21588057-scientists-think-science-selfcorrecting-alarming-degree-it-not-trouble 

Pipeline Revision: A Call to Change

https://digitalcommons.unomaha.edu/cgi/viewcontent.cgi?article=1246&context=psychfacpub

Pipeline Revision: A Call to Change

Abstract

Similar works

Full text

Available Versions

The University of Nebraska, Omaha