The Relationship of Previous Training and Experience of Journal Peer Reviewers to Subsequent Review Quality

BACKGROUND: Peer review is considered crucial to the selection and publication of quality science, but very little is known about the previous experiences and training that might identify high-quality peer reviewers. The reviewer selection processes of most journals, and thus the qualifications of their reviewers, are ill defined. More objective selection of peer reviewers might improve the journal peer review process and thus the quality of published science. METHODS AND FINDINGS: 306 experienced reviewers (71% of all those associated with a specialty journal) completed a survey of past training and experiences postulated to improve peer review skills. Reviewers performed 2,856 reviews of 1,484 separate manuscripts during a four-year study period, all prospectively rated on a standardized quality scale by editors. Multivariable analysis revealed that most variables, including academic rank, formal training in critical appraisal or statistics, or status as principal investigator of a grant, failed to predict performance of higher-quality reviews. The only significant predictors of quality were working in a university-operated hospital versus other teaching environment and relative youth (under ten years of experience after finishing training). Being on an editorial board and doing formal grant (study section) review were each predictors for only one of our two comparisons. However, the predictive power of all variables was weak. CONCLUSIONS: Our study confirms that there are no easily identifiable types of formal training or experience that predict reviewer performance. Skill in scientific peer review may be as ill defined and hard to impart as is “common sense.” Without a better understanding of those skills, it seems unlikely journals and editors will be successful in systematically improving their selection of reviewers. This inability to predict performance makes it imperative that all but the smallest journals implement routine review ratings systems to routinely monitor the quality of their reviews (and thus the quality of the science they publish)

Ion Exchange Technology Development in Support of the Urine Processor Assembly Precipitation Prevention Project for the International Space Station

In support of the Urine Processor Assembly Precipitation Prevention Project (UPA PPP), multiple technologies were explored to prevent CaSO4 dot 2H2O (gypsum) precipitation during the on-orbit distillation process. Gypsum precipitation currently limits the water recovery rate onboard the International Space Station (ISS) to 70% versus the planned 85% target water recovery rate. Due to its advanced performance in removing calcium cations in pretreated augmented urine (PTAU), ion exchange was selected as one of the technologies for further development by the PPP team. A total of 12 ion exchange resins were evaluated in various equilibrium and dynamic column tests with solutions of dissolved gypsum, urine ersatz, PTAU, and PTAU brine at 85% water recovery. While initial evaluations indicated that the Purolite SST60 resin had the highest calcium capacity in PTAU (0.30 meq/mL average), later tests showed that the Dowex G26 and Amberlite FPC12H resins had the highest capacity (0.5 meq/mL average). Further dynamic column testing proved that G26 performance is +/- 10% of that value at flow rates of 0.45 and 0.79 Lph under continuous flow, and 10.45 Lph under pulsed flow. Testing at the Marshall Spaceflight Center (MSFC) integrates the ion exchange technology with a UPA ground article under flight-like pulsed flow conditions with PTAU. To date, no gypsum precipitation has taken place in any of the initial evaluations

More insight into the fate of biomedical meeting abstracts: a systematic review

BACKGROUND: It has been estimated that about 45% of abstracts that are accepted for presentation at biomedical meetings will subsequently be published in full. The acceptance of abstracts at meetings and their fate after initial rejection are less well understood. We set out to estimate the proportion of abstracts submitted to meetings that are eventually published as full reports, and to explore factors that are associated with meeting acceptance and successful publication. METHODS: Studies analysing acceptance of abstracts at biomedical meetings or their subsequent full publication were searched in MEDLINE, OLDMEDLINE, EMBASE, Cochrane Library, CINAHL, BIOSIS, Science Citation Index Expanded, and by hand searching of bibliographies and proceedings. We estimated rates of abstract acceptance and of subsequent full publication, and identified abstract and meeting characteristics associated with acceptance and publication, using logistic regression analysis, survival-type analysis, and meta-analysis. RESULTS: Analysed meetings were held between 1957 and 1999. Of 14945 abstracts that were submitted to 43 meetings, 46% were accepted. The rate of full publication was studied with 19123 abstracts that were presented at 234 meetings. Using survival-type analysis, we estimated that 27% were published after two, 41% after four, and 44% after six years. Of 2412 abstracts that were rejected at 24 meetings, 27% were published despite rejection. Factors associated with both abstract acceptance and subsequent publication were basic science and positive study outcome. Large meetings and those held outside the US were more likely to accept abstracts. Abstracts were more likely to be published subsequently if presented either orally, at small meetings, or at a US meeting. Abstract acceptance itself was strongly associated with full publication. CONCLUSIONS: About one third of abstracts submitted to biomedical meetings were published as full reports. Acceptance at meetings and publication were associated with specific characteristics of abstracts and meetings

THE AFFORDABLE CARE ACT AND AMBULANCE RESPONSE TIMES

This study contributes to the literature on supply-side adjustments to insurance expansions by examining the effect of the Affordable Care Act (ACA) on ambulance response times. Exploiting temporal and geographic variation in the implementation of the ACA as well as pre-treatment differences in uninsured rates, we estimate that the expansions of private and Medicaid coverage under the ACA combined to slow ambulance response times by an average of 19%. We conclude that, through extending coverage to individuals who, in its absence, would not have availed themselves of emergency medical services, the ACA added strain to emergency response systems

A call for transparent reporting to optimize the predictive value of preclinical research

The US National Institute of Neurological Disorders and Stroke convened major stakeholders in June 2012 to discuss how to improve the methodological reporting of animal studies in grant applications and publications. The main workshop recommendation is that at a minimum studies should report on sample-size estimation, whether and how animals were randomized, whether investigators were blind to the treatment, and the handling of data. We recognize that achieving a meaningful improvement in the quality of reporting will require a concerted effort by investigators, reviewers, funding agencies and journal editors. Requiring better reporting of animal studies will raise awareness of the importance of rigorous study design to accelerate scientific progress

Anesthesia advanced circulatory life support

The constellation of advanced cardiac life support (ACLS) events, such as gas embolism, local anesthetic overdose, and spinal bradycardia, in the perioperative setting differs from events in the pre-hospital arena. As a result, modification of traditional ACLS protocols allows for more specific etiology-based resuscitation. Perioperative arrests are both uncommon and heterogeneous and have not been described or studied to the same extent as cardiac arrest in the community. These crises are usually witnessed, frequently anticipated, and involve a rescuer physician with knowledge of the patient's comorbidities and coexisting anesthetic or surgically related pathophysiology. When the health care provider identifies the probable cause of arrest, the practitioner has the ability to initiate medical management rapidly. Recommendations for management must be predicated on expert opinion and physiological understanding rather than on the standards currently being used in the generation of ACLS protocols in the community. Adapting ACLS algorithms and considering the differential diagnoses of these perioperative events may prevent cardiac arrest

A normative model of peer review: qualitative assessment of manuscript reviewers’ attitudes towards peer review

Objective: Peer review is considered crucial to the selection and publication of quality research, yet little is known of the values, beliefs and attitudes of peer reviewers towards the process of peer review. This study elicits reviewer beliefs about the process in order to produce a normative model of peer review. Methods and Findings: The 72 subjects were experienced reviewers at Annals of Emergency Medicine and had completed at least 5 reviews in the past 2 years. Subjects participated in 40 minute structured telephone interviews focusing on reviewer attitudes, beliefs and values towards the process of peer review. Subject responses were coded and categorized using grounded theory to produce a qualitative profile of reviewers' attitudes towards peer review and generate a normative model of the peer review process. This model was found to closely adhere to conventionally held beliefs 2 about the process of peer review. However, within it were revealed a number of areas where reviewers, aware of tensions within the process, questioned those conventional beliefs, expressing concern about methods, operations and outcomes. As researchers producing research and receiving reviews and as reviewers judging others’ research and producing reviews, the reviewer’s status as “peer” was seen as both essential to the operation of the system and problematic. In their perception of the role of the peer reviewer, though respondents identified evaluation of the manuscript (selecting submissions for publication by filtering out incorrect or inadequate work) as the primary goal of the formal process, instruction of the researcher (improving the accuracy, clarity and utility of published research) was considered by the majority of reviewers as the more important practice. Likewise, though there was recognition that the review process aims to prevent poor research from being published, there was more concern over the danger that it results in good research being “strangled in its cradle”. Though respondents believed that the quality of the review is determined primarily by the skills of the individual reviewer, they maintain that the validity of the process is determined by the corporate nature of the review panel acting as a system of checks and balances. Though perceiving this system of checks and balances as requiring a degree of separation of authorial, review and editorial functions, reviewers, at the same time, express the desire for a more open system of feedback leading towards a more consensual research outcome. Two issues of concern arose repeatedly in the interviews: frustration at a perceived lack of feedback from editors to reviewers and repeated condemnations of “mean-spirited” feedback from reviewers to authors. Defects in feedback were cited by respondents as a major barrier to optimizing research quality and editorial judgment. Conclusions: The tensions found in the peer review process, sometimes seen as barriers to its effective operation, are less defects in the process than definitive of the concept of peer review itself and thus necessary to its operation. While at a practical level peer review operates as a triage exercise, it is, at the same time, on a social level, a mode of disciplinary dialogue between peers: important not only to the maintenance of an effective knowledge base and thus disciplinary validity, but also, through its effect on researchers and reviewers, important in the construction of disciplinary identity. Peer review's practical and social operations are not antithetical to each other but rather are inherent in the hybrid concept of the peer reviewer, where one's status as a peer makes possible one's activity as reviewer

Peer Review: Consensus and Contradiction, a Qualitative Approach

Manuscript peer review is considered crucial to the selection and publication of quality scientific research, however, the practice is being increasingly challenged as a non-standardized process of unknown scientific validity with substantial weaknesses. Scientific disciplines appear to be confronted by a process of limited efficacy, resistant to rational maneuvers for its improvement, which yet continues to receive strong support from its practitioners. When a practice’s efficacy in achieving its goals is questionable and yet the practice persists, questions of its social functions arise which can only be addressed by qualitative research. This paper describes a normative model of peer review based on a qualitative profile of the attitudes of 72 peer reviewers towards the practice of manuscript peer review (obtained from extensive structured interviews). Masked by consensus amongst respondents about methods and goals were concerns centering on a series of contradictions inherent in the process. While at a practical level peer review was seen by respondents as a triage exercise, it was, at the same time, on a social level, valued as a mode of disciplinary discourse, important not only in the production of disciplinary knowledge, but also in the construction of the disciplinary identities of those who labor to bring that knowledge into being, i.e., the peers