Calibration with confidence:A principled method for panel assessment

Frequently, a set of objects has to be evaluated by a panel of assessors, but not every object is assessed by every assessor. A problem facing such panels is how to take into account different standards amongst panel members and varying levels of confidence in their scores. Here, a mathematically-based algorithm is developed to calibrate the scores of such assessors, addressing both of these issues. The algorithm is based on the connectivity of the graph of assessors and objects evaluated, incorporating declared confidences as weights on its edges. If the graph is sufficiently well connected, relative standards can be inferred by comparing how assessors rate objects they assess in common, weighted by the levels of confidence of each assessment. By removing these biases, "true" values are inferred for all the objects. Reliability estimates for the resulting values are obtained. The algorithm is tested in two case studies, one by computer simulation and another based on realistic evaluation data. The process is compared to the simple averaging procedure in widespread use, and to Fisher's additive incomplete block analysis. It is anticipated that the algorithm will prove useful in a wide variety of situations such as evaluation of the quality of research submitted to national assessment exercises; appraisal of grant proposals submitted to funding panels; ranking of job applicants; and judgement of performances on degree courses wherein candidates can choose from lists of options.Comment: 32 pages including supplementary information; 5 figure

THE COSINE-CUBED NEUTRON SPECTROMETER (thesis)

In an attempt to increase the efficiency, a proton recoil neutron spectrometer was built in which the proton detection crystal is in the shape of the surface formed by the rotation of the cos/sup 3/ THETA curve about the THETA = 0 axis where THETA is the angle between the direction of travel of the incident neutron and the direction of travel of the recoil proton. Such a construction makes the detector crystal conform to the range envelope of the protons recoiling from a neutron beam of sufficiently high energy into a gas in which the proton range is proportional to the 3/2 power of the proton energy. Sixteen pieces of thallium-activated cesium iodide were assembled as a mosaic arranged in the cos/sup 3/ THETA shape so that the maximum THETA was 30 deg . Calculations show that 25% of the protons recoil within 30 deg . The Po-Be snd Po-B spectra were measured with the instrument using xenon in the chamber. The resolution of the spectra obtained was low due to nonuniform pulse height response of the CsI crystal sections and excessive gamma-ray response even though sn anthracene proton radiator was used in a coincidence scheme designed to minimize gamma-ray background. The instrument did not exhibit the added efficiency which was being sought in the design. The necessity of single chamel operation further limits its usefulness. (auth

Beliefs and expectations about the future of personal life and future of the world in a sample of Iranian and Canadian students

This study was aimed to assess expectations about future of personal life and future of the world among a sample of Iranian and Canadian students. 60 Iranians from Tehran and Shahid Beheshti Universities in Iran and and 62 Canadians from Carleton University in Canada completed a researcher made questionnaire asking their beliefs and expectations in four areas such as personal future, generational comparison, future of the world and personal life goals. The results showed that both Iranian and Canadian participants were more optimistic about their personal future than about the future of the world but Iranians were less optimistic about the future of the world than were Canadians. Both groups expect to be richer and have a higher quality of life than their parents, and both expect their children to have a lower quality of life than their parents and to be poorer than themselves. Cultural differences were found in the importance Iranians and Canadians attach to life goals. Results and implications are discussed

Do Author-Suggested Reviewers Rate Submissions More Favorably than Editor-Suggested Reviewers? A Study on Atmospheric Chemistry and Physics

Background: Ratings in journal peer review can be affected by sources of bias. The bias variable investigated here was the information on whether authors had suggested a possible reviewer for their manuscript, and whether the editor had taken up that suggestion or had chosen a reviewer that had not been suggested by the authors. Studies have shown that authorsuggested reviewers rate manuscripts more favorably than editor-suggested reviewers do. Methodology/Principal Findings: Reviewers ’ ratings on three evaluation criteria and the reviewers ’ final publication recommendations were available for 552 manuscripts (in total 1145 reviews) that were submitted to Atmospheric Chemistry and Physics, an interactive open access journal using public peer review (authors ’ and reviewers ’ comments are publicly exchanged). Public peer review is supposed to bring a new openness to the reviewing process that will enhance its objectivity. In the statistical analysis the quality of a manuscript was controlled for to prevent favorable reviewers ’ ratings from being attributable to quality instead of to the bias variable. Conclusions/Significance: Our results agree with those from other studies that editor-suggested reviewers rated manuscripts between 30 % and 42 % less favorably than author-suggested reviewers. Against this backdrop journal editors should consider either doing without the use of author-suggested reviewers or, if they are used, bringing in more than one editor-suggested reviewer for the review process (so that the review by author-suggested reviewers can be put i

Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes

Does peer review fulfill its declared objective of identifying the best science and the best scientists? In order to answer this question we analyzed the Long-Term Fellowship and the Young Investigator programmes of the European Molecular Biology Organization. Both programmes aim to identify and support the best post doctoral fellows and young group leaders in the life sciences. We checked the association between the selection decisions and the scientific performance of the applicants. Our study involved publication and citation data for 668 applicants to the Long-Term Fellowship programme from the year 1998 (130 approved, 538 rejected) and 297 applicants to the Young Investigator programme (39 approved and 258 rejected applicants) from the years 2001 and 2002. If quantity and impact of research publications are used as a criterion for scientific achievement, the results of (zero-truncated) negative binomial models show that the peer review process indeed selects scientists who perform on a higher level than the rejected ones subsequent to application. We determined the extent of errors due to over-estimation (type I errors) and under-estimation (type 2 errors) of future scientific performance. Our statistical analyses point out that between 26% and 48% of the decisions made to award or reject an application show one of both error types. Even though for a part of the applicants, the selection committee did not correctly estimate the applicant's future performance, the results show a statistically significant association between selection decisions and the applicants' scientific achievements, if quantity and impact of research publications are used as a criterion for scientific achievement

Divided Loyalists or Conditional Cooperators? Creating Consensus about Cooperation in Multiple Simultaneous Social Dilemmas

The current social dilemma literature lacks theoretical consensus regarding how individuals behave when facing multiple simultaneous social dilemmas. The divided-loyalty hypothesis, from organizational theory, predicts that cooperation will decline as individuals experience multiple social dilemmas with different compared to the same group members. The conditional-cooperation hypothesis, from behavioral economics, predicts that cooperation will increase as individuals experience multiple social dilemmas with different compared to the same group members. We employ a laboratory experiment to create consensus between these literatures and find support for the conditional-cooperation hypothesis. The positive effect of interacting with different group members comes from participants having an opportunity to shift their cooperative behavior from the less cooperative to the more cooperative group