Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes

Does peer review fulfill its declared objective of identifying the best science and the best scientists? In order to answer this question we analyzed the Long-Term Fellowship and the Young Investigator programmes of the European Molecular Biology Organization. Both programmes aim to identify and support the best post doctoral fellows and young group leaders in the life sciences. We checked the association between the selection decisions and the scientific performance of the applicants. Our study involved publication and citation data for 668 applicants to the Long-Term Fellowship programme from the year 1998 (130 approved, 538 rejected) and 297 applicants to the Young Investigator programme (39 approved and 258 rejected applicants) from the years 2001 and 2002. If quantity and impact of research publications are used as a criterion for scientific achievement, the results of (zero-truncated) negative binomial models show that the peer review process indeed selects scientists who perform on a higher level than the rejected ones subsequent to application. We determined the extent of errors due to over-estimation (type I errors) and under-estimation (type 2 errors) of future scientific performance. Our statistical analyses point out that between 26% and 48% of the decisions made to award or reject an application show one of both error types. Even though for a part of the applicants, the selection committee did not correctly estimate the applicant's future performance, the results show a statistically significant association between selection decisions and the applicants' scientific achievements, if quantity and impact of research publications are used as a criterion for scientific achievement

The law is not enough

The publication of the Helsinki report on the situation of women in European academia shows that more than legal hurdles have to be overcome on the way to equal representatio

Data structure of this study.

<p>Data structure of this study.</p

Box plots for <i>h</i> index values of approved and rejected applicants for the LTF and YI programme.

<p>Box plots for <i>h</i> index values of approved and rejected applicants for the LTF and YI programme.</p

Box plots for the number of papers published subsequent to application (first row).

<p>Median numbers of citations for papers published prior to application (second row) and median numbers of citations for papers published subsequent to application (third row) (approved and rejected applicants for the LTF and YI programme). <i>Note</i>. Applications from 1998 (LTF programme) and 2001/2002 (YI programme); publication windows: from 1993 to the beginning of 2006 (LTF programme), from 1984 to the beginning of 2007 (YI programme); citation window: from year of publication to the beginning of 2006 and 2007, respectively. Since the downloading of citation counts was done in 2006 and 2007, respectively, one cannot expect high median citation counts yet for the most recent publications (see the graphs in the third row of the figure).</p

Aptitude or attitude?

Lawrence Summers' recent remarks reflect what little progress has been made in the public's understanding of why women are under-represented in scienc

(Zero-truncated) negative binomial regression models predicting (1) number of papers published subsequent to application, (2) citations for papers published prior to application and (3) citations for papers published subsequent to application (applicants for the YI programme).

<p><i>Note.</i> ML-point estimates (the results of the <i>z</i>-test in parentheses).</p><p>^*<i>p</i><0.05, ^**<i>p</i><0.01, ^***<i>p</i><0.001.</p>1<p>truncated sample.</p><p>Interpretation example for the parameter estimates in the table: In model 2 the number of pages of a publication has a statistically significant effect on receiving citations with a parameter estimate of 0.031. This means that for an additional page, the odds of receiving citations increase by a factor of 1.03 ( = exp(0.031)), holding all other variables in model 2 constant.</p

Description of the factors that were potentially associated with quantity and impact of research publications (applicants for the YI programme).

<p>Description of the factors that were potentially associated with quantity and impact of research publications (applicants for the YI programme).</p