Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisions

Summary: In science, peer review is the best-established method of assessing manuscripts for publication and applications for research fellowships and grants. However, the fairness of peer review, its reliability and whether it achieves its aim to select the best science and scientists has often been questioned. The paper presents the first comprehensive study on committee peer review for the selection of doctoral (Ph.D.) and post-doctoral research fellowship recipients. We analysed the selection procedure followed by the Boehringer Ingelheim Fonds (B.I.F.), a foundation for the promotion of basic research in biomedicine, with regard to the reliability, fairness and predictive validity of the procedure - the three quality criteria for professional evaluations. We analysed a total of 2,697 applications, 1,954 for doctoral and 743 for post-doctoral fellowships. In 76% of the cases, the fellowship award decision was characterized by agreement between reviewers. Similar figures for reliability have been reported for the grant selection procedures of other major funding agencies. With regard to fairness, we analysed whether potential sources of bias, i.e., gender, nationality, major field of study and institutional affiliation, could have influenced decisions made by the B.I.F. Board of Trustees. For post-doctoral fellowship applications, no statistically significant influence of any of these variables could be observed. For doctoral fellowship applications, we found evidence of an institutional, major field of study and gender bias, but not of a nationality bias. The most important aspect of our study was to investigate the predictive validity of the procedure, i.e., whether the foundation achieves its aim to select as fellowship recipients the best junior scientists. Our bibliometric analysis showed that this is indeed the case and that the selection procedure is thus highly valid: research articles by B.I.F. fellows are cited considerably more often than the “average' paper (average citation rate) published in the journal sets corresponding to the fields “Multidisciplinary', “Molecular Biology & Genetics', and “Biology & Biochemistry' in Essential Science Indicators (ESI) from the Institute for Scientific Information (ISI, Philadelphia, Pennsylvania, USA). Most of the fellows publish within these field

Education alignment

This essay reviews recent developments in embedding data management and curation skills into information technology, library and information science, and research-based postgraduate courses in various national contexts. The essay also investigates means of joining up formal education with professional development training opportunities more coherently. The potential for using professional internships as a means of improving communication and understanding between disciplines is also explored. A key aim of this essay is to identify what level of complementarity is needed across various disciplines to most effectively and efficiently support the entire data curation lifecycle

PICES Press, Vol. 21, No. 2, Summer 2013

•The 2013 Inter-sessional Science Board Meeting: A Note from the Science Board Chairman (pp. 1-4) •ICES/PICES Workshop on Global Assessment of the Implications of Climate Change on the Spatial Distribution of Fish and Fisheries (pp. 5-8) •PICES participates in a Convention on Biological Diversity Regional Workshop (pp. 9-11) •Social and Economic Indicators for Status and Change within North Pacific Ecosystems (pp. 12-13) •The Fourth International Jellyfish Bloom Symposium (pp. 14-15) •Workshop on Radionuclide Science and Environmental Quality in the North Pacific (pp. 16-17) •PICES-MAFF Project on Marine Ecosystem Health and Human Well-Being: Indonesia Workshop (pp. 18-19) •Socioeconomic Indicators for United States Fisheries and Fishing Communities (pp. 20-23) •Harmful Algal Blooms in a Changing World (pp. 24-25, 27) •Enhancing Scientific Cooperation between PICES and NPAFC (pp. 26-27) •Workshop on Marine Biodiversity Conservation and Marine Protected Areas in the Northwest Pacific (pp. 28-29) •The State of the Western North Pacific in the Second Half of 2012 (pp. 30-31) •Stuck in Neutral in the Northeast Pacific Ocean (pp. 32-33) •The Bering Sea: Current Status and Recent Trends (pp. 34-36) •For your Bookshelf (p. 37) •Howard Freeland takes home Canadian awards (p. 38

Competencies of Polish Scientists as a Contribution to the Success of Innovation Research and Development Projects

The states which top the list of the number of conducted research and development activities put substantial emphasis on a scientist’s competencies. Today, experts are able to determine the competencies that will decide on the success of projects in the next decade. They include, among others, leadership skills, team work, entrepreneurship as well as international and cross-sector mobility. What is the place of Polish scientists in relation to these competencies? What are their strongest and weakest points? How do the competencies of Polish scientists translate into the success of projects conducted in our country? These questions were to find their answers thanks to the research conducted in 2011 commissioned by the National Information Processing Institute. This article presents some of the results. It includes, amongst others, the differences between scientists from science departments and companies as well as between project managers and members of research teams.Preparation and printing funded by the National Agency for Research and Development under project “Kreator Innowacyjności – wparcie dla Przedsiębiorczości akademickiej

The selection, appraisal and retention of digital scientific data: dighlights of an ERPANET/CODATA workshop

CODATA and ERPANET collaborated to convene an international archiving workshop on the selection, appraisal, and retention of digital scientific data, which was held on 15-17 December 2003 at the Biblioteca Nacional in Lisbon, Portugal. The workshop brought together more than 65 researchers, data and information managers, archivists, and librarians from 13 countries to discuss the issues involved in making critical decisions regarding the long-term preservation of the scientific record. One of the major aims for this workshop was to provide an international forum to exchange information about data archiving policies and practices across different scientific, institutional, and national contexts. Highlights from the workshop discussions are presented

Should Top Universities Be Led By Top Researchers and Are They? A Citations Analysis

[Excerpt] This paper addresses the question: should the world’s top universities be led by top researchers, and are they? The lifetime citations are counted by hand of the leaders of the world’s top 100 universities identified in a global university ranking. These numbers are then normalized by adjusting for the different citation conventions across academic disciplines. Two statistical measures are used -- Pearson\u27s correlation coefficient and Spearman\u27s rho. This study documents a positive correlation between the lifetime citations of a University’s president and the position of that university in the global ranking. Better universities are run by better researchers. The results are not driven by outliers. That the top universities in the world -- who have the widest choice of candidates -- systematically appoint top researchers as their vice chancellors and presidents seems important to understand. This paper also shows that the pattern of presidents life-time citations follows a version of Lotka’s power law. There are two main areas of contribution. First, this paper attempts to use bibliometric data to address a performance- related question of a type not seen before (to the author’s knowledge). Second, despite the importance of research to research universities -- as described in many mission-statements -- no studies currently exist that ask whether it matters if the head of a research university is himself or herself a committed researcher. Given the importance of universities in the world, and the difficulty that many have in appointing leaders, this question seems pertinent

A Scientist's Guide to Achieving Broader Impacts through K-12 STEM Collaboration.

The National Science Foundation and other funding agencies are increasingly requiring broader impacts in grant applications to encourage US scientists to contribute to science education and society. Concurrently, national science education standards are using more inquiry-based learning (IBL) to increase students' capacity for abstract, conceptual thinking applicable to real-world problems. Scientists are particularly well suited to engage in broader impacts via science inquiry outreach, because scientific research is inherently an inquiry-based process. We provide a practical guide to help scientists overcome obstacles that inhibit their engagement in K-12 IBL outreach and to attain the accrued benefits. Strategies to overcome these challenges include scaling outreach projects to the time available, building collaborations in which scientists' research overlaps with curriculum, employing backward planning to target specific learning objectives, encouraging scientists to share their passion, as well as their expertise with students, and transforming institutional incentives to support scientists engaging in educational outreach

YouTube or You Lose: Grand Challenges Canada Explores Whether Scientists Are Ready for Web-Based Grant Competitions

It is not hard to trace the influence of technology on the way we read the literature or give scientific presentations. Not so long ago, chemists used hard copies of Chemical Abstracts to find papers and sticks of chalk to deliver talks. Only over the past decade have computer presentations become the norm. In contrast, the way that grants are evaluated has remained relatively unchanged: scientists submit written proposals that are then evaluated by committees of scientists in the field. Might this process soon change as well? The not-for-profit organization Grand Challenges Canada (GCC) recently sponsored a competition in which researchers presented audacious ideas to attack problems related to global health (Figure 1). In its search for bold ideas from scientists, the GCC organization tested a bold idea as well: each proposal had to be accompanied by a 2-min-long video for public consumption on the Internet. Web users were encouraged not only to view these video summaries but to participate in the evaluation of the proposals by means of clicking on a “thumbs up” button (similar to the “like” buttons found on YouTube and Facebook). The votes from the public video were used by GCC to evaluate each applicant’s ability to “engage the public and increase awareness in the grand challenges facing global health today”.^1 The competition collected over 180,000 votes and over 100,000 unique online visits from 156 countries in a mere 4 weeks—staggering statistics for scientific videos. While each applicant also submitted a written version of the proposal, which was privately evaluated by “standard” peer-review, the public video feature was one of the first direct implementations of Web 2.0 technology (user-interactive sites and applications) to evaluate scientific proposals. The competition raises an important question: to what extent, if any, should Web 2.0 technology or other direct evaluation by the public be used to determine the outcome of scientific grant proposals