The Convergence of Digital-Libraries and the Peer-Review Process

Pre-print repositories have seen a significant increase in use over the past fifteen years across multiple research domains. Researchers are beginning to develop applications capable of using these repositories to assist the scientific community above and beyond the pure dissemination of information. The contribution set forth by this paper emphasizes a deconstructed publication model in which the peer-review process is mediated by an OAI-PMH peer-review service. This peer-review service uses a social-network algorithm to determine potential reviewers for a submitted manuscript and for weighting the relative influence of each participating reviewer's evaluations. This paper also suggests a set of peer-review specific metadata tags that can accompany a pre-print's existing metadata record. The combinations of these contributions provide a unique repository-centric peer-review model that fits within the widely deployed OAI-PMH framework.Comment: Journal of Information Science [in press

Weighting peer reviewers

Our scientific community faces a sort of paradox. A large bulk of work has been done on data-oriented techniques devised to improve peer reputation and knowledge extraction from data, so as to improve trustworthiness of digital services involving coordination and cooperation among heterogeneous peers. But, perhaps surprisingly, to the best of our knowledge, such techniques have rarely been applied to the (for our own community, crucial) process of reducing noise in the process of peer reviewing our own papers. Goal of this work is to provide initial insights on the applicability of methodologies and tools from inferential statistical to the field of peer review quality control. Our contribution is threefold. First, we propose a statistical model where each technical program committee member (reviewer) is characterized as random noise added to the “actual” value of the paper. Second, we provide an iterative data-oriented approach based on Expectation-Maximization devised to estimate mean value and variance of the noise added by each reviewer; our approach uses only the ratings provided by the reviewers themselves and does not rely on any additional source of a-priori knowledge. Third, we make use of the estimated mean values and variances to improve the accuracy of paper's evaluation and ranking