Perception of the importance of chemistry research papers and comparison to citation rates.

Chemistry researchers are frequently evaluated on the perceived significance of their work with the citation count as the most commonly-used metric for gauging this property. Recent studies have called for a broader evaluation of significance that includes more nuanced bibliometrics as well as altmetrics to more completely evaluate scientific research. To better understand the relationship between metrics and peer judgements of significance in chemistry, we have conducted a survey of chemists to investigate their perceptions of previously published research. Focusing on a specific issue of the Journal of the American Chemical Society published in 2003, respondents were asked to select which articles they thought best matched importance and significance given several contexts: highest number of citations, most significant (subjectively defined), most likely to share among chemists, and most likely to share with a broader audience. The answers to the survey can be summed up in several observations. The ability of respondents to predict the citation counts of established research is markedly lower than the ability of those counts to be predicted by the h-index of the corresponding author of each article. This observation is conserved even when only considering responses from chemists whose expertise falls within the subdiscipline that best describes the work performed in an article. Respondents view both cited papers and significant papers differently than papers that should be shared with chemists. We conclude from our results that peer judgements of importance and significance differ from metrics-based measurements, and that chemists should work with bibliometricians to develop metrics that better capture the nuance of opinions on the importance of a given piece of research

Respondent evaluations and citations (2013) by paper.

<p>The top panel shows the composite selections of our respondents for the question asking which papers they thought had the most citations (blue) and the actual number of citations in 2013 (gray). The other panels also include the number of citations (gray) along with: selections for most significant (green, middle left), selections for which should be shared with chemists (yellow, middle right), which should be shared widely (orange, bottom left), and h-index of the corresponding author (red, bottom right) for each of the manuscripts in the journal issue.</p

Correlation of aggregate survey responses to 2013 citation counts.

<p>Correlation of aggregate survey responses to 2013 citation counts.</p

Respondent selections of most cited by subdiscipline.

<p>Each graph shows respondent selection for most cited (left axis, blue) and actual citations as of 2013 (right axis) for each paper. The top panel shows selections from all respondent subdisciplines. The middle panel shows selections where the respondent subdiscipline matches the article subdiscipline. The bottom panel shows selections where the respondent subdiscipline does not match the article subdiscipline.</p

Correlations between answers given for each question.

<p>Correlations between answers given for each question.</p

Analysis of selections within and outside of respondents’ subdisciplines.

<p>Analysis of selections within and outside of respondents’ subdisciplines.</p

Correlation between respondent choices and citation counts from 2013 and 2016.

<p>Correlation between respondent choices and citation counts from 2013 and 2016.</p

Correlations between respondent choices and citations (2013).

<p>The top panel is a graph plotting each papers number of actual citations (2013) versus the number of times it was chosen by our respondents for the question: ‘which three papers do you think have been cited most to-date’ (blue). The other panels also plot actual citations versus survey responses: most significant (green, middle left), shared with chemists (yellow, middle right), and shared widely (orange, bottom left). The number of actual citations versus the h-index of the corresponding author is shown in the bottom right panel (red).</p

Respondent demographics.

<p>Respondent demographics.</p