Applied Evaluative Informetrics: Part 1

This manuscript is a preprint version of Part 1 (General Introduction and Synopsis) of the book Applied Evaluative Informetrics, to be published by Springer in the summer of 2017. This book presents an introduction to the field of applied evaluative informetrics, and is written for interested scholars and students from all domains of science and scholarship. It sketches the field's history, recent achievements, and its potential and limits. It explains the notion of multi-dimensional research performance, and discusses the pros and cons of 28 citation-, patent-, reputation- and altmetrics-based indicators. In addition, it presents quantitative research assessment as an evaluation science, and focuses on the role of extra-informetric factors in the development of indicators, and on the policy context of their application. It also discusses the way forward, both for users and for developers of informetric tools.Comment: The posted version is a preprint (author copy) of Part 1 (General Introduction and Synopsis) of a book entitled Applied Evaluative Bibliometrics, to be published by Springer in the summer of 201

Lost in Translation: Piloting a Novel Framework to Assess the Challenges in Translating Scientific Uncertainty From Empirical Findings to WHO Policy Statements.

BACKGROUND:Calls for evidence-informed public health policy, with implicit promises of greater program effectiveness, have intensified recently. The methods to produce such policies are not self-evident, requiring a conciliation of values and norms between policy-makers and evidence producers. In particular, the translation of uncertainty from empirical research findings, particularly issues of statistical variability and generalizability, is a persistent challenge because of the incremental nature of research and the iterative cycle of advancing knowledge and implementation. This paper aims to assess how the concept of uncertainty is considered and acknowledged in World Health Organization (WHO) policy recommendations and guidelines. METHODS:We selected four WHO policy statements published between 2008-2013 regarding maternal and child nutrient supplementation, infant feeding, heat action plans, and malaria control to represent topics with a spectrum of available evidence bases. Each of these four statements was analyzed using a novel framework to assess the treatment of statistical variability and generalizability. RESULTS:WHO currently provides substantial guidance on addressing statistical variability through GRADE (Grading of Recommendations Assessment, Development, and Evaluation) ratings for precision and consistency in their guideline documents. Accordingly, our analysis showed that policy-informing questions were addressed by systematic reviews and representations of statistical variability (eg, with numeric confidence intervals). In contrast, the presentation of contextual or "background" evidence regarding etiology or disease burden showed little consideration for this variability. Moreover, generalizability or "indirectness" was uniformly neglected, with little explicit consideration of study settings or subgroups. CONCLUSION:In this paper, we found that non-uniform treatment of statistical variability and generalizability factors that may contribute to uncertainty regarding recommendations were neglected, including the state of evidence informing background questions (prevalence, mechanisms, or burden or distributions of health problems) and little assessment of generalizability, alternate interventions, and additional outcomes not captured by systematic review. These other factors often form a basis for providing policy recommendations, particularly in the absence of a strong evidence base for intervention effects. Consequently, they should also be subject to stringent and systematic evaluation criteria. We suggest that more effort is needed to systematically acknowledge (1) when evidence is missing, conflicting, or equivocal, (2) what normative considerations were also employed, and (3) how additional evidence may be accrued

Do peers see more in a paper than its authors?

Recent years have shown a gradual shift in the content of biomedical publications that is freely accessible, from titles and abstracts to full text. This has enabled new forms of automatic text analysis and has given rise to some interesting questions: How informative is the abstract compared to the full-text? What important information in the full-text is not present in the abstract? What should a good summary contain that is not already in the abstract? Do authors and peers see an article differently? We answer these questions by comparing the information content of the abstract to that in citances-sentences containing citations to that article. We contrast the important points of an article as judged by its authors versus as seen by peers. Focusing on the area of molecular interactions, we perform manual and automatic analysis, and we find that the set of all citances to a target article not only covers most information (entities, functions, experimental methods, and other biological concepts) found in its abstract, but also contains 20% more concepts. We further present a detailed summary of the differences across information types, and we examine the effects other citations and time have on the content of citances

Reading in the Disciplines: The Challenges of Adolescent Literacy

A companion report to Carnegie's Time to Act, focuses on the specific skills and literacy support needed for reading in academic subject areas in higher grades. Outlines strategies for teaching content knowledge and reading strategies together

Synopsizing “literature review” for scientific publications

Because the number of scientific publications in most disciplines is expanding rapidly, traditional academic search engines can hardly satisfy scholars’ need to retrieve and assimilate the information they are looking for. In this study we investigate a new summarization problem: creating a synopsis “Literature Review” of a collection of candidate cited papers in response to a query, via different methods and indicators. In more detail, we compare the use of different methods and indicators for generating citation clusters and summarized reviews by analyzing publication abstracts, citation contexts, and co-cite relationships. We also validate the usefulness of a user’s query during this process by comparing query-dependent and query-independent clustering and summarization. One interesting outcome of this study is that citation contexts are more suitable for clustering related papers, whereas abstracts are more accurate for generating longer review-like summaries. The initial user query is also helpful for enhancing clustering and summarization performance

Metascientific views: Challenge and opportunity for philosophy of biology in practice

In this paper I take evolutionary biology as an example to reflect on the role of philosophy and on the transformations that philosophy is constantly stimulated to do in its own approach when dealing with science. I consider that some intellectual movements within evolutionary biology (more specifically, the various calls for 'synthesis') express metascientific views, i.e., claims about 'what it is to do research' in evolutionary biology at different times. In the construction of metascientific views I see a fundamental role to be played by philosophy, and, at the same time, a need to complement the philosophical methods with many more methods coming from other sciences. What leads philosophy out of itself is its own attention to scientific practice. My humble methodological suggestions are, at this stage, only meant to help us imagine metascientific views that are built with a more scientific, interdisciplinary approach, in order to attenuate partiality, subjectivity and impressionism in describing the scientific community. And yet, we should not be naïve and imbued with the myth of 'datadriven' research, especially in this field: other complex issues about metascientific views call for a serious, constant philosophical reflection on scientific practice