Caveats for using statistical significance tests in research assessments

This paper raises concerns about the advantages of using statistical significance tests in research assessments as has recently been suggested in the debate about proper normalization procedures for citation indicators. Statistical significance tests are highly controversial and numerous criticisms have been leveled against their use. Based on examples from articles by proponents of the use statistical significance tests in research assessments, we address some of the numerous problems with such tests. The issues specifically discussed are the ritual practice of such tests, their dichotomous application in decision making, the difference between statistical and substantive significance, the implausibility of most null hypotheses, the crucial assumption of randomness, as well as the utility of standard errors and confidence intervals for inferential purposes. We argue that applying statistical significance tests and mechanically adhering to their results is highly problematic and detrimental to critical thinking. We claim that the use of such tests do not provide any advantages in relation to citation indicators, interpretations of them, or the decision making processes based upon them. On the contrary their use may be harmful. Like many other critics, we generally believe that statistical significance tests are over- and misused in the social sciences including scientometrics and we encourage a reform on these matters.Comment: Accepted version for Journal of Informetric

Does the specification of uncertainty hurt the progress of scientometrics?

In "Caveats for using statistical significance tests in research assessments,"--Journal of Informetrics 7(1)(2013) 50-62, available at arXiv:1112.2516 -- Schneider (2013) focuses on Opthof & Leydesdorff (2010) as an example of the misuse of statistics in the social sciences. However, our conclusions are theoretical since they are not dependent on the use of one statistics or another. We agree with Schneider insofar as he proposes to develop further statistical instruments (such as effect sizes). Schneider (2013), however, argues on meta-theoretical grounds against the specification of uncertainty because, in his opinion, the presence of statistics would legitimate decision-making. We disagree: uncertainty can also be used for opening a debate. Scientometric results in which error bars are suppressed for meta-theoretical reasons should not be trusted

IMF Surveillance and Financial Markets - A Political Economy Analysis

The International Monetary Fund (IMF) is in the process of re-inventing itself with bilateral and multilateral surveillance emerging as a key function. The paper analyses how IMF surveillance announcements may be influenced by political power that member countries exert at the IMF. First, we analyze the content of Article IV Public Information Notices (PIN), and second, we use the financial market reaction to the release PINs as tools to identify the role of political economy factors for IMF surveillance. For a set of emerging market economies, the paper finds that financial markets react more favorable to PIN releases for politically influential member countries. Moreover, IMF surveillance appears to be systematically more favorable for countries with larger IMF loans outstanding, consistent with the finding in the literature that the IMF may engage in ‘defensive surveillance’.IMF, surveillance, political economy, sovereign spreads, financial markets, emerging market economies

Ten simple rules for reporting voxel-based morphometry studies

Voxel-based morphometry [Ashburner, J. and Friston, K.J., 2000. Voxel-based morphometry—the methods. NeuroImage 11(6 Pt 1), 805–821] is a commonly used tool for studying patterns of brain change in development or disease and neuroanatomical correlates of subject characteristics. In performing a VBM study, many methodological options are available; if the study is to be easily interpretable and repeatable, the processing steps and decisions must be clearly described. Similarly, unusual methods and parameter choices should be justified in order to aid readers in judging the importance of such options or in comparing the work with other studies. This editorial suggests core principles that should be followed and information that should be included when reporting a VBM study in order to make it transparent, replicable and useful

An Evaluation of the Teacher Advancement Program (TAP) in Chicago: Year One Impact Report

Based on test scores, teacher records, surveys, and interviews, examines the first-year impact of the TAP program, in which teachers delivering added value to student achievement and quality classroom performance earn extra pay and become mentors

The substantive and practical significance of citation impact differences between institutions: Guidelines for the analysis of percentiles using effect sizes and confidence intervals

In our chapter we address the statistical analysis of percentiles: How should the citation impact of institutions be compared? In educational and psychological testing, percentiles are already used widely as a standard to evaluate an individual's test scores - intelligence tests for example - by comparing them with the percentiles of a calibrated sample. Percentiles, or percentile rank classes, are also a very suitable method for bibliometrics to normalize citations of publications in terms of the subject category and the publication year and, unlike the mean-based indicators (the relative citation rates), percentiles are scarcely affected by skewed distributions of citations. The percentile of a certain publication provides information about the citation impact this publication has achieved in comparison to other similar publications in the same subject category and publication year. Analyses of percentiles, however, have not always been presented in the most effective and meaningful way. New APA guidelines (American Psychological Association, 2010) suggest a lesser emphasis on significance tests and a greater emphasis on the substantive and practical significance of findings. Drawing on work by Cumming (2012) we show how examinations of effect sizes (e.g. Cohen's d statistic) and confidence intervals can lead to a clear understanding of citation impact differences

Literacy in India

Literacy refers to an individual’s ability to communicate through reading and writing. The literacy rate for any population measures the fraction of the population, above a certain cut-off age, that is literate. Based on the most recent statistics compiled by UNESCO, more than one in three Indians above the age of 15 years is unable to read and write. Further, the roughly 268 million adult illiterates in India constitute one-third of the global population of illiterates. International comparisons show that the Indian literacy rate is well below those for other populous countries like China and also below those for developing countries in general