The null hypothesis significance test in health sciences research (1995-2006): statistical analysis and interpretation

<p>Abstract</p> <p>Background</p> <p>The null hypothesis significance test (NHST) is the most frequently used statistical method, although its inferential validity has been widely criticized since its introduction. In 1988, the <it>International Committee of Medical Journal Editors </it>(ICMJE) warned against sole reliance on NHST to substantiate study conclusions and suggested supplementary use of confidence intervals (CI). Our objective was to evaluate the extent and quality in the use of NHST and CI, both in English and Spanish language biomedical publications between 1995 and 2006, taking into account the <it>International Committee of Medical Journal Editors </it>recommendations, with particular focus on the accuracy of the interpretation of statistical significance and the validity of conclusions.</p> <p>Methods</p> <p>Original articles published in three English and three Spanish biomedical journals in three fields (General Medicine, Clinical Specialties and Epidemiology - Public Health) were considered for this study. Papers published in 1995-1996, 2000-2001, and 2005-2006 were selected through a systematic sampling method. After excluding the purely descriptive and theoretical articles, analytic studies were evaluated for their use of NHST with P-values and/or CI for interpretation of statistical "significance" and "relevance" in study conclusions.</p> <p>Results</p> <p>Among 1,043 original papers, 874 were selected for detailed review. The exclusive use of P-values was less frequent in English language publications as well as in Public Health journals; overall such use decreased from 41% in 1995-1996 to 21% in 2005-2006. While the use of CI increased over time, the "significance fallacy" (to equate statistical and substantive significance) appeared very often, mainly in journals devoted to clinical specialties (81%). In papers originally written in English and Spanish, 15% and 10%, respectively, mentioned statistical significance in their conclusions.</p> <p>Conclusions</p> <p>Overall, results of our review show some improvements in statistical management of statistical results, but further efforts by scholars and journal editors are clearly required to move the communication toward ICMJE advices, especially in the clinical setting, which seems to be imperative among publications in Spanish.</p

Sorry everyone, but it didn't work (p = 0.06)

The more or less ubiquitous use of Fisher-type statistics in quantitative/numeric evaluations of drug and alcohol education initiatives takes place within the context of a literature of long standing which suggests there are areas in which null hypothesis testing and the use of conventional cut-off points (i.e. the 1% and 5% probability levels) are inappropriate. This literature is largely ignored. The paper identifies some of these issues in terms of their relevance to the problems of evaluation of drug/alcohol programmes. The paper argues that the qualitative approach does not solve these problems but merely by-passes them, as well as being unsatisfactory in a number of ways. Thus, it is concluded that there is a pressing need for change in the type of quantitative approach adopted. Suggestions are made for a variety of exploratory methods, still involving broadly numerical analysis, which have a philosophical rather than a merely technical base, and which shed light on ‘what is going on’ rather than merely providing a binary decision (it worked/it did not work) derived from an arbitrary criterion for statistical significance and a null hypothesis which is usually known to be false from the start