From p-values to objective probabilities in\ud assessing medical treatments

The assessment of the effectiveness of a treatment in a clinical trial, depends on calculating p-values. However, p-values are only indirect and partial indicators of a genuine effect. Particularly in situations where publication bias is very likely, assessment using a p-value of 0.05 may not be sufficiently cautious. In other situations it seems reasonable to believe that assessment based on p-values may be unduly conservative. Assessments could be improved by using prior information. This implies using a Bayesian approach to take account of prior probability. However, the use of prior information in the form of expert opinion can allow bias. A method is given here that applies to assessments already included or likely to be included in the Cochrane Collaboration, excluding those reviews concerning new drugs. This method uses prior information and a Bayesian approach, but the prior information comes not from expert opinion but simply from the distribution of effectiveness apparent in a random sample of summary statistics in the Cochrane Collaboration. The method takes certain types of summary statistics and their confidence intervals and with the help of a graph, translates this into probabilities that the treatments being trialled are effective

False negatives in evidence based medicine

Evidence Based Medicine (EBM) is a term used for the current dominant methodology for deciding what medical treatments should be accepted as valid. It places great emphasis on Randomised Clinical Trials (RCTs) which are analysed according to a strict frequentist paradigm, with a rigid p-value ≤0.05 criterion but with little consideration of prior probabilities or the cost of errors. Accordingly, low cost, safe treatments where there is prior knowledge of at least slight effectiveness, may often be inappropriately discarded by EBM. The Cochrane Collaboration is an online central repository of RCTs and meta-analyses of RCTs. This paper uses statistical methods applied to a random sample of outcomes listed in the Cochrane Collaboration, to estimate the negative predictive value when treatments are declared ineffective as a result of positive outcomes which do not achieve the p≤ 0.05 criterion. The data were analysed using six different models in order to determine the proportion of genuinely ineffective treatments in the set of all positive outcomes where p>0.05. All six methods give point estimates substantially less than half for the negative predictive value when the decision rule is to declare treatments to be ineffective when their outcome is positive but p>0.05. Although confidence interval estimation indicates considerable uncertainty in these estimates, it seems reasonable to conclude that when a RCT gives a positive outcome but p0.05, the conventional EBM decision to declare the treatment to be ineffective, is likely to be wrong more often than not

From evidence- to science-based medicine

Background/Aims: Evidence-based medicine (EBM) tends to dismiss treatments which lead to non-statistically significant results and labels such treatments as having 'no effect'. If all such treatments really had no effect, there would be as many non-significantly negative results as non-significantly positive results. In practice there is a considerable excess of weakly positive results. This presentation suggests possible ways of using data on the distribution of summary statistical values to improve the EBM paradigm. \ud \ud Methods: A sample of 100 summary statistical values was chosen randomly from the Cochrane collection for evidence based medicine. Several models were used to analyse the data.\ud \ud Results: Weakly positive values outnumber weakly negative by about 3:1, suggesting that about two-thirds of weakly positive results are being produced by treatments which have at least a small positive effect rather than deserving the 'no effect' label. More complex models of this data agree with this estimate, albeit with wide confidence intervals. \ud \ud Conclusion: With further data collection and research, it will become possible to give a probability that a treatment is effective rather than assign dichotomous labels 'effective' and of 'no effect' to treatments. Knowledge about the treatment aside from the statistical result, and knowledge of the cost of making a wrong assessment of treatments can be further incorporated into decision making, so that treatments are based on science and not just the vagaries of statistical assessment

Credible [software and source code]

The "credible" program is a calculator which processes conventional statistics to improve estimates of relative risks. To install "credible" (windows only), download the executable file (setupcrediblec.exe) and then click on it. This installs the program "credible" and its required parameter file. Also downloadable are the source code (credallfortran.f90) and parameters (credparams.txt). Those who wish to avoid downloading an executable or who wish to modify the program, can download the Fortran 90 file. The parameters must then be downloaded to the same folder as the Fortran 90 file. The source code can then be compiled with a Fortran compiler to give a command line version of the program that can run on any operating system (MS-DOS, Linux, Mac). A Linux version of the compiled program is available on request to: [email protected]. The following text is included in a MoreInfo window as part of the program. It provides a summary of the ideas behind the calculation and the circumstances in which they can be applied: REPLACING p-VALUES and CONFIDENCE INTERVALS with OBJECTIVE PROBABILITIES and CREDIBLE INTERVALS Someone naive to medical statistics would expect a p-value to mean “the chance that the apparently favourable results in a trial of a new treatment are due to chance”. Similarly they would expect a 95% confidence interval to mean that “the true amount of difference made by the new treatment has a 95% chance of being somewhere in the given range”. Sadly, such expectations are incorrect as the following two examples show: p-VALUES: Consider two separate trials of treatment A versus standard treatment in one trial and treatment B versus standard treatment in the other trial, with both trials being for the same disease. Now consider what we should think about these treatments if, coincidentally, they both gave the same mildly favourable results. They would then both give the same p-value. However treatment A might be based on the best medical science and treatment B might be the placement under the patient's pillow of an amethyst crystal blessed by the Great Popuff. Common-sense tells us that despite the equal p-value, it cannot be true that both treatments are equally likely to be effective. The correct interpretation of p-values is that they are the probability of getting results at least as suggestive of a real effect as the results to hand, when in fact there is no real effect and the outcome is entirely due to chance. If chance can “quite easily” explain the outcome, the effect is said to be “not significant”, otherwise it is “significant”. In this context, “quite easily” refers to any chance that is more probable than 1/20 or 0.05 . 95% CONFIDENCE INTERVALS: Consider a person who wants to establish a 95% confidence interval for the height of men, but who is lazy and types into the computer just two heights he has measured – 160cm and 180cm. A computer package will then calculate that we can be 95% confident that the average height of men is between 43cm and 297cm – but clearly we have 100% certainty, not 95% probability, that average height is in this range. The correct interpretation of 95% confidence intervals for the true amount, is that they are a range of values readily compatible with the data. The ready compatibility is determined by a criterion related to a p-value of 0.05 We see that in calculating p-values and confidence intervals, traditional or frequentist statistics doesn't give us the probabilities we want. Traditional statistics cannot use the data to tell us the “chance that it is chance” or give a 95% probability range for an amount, because probability calculations in the light of data have to be built on probabilities prior to knowing the data. Bayesian statistics takes a different approach and uses prior probabilities to calculate the relevant probabilities. However, prior probabilities have been little used in medicine so traditional statistics leaves us with p-values and confidence intervals which have only a tangential and partial connection to what we would want them to mean. Prior probabilities are little used in medicine because it was assumed that they would be based on expert opinion and even the best experts can be biased, or even corrupted by commercial interests. Instead traditional statistics lets the data “speak for itself” but that has meant we have been left with p-values and confidence intervals and their indirect and partial relationship to what we want to know about a treatment. Now however a method has been developed of obtaining objective prior probabilities about medical treatments. The method was explained in a paper by D Kault & S Kault published in PLOSOne in November 2015 entitled “From p-values to objective probabilities in assessing medical treatments”. The ideas in this paper have now been extended to the calculation of credible intervals to replace confidence intervals. The extended method was published (initially online) in “Evidence-Based Medicine” (ebm.bmj.com) in January 2017 as “The Maimed Martian, credible intervals and bias against benefit” (DOI: 10.1136/ebmed-2016-110539). These credible intervals have a number of advantages over confidence intervals – they tend to be shorter, they allow for publication bias and importantly, there is a 95% chance that the value of interest lies in the 95% credible interval. To accompany the publication in Evidence-Based Medicine, this program “credible” is being distributed. Type into “credible” a relative risk and its 95% confidence interval and out will come the 95% credible interval together with some other measures, such as the probability that the treatment has some positive effectiveness. The program “credible” applies in a somewhat restricted set of circumstances. Full details are given in the PLOSOne article but the main restrictions are as follows:- 1) The outcome is expressed as a relative risk, hazard ratio or odds ratio together with the confidence interval. All such statistics are for simplicity abbreviated as “r r”. (The statistical theory commonly used in confidence interval calculations for r r's, results in a lognormally distributed confidence interval in which the geometric mean of the confidence interval bounds is the r r measure. “Credible” is not calibrated for use on statistics where this is not true. Credible checks this requirement and gives a warning if it is not met.) 2) The clinical trial has to be of the form of a comparison of a standard treatment plus an additional treatment in one arm versus a standard treatment in the control arm. In a trial of this form it can be anticipated that the effect of the additional treatment, if any, is more likely to be in a positive rather than negative direction. A positive direction may be shown by a relative risk less than 1.00 (as in a decreased risk of dying with the additional treatment) or shown by a relative risk greater than 1.00 (as in an increased “risk” of someone trying an anti-smoking aide being successful in giving up smoking). Before proceeding with the calculation, the “credible” program therefore requires an answer to an additional question “Is the relative risk in the direction anticipated?” Occasionally there is an extra negative in this process that needs to be considered. - a trial may be of a comparison between a standard treatment and, for cost saving purposes, a treatment regime that omits some component of the standard treatment. In such a case, the anticipated effect would be in the direction favouring the full standard treatment – though if this was convincingly the result, it would be disappointing for those conducting the trial. 3) The third requirement is that the clinical trial does not involve commercial interests – new drugs or devices under patent. 4) The trial also has to be of sufficient quality that would make it eligible to be analysed in a Cochrane review. The prior probabilities used by “credible” are objective in that they are not based on expert opinion. Instead, they are based on an estimate of what may be called an “effectiveness distribution” as determined by analysing a random sample of statistics from the Cochrane Collaboration. This effectiveness distribution is calculated after making allowance for publication bias. “Credible” uses this effectiveness distribution together with the summary statistics from a clinical trial or meta-analysis and gives the probabilities that the treatment has positive or zero effectiveness and the probability that it is counter-productive. It also gives the shortest interval which has a 95% probability of containing the true value – the 95% credible interval. As well, “credible” also calculates a single best estimate for the true value of r r. This estimate takes into account the actual value obtained, the accuracy with which it was measured as indicated by the confidence interval and the appropriate adjustment in the light of the “effectiveness distribution”. This best guess is termed the “expected r r” and is the first figure given. If this work was repeated based on prior probabilities from a different random sample from Cochrane, slightly different results could be obtained. For accuracy, in quoting the probabilities obtained from “credible”, the prior effectiveness distribution should, at least implicitly, be specified. It is suggested that perhaps the credible intervals obtained with this software should be termed “MM credible intervals”, after the Maimed Martian in an anecdote in the 2017 paper which described their calculation. A Linux version and/or the source code for “credible” is available on request to: [email protected] David Kault (MBBS, BSc, PhD) College of Science and Engineering, James Cook University, Qld, Australi

Parameter estimates for the single peaked model.

<p>Parameter estimates for the single peaked model.</p

Parameter estimates for the standard model.

<p>Parameter estimates for the standard model.</p

Distribution of the effectiveness of 10000 artificially generated points using the point estimates found for the standard model.

<p>The line gives the theoretical density function used to generate the data.</p

Theoretical parameters used to generate a large sample and corresponding calculated parameter values from that sample.

<p>Theoretical parameters used to generate a large sample and corresponding calculated parameter values from that sample.</p

Relation between parameter uncertainties and posterior probability calculations.

<p>Relation between parameter uncertainties and posterior probability calculations.</p

Comparison between indigenous mortality rates in a provincial Queensland prison with the general indigenous population

There has been much concern about the gap in disadvantage and mortality between Indigenous and non-Indigenous Australians. One focus of this concern has been black deaths in custody. This has been considered to be particularly relevant as Indigenous people are vastly over-represented in Australian prisons. This author recently attended an Aboriginal protest rally and was disturbed by the discordance between the emphasis given to black deaths in custody and this author's own experience as a part-time general practitioner in the male wing of a provincial Queensland prison over the previous 5.75 years. In response, an analysis was performed comparing the Indigenous male mortality in this prison with the expected mortality of an Indigenous male population of the same size and age structure. The mortality of prisoners is much lower than expected. The reasons for the improved survival of Indigenous prisoners, and its implications, are briefly discussed