When Should Potentially False Research Findings Be Considered Acceptable?

Ioannidis estimated that most published research findings are false [1], but he did not indicate when, if at all, potentially false research results may be considered as acceptable to society. We combined our two previously published models [2,3] to calculate the probability above which research findings may become acceptable. A new model indicates that the probability above which research results should be accepted depends on the expected payback from the research (the benefits) and the inadvertent consequences (the harms). This probability may dramatically change depending on our willingness to tolerate error in accepting false research findings. Our acceptance of research findings changes as a function of what we call “acceptable regret,” i.e., our tolerance of making a wrong decision in accepting the research hypothesis. We illustrate our findings by providing a new framework for early stopping rules in clinical research (i.e., when should we accept early findings from a clinical trial indicating the benefits as true?). Obtaining absolute “truth” in research is impossible, and so society has to decide when less-than-perfect results may become acceptable

Screening for prostate cancer: systematic review and meta-analysis of randomised controlled trials

Objective To examine the evidence on the benefits and harms of screening for prostate cancer

Rational decision-making in medicine: implications for overuse and underuse

open access articleIn spite of substantial spending and resource utilization, today's health care remains characterized by poor outcomes, largely due to overuse (over-testing/treatment) or underuse (under-testing/treatment) of services. To a significant extent, this is a consequence of low-quality decision-making that appears to violate various rationality criteria. Such sub-optimal decision-making is considered a leading cause of death and is responsible for more than 80% of health expenses. In this paper, we address the issue of overuse or underuse of healthcare interventions from the perspective of rational choice theory. We show that what is considered rational under one decision theory may not be considered rational under a different theory. We posit that the questions and concerns regarding both underuse and overuse have to be addressed within a specific theoretical framework. The applicable rationality criterion, and thus the “appropriateness” of health care delivery choices, depends on theory selection that is appropriate to specific clinical situations. We provide a number of illustrations showing how the choice of theoretical framework influences both our policy and individual decision-making. We also highlight the practical implications of our analysis for the current efforts to measure the quality of care and link such measurements to the financing of healthcare services

A regret theory approach to decision curve analysis: A novel method for eliciting decision makers' preferences and decision-making

<p>Abstract</p> <p>Background</p> <p>Decision curve analysis (DCA) has been proposed as an alternative method for evaluation of diagnostic tests, prediction models, and molecular markers. However, DCA is based on expected utility theory, which has been routinely violated by decision makers. Decision-making is governed by intuition (system 1), and analytical, deliberative process (system 2), thus, rational decision-making should reflect both formal principles of rationality and intuition about good decisions. We use the cognitive emotion of regret to serve as a link between systems 1 and 2 and to reformulate DCA.</p> <p>Methods</p> <p>First, we analysed a classic decision tree describing three decision alternatives: treat, do not treat, and treat or no treat based on a predictive model. We then computed the expected regret for each of these alternatives as the difference between the utility of the action taken and the utility of the action that, in retrospect, should have been taken. For any pair of strategies, we measure the difference in net expected regret. Finally, we employ the concept of acceptable regret to identify the circumstances under which a potentially wrong strategy is tolerable to a decision-maker.</p> <p>Results</p> <p>We developed a novel dual visual analog scale to describe the relationship between regret associated with "omissions" (e.g. failure to treat) vs. "commissions" (e.g. treating unnecessary) and decision maker's preferences as expressed in terms of threshold probability. We then proved that the Net Expected Regret Difference, first presented in this paper, is equivalent to net benefits as described in the original DCA. Based on the concept of acceptable regret we identified the circumstances under which a decision maker tolerates a potentially wrong decision and expressed it in terms of probability of disease.</p> <p>Conclusions</p> <p>We present a novel method for eliciting decision maker's preferences and an alternative derivation of DCA based on regret theory. Our approach may be intuitively more appealing to a decision-maker, particularly in those clinical situations when the best management option is the one associated with the least amount of regret (e.g. diagnosis and treatment of advanced cancer, etc).</p

Impact of quality of evidence on the strength of recommendations: an empirical study

<p>Abstract</p> <p>Background</p> <p>Evidence is necessary but not sufficient for decision-making, such as making recommendations by clinical practice guideline panels. However, the fundamental premise of evidence-based medicine (EBM) rests on the assumed link between the quality of evidence and "truth" and/or correctness in making guideline recommendations. If this assumption is accurate, then the quality of evidence ought to play a key role in making guideline recommendations. Surprisingly, and despite the widespread penetration of EBM in health care, there has been no empirical research to date investigating the impact of quality of evidence on the strength of recommendations made by guidelines panels.</p> <p>Methods</p> <p>The American Association of Blood Banking (AABB) has recently convened a 12 member panel to develop clinical practice guidelines (CPG) for the use of fresh-frozen plasma (FFP) for 6 different clinical indications. The panel was instructed that 4 factors should play a role in making recommendation: quality of evidence, uncertainty about the balance between desirable (benefits) and undesirable effects (harms), uncertainty or variability in values and preferences, and uncertainty about whether the intervention represents a wise use of resources (costs). Each member of the panel was asked to make his/her final judgments on the strength of recommendation and the overall quality of the body of evidence. "Voting" was anonymous and was based on the use of GRADE (Grading quality of evidence and strength of recommendations) system, which clearly distinguishes between quality of evidence and strength of recommendations.</p> <p>Results</p> <p>Despite the fact that many factors play role in formulating CPG recommendations, we show that when the quality of evidence is higher, the probability of making a strong recommendation for or against an intervention dramatically increases. Probability of making strong recommendation was 62% when evidence is "moderate", while it was only 23% and 13% when evidence was "low" or "very low", respectively.</p> <p>Conclusion</p> <p>We report the first empirical evaluation of the relationship between quality of evidence pertinent to a clinical question and strength of the corresponding guideline recommendations. Understanding the relationship between quality of evidence and probability of making (strong) recommendation has profound implications for the science of quality measurement in health care.</p

Quality of Pharmaceutical Advertisements in Medical Journals: A Systematic Review

Journal advertising is one of the main sources of medicines information to doctors. Despite the availability of regulations and controls of drug promotion worldwide, information on medicines provided in journal advertising has been criticized in several studies for being of poor quality. However, no attempt has been made to systematically summarise this body of research. We designed this systematic review to assess all studies that have examined the quality of pharmaceutical advertisements for prescription products in medical and pharmacy journals.Studies were identified via searching electronic databases, web library, search engine and reviewing citations (1950 - February 2006). Only articles published in English and examined the quality of information included in pharmaceutical advertisements for prescription products in medical or pharmacy journals were included. For each eligible article, a researcher independently extracted the data on the study methodology and outcomes. The data were then reviewed by a second researcher. Any disagreements were resolved by consensus. The data were analysed descriptively. The final analysis included 24 articles. The studies reviewed advertisements from 26 countries. The number of journals surveyed in each study ranged from four to 24 journals. Several outcome measures were examined including references and claims provided in advertisements, availability of product information, adherence to codes or guidelines and presentation of risk results. The majority of studies employed a convenience-sampling method. Brand name, generic name and indications were usually provided. Journal articles were commonly cited to support pharmaceutical claims. Less than 67% of the claims were supported by a systematic review, a meta-analysis or a randomised control trial. Studies that assessed misleading claims had at least one advertisement with a misleading claim. Two studies found that less than 28% of claims were unambiguous clinical claims. Most advertisements with quantitative information provided risk results as relative risk reduction. Studies were conducted in 26 countries only and then the generalizability of the results is limited.Evidence from this review indicates that low quality of journal advertising is a global issue. As information provided in journal advertising has the potential to change doctors' prescribing behaviour, ongoing efforts to increase education about drug promotion are crucial. The results from our review suggest the need for a global pro-active and effective regulatory system to ensure that information provided in medical journal advertising is supporting the quality use of medicines