Briggs, A.H.

Fenwick, E.

English

Enlighten: Publications

 
 
 
 
 
 
Fenwick, E. and Briggs, A.H. (2007) Cost-effectiveness acceptability 
curves in the dock: case not proven? Medical Decision Making 
27(2):pp. 93-95. 
 
 
 
 
 
 
 
http://eprints.gla.ac.uk/4167/ 
 
Deposited on: 12 May 2008 
 
 
Glasgow ePrints Service 
http://eprints.gla.ac.uk 
Cost-Effectiveness Acceptability Curves 
in the Dock: Case Not Proven?
Elisabeth Fenwick, PhD, Andrew Briggs, DPhil
The article in this issue by Groot Koerkamp and oth-ers1 discusses the limitations of cost-effectiveness
acceptability curves (CEACs) for presenting uncer-
tainty in cost-effectiveness analysis to policy makers
and urges us to “rethink their use in communicating
uncertainty.”1 We agree with the authors that uncer-
tainty is important for policy makers and that the
appropriate method to evaluate uncertainty is through
an assessment of both the probability of making an
error (“not selecting the ‘true’ preferred alternative”)
and the consequences associated with making an
error. A Bayesian value of information (VOI) analysis
provides this assessment of uncertainty. The question
remains about what is/are the appropriate means to
present information about uncertainty to policy mak-
ers. We take this opportunity to review the case against
CEACs presented within the article and to defend the
use and usefulness of CEACs for policy making in
health care.
BRING FORTH THE CHARGES
The authors state that CEACs are “unable to dis-
tinguish dramatically different joint distributions of
incremental cost and effect” and that this limits their
usefulness for policy making.1 They argue that the
use of CEACs to represent uncertainty restricts the
ability to synthesize evidence on cost-effectiveness
from other sources. In addition, they suggest that the
CEAC does not allow for integration of risk attitude,
provides no insight into the assessment of the need
for further research, and may mislead policy makers
regarding the appropriate treatment choice.
THE CASE FOR THE DEFENSE
We accept that CEACs are insensitive to any change
in the joint distribution of incremental cost and effect
in the northwest (NW) or southeast (SE) quadrants of
the cost-effectiveness plane. This is not, however, a
product of the cost-effectiveness acceptability curve
but rather of cost-effectiveness analysis itself, which
considers all points in each of these quadrants to be
equivalent in cost-effectiveness terms—that is, domi-
nant (SE) or dominated (NW). If information about
dominance is considered important to policy makers,
then the CEAC could be adapted very easily to provide
it (see Severens and others2). We also accept that
CEACs are insensitive to radial shifts in the joint dis-
tribution of incremental cost and effect in the north-
east or southwest quadrant of the cost-effectiveness
plane. This again is a consequence of cost-effectiveness
analysis. Points lying on the same ray through the ori-
gin have the same incremental cost-effectiveness ratio
(ICER) and thus, for any value of the maximum accept-
able incremental cost-effectiveness ratio, will have the
same outcome regarding cost-effectiveness (i.e., either
cost-effective or not). As such, any radial shift of the
incremental joint distribution, either toward or away
from the origin, represents identical uncertainty
regarding cost-effectiveness and will result in identical
CEACs.
With regard to the integration of other evidence
and opinions, we agree with the authors that “a
quantitative analysis is rarely the only or final word
on a decision.”1 Within a Bayesian framework, all
available evidence should be included and consid-
ered within the decision context. However, we dis-
pute the use of a summarized measure (“quantitative
statement”) of uncertainty as a means of weighting
results for a synthesis of the evidence from other
From Public Health and Health Policy, University of Glasgow, Glasgow,
Scotland.
Address correspondence to Elisabeth Fenwick, Public Health and Health
Policy, University of Glasgow, 1 Lilybank Gardens, Glasgow, G12 8RZ,
Scotland; telephone: 0141 330 5615; fax: 0141 330 5018; e-mail:
e.fenwick@clinmed.gla.ac.uk.
 
For Evaluation Only.
Copyright (c) by Foxit Software Company, 2004 - 2007
Edited by Foxit PDF Editor
sources. In our view, this type of integration of evi-
dence should be done formally and should involve
the individual components of the decision (e.g.,
parameters, costs, and effects) rather than the mea-
sure of cost-effectiveness (see Cooper and others3
and Fenwick and others4 for examples).
The misinterpretation of a high probability that a
treatment alternative is cost-effective with medical
urgency or importance is equivalent to confusing
statistical significance and clinical importance. This
is purely an education issue and is not specific to
the use of CEACs, hypothesis testing, or confidence
intervals.
In addition, we dispute the authors’ suggestions
that the CEAC does not allow for integration of risk
attitude, provides no insight into the assessment 
of the need for further research, and may mislead
policy makers regarding the appropriate treatment
choice. Much of the article centers on the argument
between the use of ICERs and net benefit (NB) for
economic evaluation and confuses this with issues of
presenting uncertainty. It was the difficulties inherent
with constructing confidence intervals (CIs) for ICERs
that led ultimately to the production of the CEAC and
to its proliferation within the medical literature. The
absence of these difficulties for NB promotes the
presentation of CI, but there is no reason that CEACs
cannot be presented for an analysis involving 
NB. Contrary to the authors’ statement that “CEACs
fail to capture the importance of uncertainty,”1 we
believe that CEACs encourage policy makers to
think about decision uncertainty and the conse-
quences of such uncertainty for policy making,
allowing them to cast off the shackles of confidence
intervals and hypothesis testing. The production
and presentation of intervals for NB, however, are
likely to encourage classical thinking regarding cost-
effectiveness (i.e., if the interval excludes zero, then
the intervention is cost-effective).
We accept that the CEAC does not present the pol-
icy maker with information concerning the appropri-
ate treatment choice (this is given by the expected
utilities), nor does it present information regarding
the consequences of an incorrect decision, but it does
present policy makers with graphical information
regarding the probability that a treatment alternative
is cost-effective over a range of values for the maxi-
mum acceptable ICER. In turn, the cost-effectiveness
acceptability frontier (CEAF) presents the probability
that the treatment choice made on the basis of
expected values is correct. As such, the complement
of the CEAF provides a graphical representation of
the probability of making an error, which is a major
component of the expected value of perfect informa-
tion (EVPI).5 Thus, there is a link between the CEAF
and the EVPI: over the range where the CEAF is
falling, the EVPI must be increasing as both the error
probability and the maximum acceptable ratio (used in
the calculation of the consequences of error) are increas-
ing. Over the range where the CEAF is increasing, the
direction of the EVPI will depend on the interaction
between the rate of change in the uncertainty and the
rate of change in the consequences. We have found
that for ranges where the CEAF is flattening off toward
a maximum (uncertainty is falling slightly), the change
in the consequences will dominate, and EVPI will
tend to increase.
THE WAY FORWARD
Having started boldly by proclaiming the issues and
limitations associated with the CEAC, the authors are
less clear when it comes to suggestions for alternative
presentations of uncertainty. Indeed, the abstract
favors NB and interval estimation, whereas the article
itself suggests a more varied approach. For decisions
involving 2 treatment alternatives, they appear to sug-
gest the use of the (incremental) cost-effectiveness
plane with the addition of “summary measures,”
although the exact nature of the summary measures
are less clear—intervals for the ICER are dismissed,
due to the issues surrounding ratios, but intervals for
the incremental net benefit (INB) and EVPI are both
suggested. We would suggest the use of CEACs and the
CEAF. For decisions involving more than 2 treatment
alternatives, the authors dismiss the use of the cost-
effectiveness plane as “ambiguous”; intervals for INB
should also be dismissed as an alternative in this situ-
ation as they are unable to provide a simultaneous
assessment of the uncertainty in the decision when
there are more than 2 treatment alternatives. This
leaves the expected value of perfect information as the
sole summary measure of decision uncertainty. We do
not dispute that this is the appropriate measure of
decision uncertainty. However, we argue for the pre-
sentation of CEACs and the CEAF for pragmatic rea-
sons. They provide a useful bridge to the EVPI for
policy makers unused to seeing this information pre-
sented. We acknowledge that there is still a place for
education regarding their interpretation but feel that if
they are ill understood, then the presentation of EVPI
alone is unlikely to be well received by policy makers
and journals. In addition, the EVPI simply identifies
the maximum worth of further research to reduce
 
For Evaluation Only.
Copyright (c) by Foxit Software Company, 2004 - 2007
Edited by Foxit PDF Editor
uncertainty. VOI analysis cannot and does not resolve
the decision regarding the collection of further
research without information regarding the costs of
research or the “informativeness” of specific research.
This information is presented via an analysis of the
expected value of sample information (EVSI) and the
expected net benefits of sampling (ENBS). At the pre-
sent time, neither of these concepts is likely to be pre-
sented in a clinical paper. The best we can do is
encourage policy makers to think about uncertainty
and its consequences, and we believe that the CEAC
and CEAF do this.
CONCLUDING STATEMENT
In Scottish courts, the jury in a criminal case has 
another option besides the usual “guilty” or “not
guilty” verdicts. Case “not proven” reflects those situa-
tions where the jury feels unable to convict a defendant
but nevertheless considers the verdict of not guilty too
lenient. The authors’ charges against the usefulness of
CEACs curves have some merit; therefore, a “not
guilty” verdict is inappropriate. However, we hope we
have convinced you, the jury, that there remains a use
for CEACs and that the case against them is, therefore,
“not proven.”
REFERENCES
1. Groot Koerkamp B, Hunink MG, Stijnen T, Hammitt JK, Kuntz
KM, Weinstein MC. Limitations of acceptability curves for present-
ing uncertainty in cost-effectiveness analysis. Med Decis Making.
2007;27:101–111.
2. Severens JL, Brunenberg DEM, Fenwick E, O’Brien B, Joore
MA. Cost-effectiveness acceptability curves and a reluctance to
lose. Pharmacoeconomics. 2005;23:1207–14.
3. Cooper NJ, Sutton AJ, Abrams KR, Turner D, Wailoo A.
Comprehensive decision analytical modelling in economic evalu-
ation: a Bayesian approach. Health Econ. 2004;13:203–26.
4. Fenwick E, Palmer S, Claxton K, Sculpher M, Abrams KR, Sutton
AJ. An iterative Bayesian approach to health technology assess-
ment: application to a policy of pre-operative optimization for
patients undergoing major elective surgery. Med Decis Making.
2006;26:480–497.
5. Fenwick E, Claxton K, Sculpher M. Representing uncertainty:
the role of cost-effectiveness acceptability curves. Health Econ.
2001;10:779–87.
 
For Evaluation Only.
Copyright (c) by Foxit Software Company, 2004 - 2007
Edited by Foxit PDF Editor


Cost-effectiveness acceptability curves in the dock: case not proven?

Elisabeth Fenwick

Andrew Briggs

Crossref

Cost-Effectiveness Acceptability Curves in the Dock

Enlighten

      Fenwick, E. and Briggs, A.H. (2007) Cost-effectiveness acceptability curves in the dock: case not proven? Medical Decision Making 27(2):pp. 93-95.        http://eprints.gla.ac.uk/4167/  Deposited on: 12 May 2008   Glasgow ePrints Service http://eprints.gla.ac.uk Cost-Effectiveness Acceptability Curves in the Dock: Case Not Proven?Elisabeth Fenwick, PhD, Andrew Briggs, DPhilThe article in this issue by Groot Koerkamp and oth-ers1 discusses the limitations of cost-effectivenessacceptability curves (CEACs) for presenting uncer-tainty in cost-effectiveness analysis to policy makersand urges us to “rethink their use in communicatinguncertainty.”1 We agree with the authors that uncer-tainty is important for policy makers and that theappropriate method to evaluate uncertainty is throughan assessment of both the probability of making anerror (“not selecting the ‘true’ preferred alternative”)and the consequences associated with making anerror. A Bayesian value of information (VOI) analysisprovides this assessment of uncertainty. The questionremains about what is/are the appropriate means topresent information about uncertainty to policy mak-ers. We take this opportunity to review the case againstCEACs presented within the article and to defend theuse and usefulness of CEACs for policy making inhealth care.BRING FORTH THE CHARGESThe authors state that CEACs are “unable to dis-tinguish dramatically different joint distributions ofincremental cost and effect” and that this limits theirusefulness for policy making.1 They argue that theuse of CEACs to represent uncertainty restricts theability to synthesize evidence on cost-effectivenessfrom other sources. In addition, they suggest that theCEAC does not allow for integration of risk attitude,provides no insight into the assessment of the needfor further research, and may mislead policy makersregarding the appropriate treatment choice.THE CASE FOR THE DEFENSEWe accept that CEACs are insensitive to any changein the joint distribution of incremental cost and effectin the northwest (NW) or southeast (SE) quadrants ofthe cost-effectiveness plane. This is not, however, aproduct of the cost-effectiveness acceptability curvebut rather of cost-effectiveness analysis itself, whichconsiders all points in each of these quadrants to beequivalent in cost-effectiveness terms—that is, domi-nant (SE) or dominated (NW). If information aboutdominance is considered important to policy makers,then the CEAC could be adapted very easily to provideit (see Severens and others2). We also accept thatCEACs are insensitive to radial shifts in the joint dis-tribution of incremental cost and effect in the north-east or southwest quadrant of the cost-effectivenessplane. This again is a consequence of cost-effectivenessanalysis. Points lying on the same ray through the ori-gin have the same incremental cost-effectiveness ratio(ICER) and thus, for any value of the maximum accept-able incremental cost-effectiveness ratio, will have thesame outcome regarding cost-effectiveness (i.e., eithercost-effective or not). As such, any radial shift of theincremental joint distribution, either toward or awayfrom the origin, represents identical uncertaintyregarding cost-effectiveness and will result in identicalCEACs.With regard to the integration of other evidenceand opinions, we agree with the authors that “aquantitative analysis is rarely the only or final wordon a decision.”1 Within a Bayesian framework, allavailable evidence should be included and consid-ered within the decision context. However, we dis-pute the use of a summarized measure (“quantitativestatement”) of uncertainty as a means of weightingresults for a synthesis of the evidence from otherFrom Public Health and Health Policy, University of Glasgow, Glasgow,Scotland.Address correspondence to Elisabeth Fenwick, Public Health and HealthPolicy, University of Glasgow, 1 Lilybank Gardens, Glasgow, G12 8RZ,Scotland; telephone: 0141 330 5615; fax: 0141 330 5018; e-mail:e.fenwick@clinmed.gla.ac.uk. For Evaluation Only.Copyright (c) by Foxit Software Company, 2004 - 2007Edited by Foxit PDF Editorsources. In our view, this type of integration of evi-dence should be done formally and should involvethe individual components of the decision (e.g.,parameters, costs, and effects) rather than the mea-sure of cost-effectiveness (see Cooper and others3and Fenwick and others4 for examples).The misinterpretation of a high probability that atreatment alternative is cost-effective with medicalurgency or importance is equivalent to confusingstatistical significance and clinical importance. Thisis purely an education issue and is not specific tothe use of CEACs, hypothesis testing, or confidenceintervals.In addition, we dispute the authors’ suggestionsthat the CEAC does not allow for integration of riskattitude, provides no insight into the assessment of the need for further research, and may misleadpolicy makers regarding the appropriate treatmentchoice. Much of the article centers on the argumentbetween the use of ICERs and net benefit (NB) foreconomic evaluation and confuses this with issues ofpresenting uncertainty. It was the difficulties inherentwith constructing confidence intervals (CIs) for ICERsthat led ultimately to the production of the CEAC andto its proliferation within the medical literature. Theabsence of these difficulties for NB promotes thepresentation of CI, but there is no reason that CEACscannot be presented for an analysis involving NB. Contrary to the authors’ statement that “CEACsfail to capture the importance of uncertainty,”1 webelieve that CEACs encourage policy makers tothink about decision uncertainty and the conse-quences of such uncertainty for policy making,allowing them to cast off the shackles of confidenceintervals and hypothesis testing. The productionand presentation of intervals for NB, however, arelikely to encourage classical thinking regarding cost-effectiveness (i.e., if the interval excludes zero, thenthe intervention is cost-effective).We accept that the CEAC does not present the pol-icy maker with information concerning the appropri-ate treatment choice (this is given by the expectedutilities), nor does it present information regardingthe consequences of an incorrect decision, but it doespresent policy makers with graphical informationregarding the probability that a treatment alternativeis cost-effective over a range of values for the maxi-mum acceptable ICER. In turn, the cost-effectivenessacceptability frontier (CEAF) presents the probabilitythat the treatment choice made on the basis ofexpected values is correct. As such, the complementof the CEAF provides a graphical representation ofthe probability of making an error, which is a majorcomponent of the expected value of perfect informa-tion (EVPI).5 Thus, there is a link between the CEAFand the EVPI: over the range where the CEAF isfalling, the EVPI must be increasing as both the errorprobability and the maximum acceptable ratio (used inthe calculation of the consequences of error) are increas-ing. Over the range where the CEAF is increasing, thedirection of the EVPI will depend on the interactionbetween the rate of change in the uncertainty and therate of change in the consequences. We have foundthat for ranges where the CEAF is flattening off towarda maximum (uncertainty is falling slightly), the changein the consequences will dominate, and EVPI willtend to increase.THE WAY FORWARDHaving started boldly by proclaiming the issues andlimitations associated with the CEAC, the authors areless clear when it comes to suggestions for alternativepresentations of uncertainty. Indeed, the abstractfavors NB and interval estimation, whereas the articleitself suggests a more varied approach. For decisionsinvolving 2 treatment alternatives, they appear to sug-gest the use of the (incremental) cost-effectivenessplane with the addition of “summary measures,”although the exact nature of the summary measuresare less clear—intervals for the ICER are dismissed,due to the issues surrounding ratios, but intervals forthe incremental net benefit (INB) and EVPI are bothsuggested. We would suggest the use of CEACs and theCEAF. For decisions involving more than 2 treatmentalternatives, the authors dismiss the use of the cost-effectiveness plane as “ambiguous”; intervals for INBshould also be dismissed as an alternative in this situ-ation as they are unable to provide a simultaneousassessment of the uncertainty in the decision whenthere are more than 2 treatment alternatives. Thisleaves the expected value of perfect information as thesole summary measure of decision uncertainty. We donot dispute that this is the appropriate measure ofdecision uncertainty. However, we argue for the pre-sentation of CEACs and the CEAF for pragmatic rea-sons. They provide a useful bridge to the EVPI forpolicy makers unused to seeing this information pre-sented. We acknowledge that there is still a place foreducation regarding their interpretation but feel that ifthey are ill understood, then the presentation of EVPIalone is unlikely to be well received by policy makersand journals. In addition, the EVPI simply identifiesthe maximum worth of further research to reduce For Evaluation Only.Copyright (c) by Foxit Software Company, 2004 - 2007Edited by Foxit PDF Editoruncertainty. VOI analysis cannot and does not resolvethe decision regarding the collection of furtherresearch without information regarding the costs ofresearch or the “informativeness” of specific research.This information is presented via an analysis of theexpected value of sample information (EVSI) and theexpected net benefits of sampling (ENBS). At the pre-sent time, neither of these concepts is likely to be pre-sented in a clinical paper. The best we can do isencourage policy makers to think about uncertaintyand its consequences, and we believe that the CEACand CEAF do this.CONCLUDING STATEMENTIn Scottish courts, the jury in a criminal case has another option besides the usual “guilty” or “notguilty” verdicts. Case “not proven” reflects those situa-tions where the jury feels unable to convict a defendantbut nevertheless considers the verdict of not guilty toolenient. The authors’ charges against the usefulness ofCEACs curves have some merit; therefore, a “notguilty” verdict is inappropriate. However, we hope wehave convinced you, the jury, that there remains a usefor CEACs and that the case against them is, therefore,“not proven.”REFERENCES1. Groot Koerkamp B, Hunink MG, Stijnen T, Hammitt JK, KuntzKM, Weinstein MC. Limitations of acceptability curves for present-ing uncertainty in cost-effectiveness analysis. Med Decis Making.2007;27:101–111.2. Severens JL, Brunenberg DEM, Fenwick E, O’Brien B, JooreMA. Cost-effectiveness acceptability curves and a reluctance tolose. Pharmacoeconomics. 2005;23:1207–14.3. Cooper NJ, Sutton AJ, Abrams KR, Turner D, Wailoo A.Comprehensive decision analytical modelling in economic evalu-ation: a Bayesian approach. Health Econ. 2004;13:203–26.4. Fenwick E, Palmer S, Claxton K, Sculpher M, Abrams KR, SuttonAJ. An iterative Bayesian approach to health technology assess-ment: application to a policy of pre-operative optimization forpatients undergoing major elective surgery. Med Decis Making.2006;26:480–497.5. Fenwick E, Claxton K, Sculpher M. Representing uncertainty:the role of cost-effectiveness acceptability curves. Health Econ.2001;10:779–87. For Evaluation Only.Copyright (c) by Foxit Software Company, 2004 - 2007Edited by Foxit PDF Editor

An iterative Bayesian approach to health technology assessment: application to a policy of pre-operative optimization for patients undergoing major elective surgery. Med Decis Making.

Comprehensive decision analytical modelling in economic evaluation: a Bayesian approach. Health Econ.

Cost-effectiveness acceptability curves and a reluctance to lose.

Limitations of acceptability curves for presenting uncertainty in cost-effectiveness analysis. Med Decis Making.

Representing uncertainty: the role of cost-effectiveness acceptability curves. Health Econ. 2001;10:779–87. For Evaluation Only. Copyright (c) by Foxit Software Company,

http://eprints.gla.ac.uk/4167/1/Fenwick4167.pdf

Cost-effectiveness acceptability curves in the dock: case not proven?

Abstract

Similar works

Full text

Available Versions

Enlighten: Publications

Crossref

Enlighten