The use of video vignettes to measure health worker knowledge. Evidence from Burkina Faso

The quality of care is a crucial determinant of good health outcomes, but is difficult to measure. Survey vignettes are a standard approach to measuring medical knowledge among health care providers. Given that written vignettes or knowledge tests may be too removed from clinical practice, particularly where “learning by doing” may be an important form of training, we developed a new type of provider vignette. It uses videos presenting a patient visiting the clinic with maternal/early childhood symptoms. We tested these video vignettes with current and future (students) health professionals in Burkina Faso. Participants indicated that the cases used were interesting, understandable and common. Their performance was consistent with expectations. Participants with greater training (medical doctors vs. nurses and midwives) and experience (health professionals vs. students) performed better. The video vignettes can easily be embedded in computers, tablets and smart phones; they are a convenient tool to measure provider knowledge; and they are cost-effective instruction and testing tools

Does enrollment status in community-based insurance lead to poorer quality of care? Evidence from Burkina Faso

Introduction: In 2004, a community-based health insurance (CBI) scheme was introduced in Nouna health district, Burkina Faso, with the objective of improving financial access to high quality health services. We investigate the role of CBI enrollment in the quality of care provided at primary-care facilities in Nouna district, and measure differences in objective and perceived quality of care and patient satisfaction between enrolled and non-enrolled populations who visit the facilities. Methods: We interviewed a systematic random sample of 398 patients after their visit to one of the thirteen primary-care facilities contracted with the scheme; 34% (n = 135) of the patients were currently enrolled in the CBI scheme. We assessed objective quality of care as consultation, diagnostic and counselling tasks performed by providers during outpatient visits, perceived quality of care as patient evaluations of the structures and processes of service delivery, and overall patient satisfaction. Two-sample t-tests were performed for group comparison and ordinal logistic regression (OLR) analysis was used to estimate the association between CBI enrollment and overall patient satisfaction. Results: Objective quality of care evaluations show that CBI enrollees received substantially less comprehensive care for outpatient services than non-enrollees. In contrast, CBI enrollment was positively associated with overall patient satisfaction (aOR = 1.51, p = 0.014), controlling for potential confounders such as patient socio-economic status, illness symptoms, history of illness and characteristics of care received. Conclusions: CBI patients perceived better quality of care, while objectively receiving worse quality of care, compared to patients who were not enrolled in CBI. Systematic differences in quality of care expectations between CBI enrollees and non-enrollees may explain this finding. One factor influencing quality of care may be the type of provider payment used by the CBI scheme, which has been identified as a leading factor in reducing provider motivation to deliver high quality care to CBI enrollees in previous studies. Based on this study, it is unlikely that perceived quality of care and patient satisfaction explain the low CBI enrollment rates in this community

Health Worker Preferences for Community-Based Health Insurance Payment Mechanisms: A Discrete Choice Experiment

Background: In 2004, a community-based health insurance scheme (CBI) was introduced in Nouna health district, Burkina Faso. Since its inception, coverage has remained low and dropout rates high. One important reason for low coverage and high dropout is that health workers do not support the CBI scheme because they are dissatisfied with the provider payment mechanism of the CBI. Methods: A discrete choice experiment (DCE) was used to examine CBI provider payment attributes that influence health workers’ stated preferences for payment mechanisms. The DCE was conducted among 176 health workers employed at one of the 34 primary care facilities or the district hospital in Nouna health district. Conditional logit models with main effects and interactions terms were used for analysis. Results: Reimbursement of service fees (adjusted odds ratio (aOR) 1.49, p < 0.001) and CBI contributions for medical supplies and equipment (aOR 1.47, p < 0.001) had the strongest effect on whether the health workers chose a given provider payment mechanism. The odds of selecting a payment mechanism decreased significantly if the mechanism included (i) results-based financing (RBF) payments made through the local health management team (instead of directly to the health workers (aOR 0.86, p < 0.001)) or (ii) RBF payments based on CBI coverage achieved in the health worker’s facility relative to the coverage achieved at other facilities (instead of payments based on the numbers of individuals or households enrolled at the health worker’s facility (aOR 0.86, p < 0.001)). Conclusions: Provider payment mechanisms can crucially determine CBI performance. Based on the results from this DCE, revised CBI payment mechanisms were introduced in Nouna health district in January 2011, taking into consideration health worker preferences on how they are paid

Addressing Health Workforce Distribution Concerns: A Discrete Choice Experiment to Develop Rural Retention Strategies in Cameroon

Background: Nearly every nation in the world faces shortages of health workers in remote areas. Cameroon is no exception to this. The Ministry of Public Health (MoPH) is currently considering several rural retention strategies to motivate qualified health personnel to practice in remote rural areas. Methods: To better calibrate these mechanisms and to develop evidence-based retention strategies that are attractive and motivating to health workers, a Discrete Choice Experiment (DCE) was conducted to examine what job attributes are most attractive and important to health workers when considering postings in remote areas. The study was carried out between July and August 2012 among 351 medical students, nursing students and health workers in Cameroon. Mixed logit models were used to analyze the data. Results: Among medical and nursing students a rural retention bonus of 75% of base salary (aOR= 8.27, 95% CI: 5.28-12.96, P < 0.001) and improved health facility infrastructure (aOR= 3.54, 95% CI: 2.73-4.58) respectively were the attributes with the largest effect sizes. Among medical doctors and nurse aides, a rural retention bonus of 75% of base salary was the attribute with the largest effect size (medical doctors aOR= 5.60, 95% CI: 4.12-7.61, P < 0.001; nurse aides aOR= 4.29, 95% CI: 3.11-5.93, P < 0.001). On the other hand, improved health facility infrastructure (aOR= 3.56, 95% CI: 2.75-4.60, P < 0.001), was the attribute with the largest effect size among the state registered nurses surveyed. Willingness-to-Pay (WTP) estimates were generated for each health worker cadre for all the attributes. Preference impact measurements were also estimated to identify combination of incentives that health workers would find most attractive. Conclusion: Based on these findings, the study recommends the introduction of a system of substantial monetary bonuses for rural service along with ensuring adequate and functional equipment and uninterrupted supplies. By focusing on the analysis of locally relevant, actionable incentives, generated through the involvement of policy- makers at the design stage, this study provides an example of research directly linked to policy action to address a vitally important issue in global health

Why did performance-based financing in Burkina Faso fail to achieve the intended equity effects? A process tracing study

In recent years, performance-based financing (PBF) has attracted attention as a means of reforming provider payment mechanisms in low- and middle-income countries. Particularly in combination with demand-side interventions, PBF has been assumed to benefit also the most vulnerable and disadvantaged groups. However, impact evaluations have often found this not to be the case. In Burkina Faso, PBF was coupled with specific equity measures to enhance healthcare utilization among the ultra-poor, but failed to produce the expected effects. Our study used the process tracing methodology to unravel the reasons for the lack of impact produced by the equity measures. We relied on published evidence, secondary data analysis, and findings from a qualitative study to support or invalidate the hypothesized causal mechanism, that is the reconstructed theory of change of the equity measures. Our findings show how various contextual, design, and implementation challenges hindered the causal mechanism from unfolding as planned. These included issues with the identification and exemption of the ultra-poor on the demand side, and with financial issues and considerations on the supply side. In broader terms, our findings underline the difficulty in improving access to care for the ultra-poor, given the multifaceted and complex nature of barriers to care the most vulnerable face. From a methodological point of view, our study demonstrates the value and applicability of process tracing in complementing other forms of evaluation for complex interventions in global health

Design of an impact evaluation using a mixed methods model – an explanatory assessment of the effects of results-based financing mechanisms on maternal healthcare services in Malawi

Background: In this article we present a study design to evaluate the causal impact of providing supply-side performance-based financing incentives in combination with a demand-side cash transfer component on equitable access to and quality of maternal and neonatal healthcare services. This intervention is introduced to selected emergency obstetric care facilities and catchment area populations in four districts in Malawi. We here describe and discuss our study protocol with regard to the research aims, the local implementation context, and our rationale for selecting a mixed methods explanatory design with a quasi-experimental quantitative component. Design: The quantitative research component consists of a controlled pre- and post-test design with multiple post-test measurements. This allows us to quantitatively measure ‘equitable access to healthcare services’ at the community level and ‘healthcare quality’ at the health facility level. Guided by a theoretical framework of causal relationships, we determined a number of input, process, and output indicators to evaluate both intended and unintended effects of the intervention. Overall causal impact estimates will result from a difference-in-difference analysis comparing selected indicators across intervention and control facilities/catchment populations over time. To further explain heterogeneity of quantitatively observed effects and to understand the experiential dimensions of financial incentives on clients and providers, we designed a qualitative component in line with the overall explanatory mixed methods approach. This component consists of in-depth interviews and focus group discussions with providers, service user, non-users, and policy stakeholders. In this explanatory design comprehensive understanding of expected and unexpected effects of the intervention on both access and quality will emerge through careful triangulation at two levels: across multiple quantitative elements and across quantitative and qualitative elements. Discussion Combining a traditional quasi-experimental controlled pre- and post-test design with an explanatory mixed methods model permits an additional assessment of organizational and behavioral changes affecting complex processes. Through this impact evaluation approach, our design will not only create robust evidence measures for the outcome of interest, but also generate insights on how and why the investigated interventions produce certain intended and unintended effects and allows for a more in-depth evaluation approach

How much does community-based targeting of the ultra-poor in the health sector cost? Novel evidence from Burkina Faso

Background: Targeting efforts aimed at increasing access to care for the poorest by reducing to a minimum or completely eliminating payments at point of use are increasingly being adopted across low and middle income countries, within the framework of Universal Health Coverage policies. No evidence, however, is available on the real cost of designing and implementing these efforts. Our study aimed to fill this gap in knowledge through the systematic assessment of both the financial and economic costs associated with designing and implementing a pro-poor community-based targeting intervention across eight districts in rural Burkina Faso. Methods: We conducted a partial retrospective economic evaluation (i.e. estimating costs, but not benefits) associated with the abovementioned targeting intervention. We adopted a health system perspective, including all costs incurred by the government and its development partners as well as costs incurred by the community when working as volunteers on behalf of government structures. To trace both financial and economic costs, we combined Activity-Based Costing with Resource Consumption Accounting. To this purpose, we consulted and extracted information from all relevant design/implementation documents and conducted additional key informant structured interviews to assess the resource consumption that was not valued in the documents. Results: For the entire community-based targeting intervention, we estimated a financial cost of USD 587,510 and an economic cost of USD 1,213,447. The difference was driven primarily by the value of the time contributed by the community. Communities carried the main economic burden. With a total of 102,609 ultra-poor identified, the financial cost and the economic cost per ultra-poor person were respectively USD 5,73 and USD 11,83. Conclusion: The study is first of its kind to accurately trace the financial and economic costs of a community-based targeting intervention aiming to identify the ultra-poor. The financial costs amounted to USD 5,73 and the economic costs to USD 11,83 per ultra-poor person identified. The financial costs of almost USD 6 represents 21% of the per capita government expenditure on health

Unraveling PBF effects beyond impact evaluation: results from a qualitative study in Cameroon

Introduction Performance-based financing (PBF) has acquired increased prominence as a means of reforming health system purchasing structures in low-income and middle-income countries. A number of impact evaluations have noted that PBF often produces mixed and heterogeneous effects. Still, little systematic effort has been channelled towards understanding what causes such heterogeneity, including looking more closely at implementation processes. Methods Our qualitative study aimed at closing this gap in knowledge by attempting to unpack the mixed and heterogeneous effects detected by the PBF impact evaluation in Cameroon to inform further implementation as the country scales up the PBF approach. We collected data at all levels of the health system (national, district, facility) and at the community level, using a mixture of in-depth interviews and focus group discussions. We combined deductive and inductive analytical techniques and applied analyst triangulation. Results Our findings indicate that heterogeneity in effects across facilities could be explained by preexisting infrastructural weaknesses coupled with rigid administrative processes and implementation challenges, while heterogeneity across indicators could be explained by providers' practices, privileging services where demand-side barriers were less substantive. Conclusion In light of the country's commitment to scaling up PBF, it follows that substantial efforts (particularly entrusting facilities with more financial autonomy) should be made to overcome infrastructural and demand-side barriers and to smooth implementation processes, thus, enabling healthcare providers to use PBF resources and management models to a fuller potential.sch_iih3pub5290pub

No impact of performance-based financing on the availability of essential medicines in Burkina Faso: A mixed-methods study

Access to safe, effective, and affordable essential medicines (EM) is critical to quality health services and as such has played a key role in innovative health system strengthening approaches such as Performance-based Financing (PBF). Available literature indicates that PBF can improve EM availability, but has not done so consistently in the past. Qualitative explorations of the reasons are yet scarce. We contribute to expanding the literature by estimating the impact of PBF on EM availability and stockout in Burkina Faso and investigating mechanisms of and barriers to change. The study used an explanatory mixed methods design. The quantitative study component followed a quasi-experimental design (difference-in-differences), comparing how EM availability and stockout had changed three years after implementation in 12 PBF and in 12 control districts. Qualitative data was collected from purposely selected policy and implementation stakeholders at all levels of the health system and community, using in-depth interviews and focus group discussions, and explored using deductive coding and thematic analysis. We found no impact of PBF on EM availability and stockouts in the quantitative data. Qualitative narratives converge in that EM supply had increased as a result of PBF, albeit not fully satisfactorily and sustainably so. Reasons include persisting contextual challenges, most importantly a public medicine procurement monopoly; design challenges, specifically a disconnect and disbalance in incentive levels between service provision and service quality indicators; implementation challenges including payment delays, issues around performance verification, and insufficient implementation of activities to strengthen stock management skills; and concurrently implemented policies, most importantly a national user fee exemption for children and pregnant women half way through the impact evaluation period. The case of PBF and EM availability in Burkina Faso illustrates the difficulty of incentivizing and effecting holistic change in EM availability in the presence of strong contextual constraints and powerful concurrent policies.</jats:p

Responding to policy makers' evaluation needs: combining experimental and quasi-experimental approaches to estimate the impact of performance based financing in Burkina Faso.

BACKGROUND: The last two decades have seen a growing recognition of the need to expand the impact evaluation toolbox from an exclusive focus on randomized controlled trials to including quasi-experimental approaches. This appears to be particularly relevant when evaluation complex health interventions embedded in real-life settings often characterized by multiple research interests, limited researcher control, concurrently implemented policies and interventions, and other internal validity-threatening circumstances. To date, however, most studies described in the literature have employed either an exclusive experimental or an exclusive quasi-experimental approach. METHODS: This paper presents the case of a study design exploiting the respective advantages of both approaches by combining experimental and quasi-experimental elements to evaluate the impact of a Performance-Based Financing (PBF) intervention in Burkina Faso. Specifically, the study employed a quasi-experimental design (pretest-posttest with comparison) with a nested experimental component (randomized controlled trial). A difference-in-differences approach was used as the main analytical strategy. DISCUSSION: We aim to illustrate a way to reconcile scientific and pragmatic concerns to generate policy-relevant evidence on the intervention's impact, which is methodologically rigorous in its identification strategy but also considerate of the context within which the intervention took place. In particular, we highlight how we formulated our research questions, ultimately leading our design choices, on the basis of the knowledge needs expressed by the policy and implementing stakeholders. We discuss methodological weaknesses of the design arising from contextual constraints and the accommodation of various interests, and how we worked ex-post to address them to the best extent possible to ensure maximal accuracy and credibility of our findings. We hope that our case may be inspirational for other researchers wishing to undertake research in settings where field circumstances do not appear to be ideal for an impact evaluation. TRIAL REGISTRATION: Registered with RIDIE (RIDIE-STUDY-ID- 54412a964bce8 ) on 10/17/2014