Application of system thinking concepts in health system strengthening in low-income settings: a proposed conceptual framework for the evaluation of a complex health system intervention: the case of the BHOMA intervention in Zambia.

BACKGROUND: The current drive to strengthen health systems provides an opportunity to develop new strategies that will enable countries to achieve targets for millennium development goals. In this paper, we present a proposed framework for evaluating a new health system strengthening intervention in Zambia known as Better Health Outcomes through Mentoring and Assessment. APPROACH: We briefly describe the intervention design and focus on the proposed evaluation approach through the lens of systems thinking. DISCUSSION: In this paper, we present a proposed framework to evaluate a complex health system intervention applying systems thinking concepts. We hope that lessons learnt from this process will help to adapt the intervention and limit unintended negative consequences while promoting positive effects. Emphasis will be paid to interaction and interdependence between health system building blocks, context and the community

Systems thinking in practice: the current status of the six WHO building blocks for health system strengthening in three BHOMA intervention districts of Zambia: a baseline qualitative study.

BACKGROUND: The primary bottleneck to achieving the MDGs in low-income countries is health systems that are too fragile to deliver the volume and quality of services to those in need. Strong and effective health systems are increasingly considered a prerequisite to reducing the disease burden and to achieving the health MDGs. Zambia is one of the countries that are lagging behind in achieving millennium development targets. Several barriers have been identified as hindering the progress towards health related millennium development goals. Designing an intervention that addresses these barriers was crucial and so the Better Health Outcomes through Mentorship (BHOMA) project was designed to address the challenges in the Zambia's MOH using a system wide approach. We applied systems thinking approach to describe the baseline status of the Six WHO building blocks for health system strengthening. METHODS: A qualitative study was conducted looking at the status of the Six WHO building blocks for health systems strengthening in three BHOMA districts. We conducted Focus group discussions with community members and In-depth Interviews with key informants. Data was analyzed using Nvivo version 9. RESULTS: The study showed that building block specific weaknesses had cross cutting effect in other health system building blocks which is an essential element of systems thinking. Challenges noted in service delivery were linked to human resources, medical supplies, information flow, governance and finance building blocks either directly or indirectly. Several barriers were identified as hindering access to health services by the local communities. These included supply side barriers: Shortage of qualified health workers, bad staff attitude, poor relationships between community and health staff, long waiting time, confidentiality and the gender of health workers. Demand side barriers: Long distance to health facility, cost of transport and cultural practices. Participating communities seemed to lack the capacity to hold health workers accountable for the drugs and services. CONCLUSION: The study has shown that building block specific weaknesses had cross cutting effect in other health system building blocks. These linkages emphasised the need to use system wide approaches in assessing the performance of health system strengthening interventions

Application of balanced scorecard in the evaluation of a complex health system intervention: 12 months post intervention findings from the BHOMA intervention: a cluster randomised trial in Zambia.

INTRODUCTION: In many low income countries, the delivery of quality health services is hampered by health system-wide barriers which are often interlinked, however empirical evidence on how to assess the level and scope of these barriers is scarce. A balanced scorecard is a tool that allows for wider analysis of domains that are deemed important in achieving the overall vision of the health system. We present the quantitative results of the 12 months follow-up study applying the balanced scorecard approach in the BHOMA intervention with the aim of demonstrating the utility of the balanced scorecard in evaluating multiple building blocks in a trial setting. METHODS: The BHOMA is a cluster randomised trial that aims to strengthen the health system in three rural districts in Zambia. The intervention aims to improve clinical care quality by implementing practical tools that establish clear clinical care standards through intensive clinic implementations. This paper reports the findings of the follow-up health facility survey that was conducted after 12 months of intervention implementation. Comparisons were made between those facilities in the intervention and control sites. STATA version 12 was used for analysis. RESULTS: The study found significant mean differences between intervention(I) and control (C) sites in the following domains: Training domain (Mean I:C; 87.5.vs 61.1, mean difference 23.3, p = 0.031), adult clinical observation domain (mean I:C; 73.3 vs.58.0, mean difference 10.9, p = 0.02 ) and health information domain (mean I:C; 63.6 vs.56.1, mean difference 6.8, p = 0.01. There was no gender differences in adult service satisfaction. Governance and motivation scores did not differ between control and intervention sites. CONCLUSION: This study demonstrates the utility of the balanced scorecard in assessing multiple elements of the health system. Using system wide approaches and triangulating data collection methods seems to be key to successful evaluation of such complex health intervention. TRIAL NUMBER: ClinicalTrials.gov NCT01942278

Measuring governance at health facility level: developing and validation of simple governance tool in Zambia.

BACKGROUND: Governance has been cited as a key determinant of economic growth, social advancement and overall development. Achievement of millennium development goals is partly dependant on governance practices. In 2007, Health Systems 20/20 conducted an Internet-based survey on the practice of good governance. The survey posed a set of good practices related to health governance and asked respondents to indicate whether their experience confirmed or disconfirmed those practices. We applied the 17 governance statements in rural health facilities of Zambia. The aim was to establish whether the statements were reliable and valid for assessing governance practices at primary care level. METHODS: Both quantitative and qualitative methods were used. We first applied the governance statements developed by the health system 20/20 and then conducted focus group discussion and In-depth interviews to explore some elements of governance including accountability and community participation. The target respondents were the health facility management team and community members. The sample size include 42 health facilities. Data was analyzed using SPSS version 17 and Nvivo version 9. RESULTS: The 95% one-sided confidence interval for Cronbach's alpha was between 0.69 and 0.74 for the 16 items.The mean score for most of the items was above 3. Factor analysis yielded five principle components: Transparency, community participation, Intelligence & vision, Accountability and Regulation & oversight. Most of the items (6) clustered around the transparency latent factor. Chongwe district performed poorly in overall mean governance score and across the five domains of governance. The overall scores in Chongwe ranged between 51 and 94% with the mean of 80%. Kafue and Luangwa districts had similar overall mean governance scores (88%). Community participation was generally low. Generally, it was noted that community members lacked capacity to hold health workers accountable for drugs and medical supplies. CONCLUSIONS: The study successfully validated and applied the new tool for evaluating health system governance at health facility level. The results have shown that it is feasible to measure governance practices at health facility level and that the adapted tool is fairly reliable with the 95% one-sided confidence interval for Cronbach's alpha laying between 0.69 and 0.74 for the 16 items. Caution should be taken when interpreting overall scores as they tended to mask domain specific variations

Application of systems thinking: 12-month postintervention evaluation of a complex health system intervention in Zambia: the case of the BHOMA.

RATIONALE, AIMS AND OBJECTIVES: Strong health systems are said to be paramount to achieving effective and equitable health care. The World Health Organization has been advocating for using system-wide approaches such as 'systems thinking' to guide intervention design and evaluation. In this paper we report the system-wide effects of a complex health system intervention in Zambia known as Better Health Outcome through Mentorship and Assessment (BHOMA) that aimed to improve service quality. METHODS: We conducted a qualitative study in three target districts. We used a systems thinking conceptual framework to guide the analysis focusing on intended and unintended consequences of the intervention. NVivo version 10 was used for data analysis. RESULTS: The addressed community responded positively to the BHOMA intervention. The indications were that in the short term there was increased demand for services but the health worker capacity was not severely affected. This means that the prediction that service demand would increase with implementation of BHOMA was correct and the workload also increased, but the help of clinic lay supporters meant that some of the work of clinicians was transferred to these lay workers. However, from a systems perspective, unintended consequences also occurred during the implementation of the BHOMA. CONCLUSIONS: We applied an innovative approach to evaluate a complex intervention in low-income settings, exploring empirically how systems thinking can be applied in the context of health system strengthening. Although the intervention had some positive outcomes by employing system-wide approaches, we also noted unintended consequences

District balanced Scorecard stratified by domain.

<p>This figure shows district scores stratified by domain. The domain comprised six indices, each made up from an aggregate of indicators. Across the three study districts the basic infrastructure score was similar at 76%. Basic equipment and laboratory capacity scores showed major variation with Kafue and Luangwa having lower scores when compared to Chongwe. For basic equipment Luangwa scored lowest (65%), followed by Kafue (67%). Chongwe had the highest basic equipment score of 84%, and the laboratory capacity score was lowest in Kafue (63%) and highest in Chongwe (77%). Infection control scores were highest in Luangwa (90%) and lowest in Kafue (80%).</p

Baseline Performance Stratified by Residence in the Six Health System Domains.

*<p> <i>The mean difference is significant at p<0.05, using ANOVA.</i></p

Sample profile for the baseline study in the three BHOMA districts.

<p>Sample profile for the baseline study in the three BHOMA districts.</p

Baseline demographic characteristics of the health facilities in the BHOMA study.

<p>Baseline demographic characteristics of the health facilities in the BHOMA study.</p

Baseline District Performance in Six Health System Domains.

*<p> <i>The mean difference is significant at p<0.05, using ANOVA.</i></p