Economic evaluation and decision making for quality improvement in complex community health systems

Community health is a fundamental part of many healthcare systems and is widely advocated as a means to increase access to and coverage of health services, yet the quality of care in large-scale community health programmes is mixed. Quality improvement (QI) approaches are now being tested in community settings and there is limited evidence that integrating QI approaches can underpin success of community programmes. However, how best to measure that success and the cost and value thereof to the different decision makers in complex community health systems is not yet known. This thesis provides the first economic evaluation of QI in community health systems, linking this to an exploration of decision making that includes an assessment of how economic evidence like this is used. Using an interdisciplinary mixed methods approach, I worked across several countries (Ethiopia, Kenya, Indonesia, Malawi, and Mozambique) to provide evidence to inform policy decisions. I first examined the costs of a QI intervention in all five countries and then used those data as the foundation of a cost-effectiveness decision tree model for the intervention in Kenya. Through interviews with national and global decision makers, I qualitatively examined the use and value of evidence in community health programmes. I present the results in a series of three related publications, linking them together with a literature review and discussion that show how these studies build upon each other and what they add to the existing evidence base. This thesis shows that QI for community health is a good investment contingent on an existing cadre of community health workers. The budget impact of the QI intervention is low (less than 0.53% of general government health expenditure) and the modelled cost-effectiveness yields an incremental cost-effectiveness ratio of US$249.43 per disability-adjusted life year. The absolute costs are highly dependent on context and the intensity of the intervention. Qualitative findings indicate that decision makers are not satisfied with existing evidence and have limited capacity to assess its relevance for their settings and perspectives. As a result, power and politics fill this evidence gap. Evidence must be at the heart of decisions in funding universal health coverage for them to be sustainable. To achieve this, the global community must strengthen the relevance of evidence and build the capacity of decision makers to understand and apply it. For a complex system, useful evaluation should describe context and mechanism of an intervention, estimate the effect size on both programmatic and health impacts and accurately reflect the opportunity costs

'Do you trust those data?'-a mixed-methods study assessing the quality of data reported by community health workers in Kenya and Malawi.

High-quality data are essential to monitor and evaluate community health worker (CHW) programmes in low- and middle-income countries striving towards universal health coverage. This mixed-methods study was conducted in two purposively selected districts in Kenya (where volunteers collect data) and two in Malawi (where health surveillance assistants are a paid cadre). We calculated data verification ratios to quantify reporting consistency for selected health indicators over 3 months across 339 registers and 72 summary reports. These indicators are related to antenatal care, skilled delivery, immunization, growth monitoring and nutrition in Kenya; new cases, danger signs, drug stock-outs and under-five mortality in Malawi. We used qualitative methods to explore perceptions of data quality with 52 CHWs in Kenya, 83 CHWs in Malawi and 36 key informants. We analysed these data using a framework approach assisted by NVivo11. We found that only 15% of data were reported consistently between CHWs and their supervisors in both contexts. We found remarkable similarities in our qualitative data in Kenya and Malawi. Barriers to data quality mirrored those previously reported elsewhere including unavailability of data collection and reporting tools; inadequate training and supervision; lack of quality control mechanisms; and inadequate register completion. In addition, we found that CHWs experienced tensions at the interface between the formal health system and the communities they served, mediated by the social and cultural expectations of their role. These issues affected data quality in both contexts with reports of difficulties in negotiating gender norms leading to skipping sensitive questions when completing registers; fabrication of data; lack of trust in the data; and limited use of data for decision-making. While routine systems need strengthening, these more nuanced issues also need addressing. This is backed up by our finding of the high value placed on supportive supervision as an enabler of data quality

Influence of China's 2009 healthcare reform on the utilisation of continuum of care for maternal health services: evidence from two cross-sectional household surveys in Shaanxi Province.

BACKGROUND: Continuum of care for maternal health services (CMHS) is a proven approach to improve health and safety for mothers and newborns. This study aims to explore the influence of China's 2009 healthcare reform on improving the CMHS utilisation. METHODS: This population-based cross-sectional quantitative study included 2332 women drawn from the fourth and fifth National Health Service Surveys of Shaanxi Province, conducted in 2008 and 2013 respectively, before and after China's 2009 healthcare reform. A generalised linear mixed model (GLMM) was applied to analyse the influence of this healthcare reform on utilisation of CMHS. Concentration curves, concentration indexes and its decomposition method were used to analyse the equity of changes in utilisation. RESULTS: This study showed post-reform CMHS utilisation was higher in both rural and urban women than the CMHS utilisation pre-reform (according to China's policy defining CMHS). The rate of CMHS utilisation increased from 24.66 to 41.55% for urban women and from 18.31 to 50.49% for rural women (urban: χ2 = 20.64, P < 0.001; rural: χ2 = 131.38, P < 0.001). This finding is consistent when the WHO's definition of CMHS is applied for rural women after reform (12.13% vs 19.26%; χ2 = 10.99, P = 0.001); for urban women, CMHS utilisation increased from 15.70 to 20.56% (χ2 = 2.57, P = 0.109). The GLMM showed that the rate of CMHS utilisation for urban women post-reform was five times higher than pre-reform rates (OR = 5.02, 95%CL: 1.90, 13.31); it was close to 15 times higher for rural women (OR = 14.70, 95%CL: 5.43, 39.76). The concentration index for urban women decreased from 0.130 pre-reform (95%CI: - 0.026, 0.411) to - 0.041 post-reform (95%CI: - 0.096, 0.007); it decreased from 0.104 (95%CI: - 0.012, 0.222) to 0.019 (95%CI: - 0.014, 0.060) for rural women. The horizontal inequity index for both groups of women also decreased (0.136 to - 0.047 urban and 0.111 to 0.019 for rural). CONCLUSIONS: China's 2009 healthcare reform has positively influenced utilisation rates and equity of CMHS's utilisation among both urban and rural women in Shaanxi Province. Addressing economic and educational attainment gaps between the rich and the poor may be effective ways to improve the persistent health inequities for rural women

SEEP-CI : a structured economic evaluation process for complex Health system interventions

The economic evaluation of health system interventions is challenging, and methods guidance on how to respond to these challenges is lacking. The REACHOUT consortium developed and evaluated complex interventions for community health program quality improvement in six countries in Africa and Asia. Reflecting on the challenges we faced in conducting an economic evaluation alongside REACHOUT, we developed a Structured Economic Evaluation Process for Complex Health System Interventions (SEEP-CI). The SEEP-CI aims to establish the threshold effect size that would justify investment in a complex intervention, and provide an assessment to a decision-maker of how likely it is that the intervention can achieve this impact. We illustrate how the SEEP-CI could have been applied to REACHOUT to identify outcomes where the intervention might have impact and causal mechanisms, through which that impact might occur, guide data collection by focusing on proximal outcomes most likely to illustrate the effectiveness of the intervention, identify the size of health gain required to justify investment in the intervention, and indicate the assumptions required to accept that such health gains are credible. Further research is required to determine the feasibility and acceptability of the SEEP-CI, and the contexts in which it could be used

Decolonisation and quality of care.

Delivering high quality healthcare for all requires recognising the legacies of colonialism in driving power asymmetries and producing inequitable health outcomes both within and between countries say Bernice Yanful and colleagues

What factors do make quality improvement work in primary health care? Experiences of maternal health quality improvement teams in three Puskesmas in Indonesia.

BACKGROUND: Indonesia has been shifting from ensuring access to health services towards improving service quality. Accreditation has been used as quality assurance (QA) mechanism, first in hospitals and subsequently in primary health care facilities, including Puskesmas (community health centres). QA provides measures of whether services meet quality targets, but quality improvement (QI) is needed to make change and achieve improvements. QI is a cyclical process with cycles of problem identification, solution testing and observation. We investigated the factors which influenced the process of QI based on experience of maternal health QI teams in three Puskesmas in Cianjur district, West Java province, Indonesia. METHODS: Qualitative data were collected using 28 in-depth interviews at two points of time: pre- (April 2016) and post- QI intervention (April 2017), involving national, provincial, district and Puskesmas managers; and Puskesmas QI team members. Thematic analysis of transcripts was conducted. RESULTS: We found four main factors contributed to the process of QI: 1) leadership, including awareness and attitude of leader(s) towards QI, involvement of leader(s) in the QI process and decision-making in budget allocation for QI; 2) staff enthusiasm and multidisciplinary collaboration; 3) a culture where QI is integrated in existing responsibilities; and 4) the ongoing Puskesmas accreditation process, which increased the value of QI to the organisation. CONCLUSION: Making QI a success in the decentralised Indonesian system requires action at four levels. At individual level, leadership attributes can create an internal quality environment and drive organisational cultural change. At team level, staff enthusiasm and collaboration can be triggered through engaging and tasking everyone in the QI process and having a shared vision of what quality should look like. At organisational level, QI should be integrated in planned activities, ensuring financial and human resources. Lastly, QI can be encouraged when it is implemented by the wider health system as part of national accreditation programmes

Community participation and maternal health service utilization: lessons from the health extension programme in rural southern Ethiopia

Background Health extension workers (HEWs) are the implementers of the unique primary health care programme of Ethiopia. They facilitate community participation in maternal health service delivery via the health development army (HDA) and pregnant women forums (PWFs). As part of a quality improvement intervention, HEWs received training, guidance and supervision focused on facilitation of HDA meetings and PWFs. We aimed to assess the effect of the intervention on maternal health service utilization and explore the perceptions of stakeholders regarding efforts to enhance community participation in maternal health. Methods We conducted a mixed method study in Shebedino woreda (district), Sidama Zone, southern Ethiopia. The research team observed HDA meetings and PWFs (15), conducted in-depth interviews with 32 HEWs, 8 HEW supervisors and maternal health program managers, and conducted 8 focus group discussions (FGDs) with community members. The interviews and FGDs were recorded, transcribed, translated, coded in Nvivo and thematically analysed. We also collected quantitative data on HDA and PWF participation, antenatal care attendance and skilled delivery and analysed using Excel (Microsoft Inc, Seattle, WA, USA). Results The proportion of HDA leaders and pregnant women who attended the HDA and PWF meetings increased by 30.6% and 36% respectively, over 18 months of the intervention. The percentage of pregnant women identified and referred by HDA leaders increased from 42% to 85%, the antenatal care utilization increased from 73.4% to 77.6% and skilled delivery increased from 76.7% to 83.3%,) (p<0.05). From interviews with stakeholders, we found improved awareness about maternal health services and increased health seeking behaviour. However, lack of incentives and reporting formats for HDA leaders, absenteeism and limited support from kebele administrators constrained community participation in maternal health. Conclusion With focused training, guidance and regular supportive supervision, HEWs were able to stimulate and enhance community participation, resulting in better maternal health service utilization in rural communities. HEWs, volunteer HDAs, pregnant women and the wider community have a role to play in quality improvement of maternal health services

Donor aid mentioning newborns and stillbirths, 2002-19: an analysis of levels, trends, and equity.

BACKGROUND: Global aid for reproductive, maternal, newborn, and child health has stagnated in recent years, and aid mentioning newborns or stillbirths has previously represented a very small proportion of aid for reproductive, maternal, newborn, and child health. Neonatal survival targets have been set by 78 countries, and stillbirth prevention targets have been set by 30 countries, to address the 4·4 million newborn deaths and stillbirths globally. We aimed to generate novel estimates of current levels of, and trends in, aid mentioning newborns and stillbirths over 2002-19, and to assess whether the amount of aid disbursed aligns with the associated mortality burden. METHODS: For this analysis, we did a manual review and coding of the Organisation for Economic Co-operation and Development (OECD)'s Creditor Reporting System database from 2002 to 2019 using key search terms for aid mentioning newborns and stillbirths. We compared these findings with estimates of aid for reproductive, maternal, newborn, and child health for 2002-19 based on the Muskoka2 method. Findings are presented in 2019 US$according to the OECD's Development Assistance Committee deflators, which account for variation in exchange rates and inflation in donor countries. FINDINGS: We identified 21 957 unique records in the 2002-19 period. Aid mentioning newborns and stillbirths comprised approximately 10$1·6 billion) of reproductive, maternal, newborn, and child health funding overall in 2019 ($15·9 billion), with a small decrease in value between 2015 and 2019. 1284 (6$535 million) of their total value mentioned aid focused only on newborn health. Ten donors contributed 87% ($13·7 billion) of the total value of aid mentioning newborns and stillbirths during 2002-19. Aid mentioning newborns and stillbirths was inequitably allocated in the least developed countries (as defined by the UN), ranging from$18 per death in Angola to $1389 per death in Timor-Leste. Stillbirths were not mentioned in any funding in 2002-09, and they were only mentioned in 46 of 21 957 records in 2010-19, comprising$44·4 million of aid disbursed during this period. INTERPRETATION: Aid mentioning newborns and stillbirths is poorly matched to their corresponding mortality burden (representing 10% of aid for reproductive, maternal, newborn, and child health overall, yet accounting for approximately 50% of mortality in children <5 years) and across recipient countries (with substantial variation in the amount of aid received per newborn death and stillbirth between countries with similar health and economic needs). Our findings indicate that aid needs to be better targeted to populations with the highest mortality burdens, creating greater potential for impact. FUNDING: John D and Catherine T MacArthur Foundation, Bill & Melinda Gates Foundation, ELMA Philanthropies, Children's Investment Fund Foundation UK, Lemelson Foundation, and Ting Tsung and Wei Fong Chao Foundation. TRANSLATION: For the French translation of the abstract see Supplementary Materials section

Using research networks to generate trustworthy qualitative public health research findings from multiple contexts

Background: Qualitative research networks (QRNs) bring together researchers from diverse contexts working on multi-country studies. The networks may themselves form a consortium or may contribute to a wider research agenda within a consortium with colleagues from other disciplines. The purpose of a QRN is to ensure robust methods and processes that enable comparisons across contexts. Under the Self-Testing Africa (STAR) initiative and the REACHOUT project on community health systems, QRNs were established, bringing together researchers across countries to coordinate multi-country qualitative research and to ensure robust methods and processes allowing comparisons across contexts. QRNs face both practical challenges in facilitating this iterative exchange process across sites and conceptual challenges interpreting findings between contexts. This paper distils key lessons and reflections from both QRN experiences on how to conduct trustworthy qualitative research across different contexts with examples from Bangladesh, Ethiopia, Kenya, Indonesia, Malawi, Mozambique, Zambia and Zimbabwe. Methods: The process of generating evidence for this paper followed a thematic analysis method: themes initially identified were refined during several rounds of discussions in an iterative process until final themes were agreed upon in a joint learning process. Results: Four guiding principles emerged from our analysis: a) explicit communication strategies that sustain dialogue and build trust and collective reflexivity; b) translation of contextually embedded concepts; c) setting parameters for contextualizing, and d) supporting empirical and conceptual generalisability. Under each guiding principle, we describe how credibility, dependability, confirmability and transferability can be enhanced and share good practices to be considered by other researchers. Conclusions: Qualitative research is often context-specific with tools designed to explore local experiences and understandings. Without efforts to synthesise and systematically share findings, common understandings, experiences and lessons are missed. The logistical and conceptual challenges of qualitative research across multiple partners and contexts must be actively managed, including a shared commitment to continuous ‘joint learning’ by partners. Clarity and agreement on concepts and common methods and timelines at an early stage is critical to ensure alignment and focus in intercountry qualitative research and analysis processes. Building good relationships and trust among network participants enhance the quality of qualitative research findings. Keywords: Qualitative research, Research networks, trustworthiness, Generalisable research, Research guiding principles, Research good practice