Reliability and validity of the World Health Organization reading standards for paediatric chest radiographs used in the field in an impact study of Pneumococcal Conjugate Vaccine in Kilifi, Kenya

Background Radiologically-confirmed pneumonia (RCP) is a specific end-point used in trials of Pneumococcal Conjugate Vaccine (PCV) to estimate vaccine efficacy. However, chest radiograph (CXR) interpretation varies within and between readers. We measured the repeatability and reliability of paediatric CXR interpretation using percent agreement and Cohen’s Kappa and the validity of field readings against expert review in a study of the impact of PCV on pneumonia. Methods CXRs were obtained from 2716 children admitted between 2006 and 2014 to Kilifi County Hospital, Kilifi, Kenya, with clinically-defined severe or very-severe pneumonia. Five clinicians and radiologists attended a three-day training course on CXR interpretation using a WHO standard. All CXRs were read once by two local primary readers. Discordant readings and 13% of concordant readings were arbitrated by a panel of three expert radiologists. To assess repeatability, a 5% median random sample was presented twice. Sensitivity and specificity of the primary readers’ interpretations was estimated against the ‘gold-standard’ of the arbitrators’ results. Results Of 2716 CXRs, 2 were uninterpretable and 159 were evaluated twice. The percent agreement and Kappa for RCP were 89% and 0.68 and ranged between 84–97% and 0.19–0.68, respectively, for all pathological findings. Intra-observer repeatability was similar to inter-observer reliability. Sensitivities of the primary readers to detect RCP were 69% and 73%; specificities were 96% and 95%. Conclusion Intra- and inter-observer agreements on interpretations of radiologically-confirmed pneumonia are fair to good. Reasonable sensitivity and high specificity make radiologically-confirmed pneumonia, determined in the field, a suitable measure of relative vaccine effectiveness

Effect of 10-valent pneumococcal conjugate vaccine on the incidence of radiologically-confirmed pneumonia and clinically-defined pneumonia in Kenyan children: an interrupted time-series analysis

BACKGROUND: Pneumococcal conjugate vaccines (PCV) are highly protective against invasive pneumococcal disease caused by vaccine serotypes, but the burden of pneumococcal disease in low-income and middle-income countries is dominated by pneumonia, most of which is non-bacteraemic. We examined the effect of 10-valent PCV on the incidence of pneumonia in Kenya. METHODS: We linked prospective hospital surveillance for clinically-defined WHO severe or very severe pneumonia at Kilifi County Hospital, Kenya, from 2002 to 2015, to population surveillance at Kilifi Health and Demographic Surveillance System, comprising 45 000 children younger than 5 years. Chest radiographs were read according to a WHO standard. A 10-valent pneumococcal non-typeable Haemophilus influenzae protein D conjugate vaccine (PCV10) was introduced in Kenya in January, 2011. In Kilifi, there was a three-dose catch-up campaign for infants (aged <1 year) and a two-dose catch-up campaign for children aged 1-4 years, between January and March, 2011. We estimated the effect of PCV10 on the incidence of clinically-defined and radiologically-confirmed pneumonia through interrupted time-series analysis, accounting for seasonal and temporal trends. FINDINGS: Between May 1, 2002 and March 31, 2015, 44 771 children aged 2-143 months were admitted to Kilifi County Hospital. We excluded 810 admissions between January and March, 2011, and 182 admissions during nurses' strikes. In 2002-03, the incidence of admission with clinically-defined pneumonia was 2170 per 100 000 in children aged 2-59 months. By the end of the catch-up campaign in 2011, 4997 (61·1%) of 8181 children aged 2-11 months had received at least two doses of PCV10 and 23 298 (62·3%) of 37 416 children aged 12-59 months had received at least one dose. Across the 13 years of surveillance, the incidence of clinically-defined pneumonia declined by 0·5% per month, independent of vaccine introduction. There was no secular trend in the incidence of radiologically-confirmed pneumonia over 8 years of study. After adjustment for secular trend and season, incidence rate ratios for admission with radiologically-confirmed pneumonia, clinically-defined pneumonia, and diarrhoea (control condition), associated temporally with PCV10 introduction and the catch-up campaign, were 0·52 (95% CI 0·32-0·86), 0·73 (0·54-0·97), and 0·63 (0·31-1·26), respectively. Immediately before PCV10 was introduced, the annual incidence of clinically-defined pneumonia was 1220 per 100 000; this value was reduced by 329 per 100 000 at the point of PCV10 introduction. INTERPRETATION: Over 13 years, admissions to Kilifi County Hospital for clinically-defined pneumonia decreased sharply (by 27%) in association with the introduction of PCV10, as did the incidence of radiologically-confirmed pneumonia (by 48%). The burden of hospital admissions for childhood pneumonia in Kilifi, Kenya, has been reduced substantially by the introduction of PCV10. FUNDING: Gavi, The Vaccine Alliance and Wellcome Trust

Coverage and timeliness of vaccination and the validity of routine estimates: Insights from a vaccine registry in Kenya

Background The benefits of childhood vaccines are critically dependent on vaccination coverage. We used a vaccine registry (as gold standard) in Kenya to quantify errors in routine coverage methods (surveys and administrative reports), to estimate the magnitude of survivor bias, contrast coverage with timeliness and use both measures to estimate population immunity. Methods Vaccination records of children in the Kilifi Health and Demographic Surveillance System (KHDSS), Kenya were combined with births, deaths, migration and residence data from 2010 to 17. Using inverse survival curves, we estimated up-to-date and age-appropriate vaccination coverage, calculated mean vaccination coverage in infancy as the area under the inverse survival curves, and estimated the proportion of fully immunised children (FIC). Results were compared with published coverage estimates. Risk factors for vaccination were assessed using Cox regression models. Results We analysed data for 49,090 infants and 48,025 children aged 12–23 months in 6 birth cohorts and 6 cross-sectional surveys respectively, and found 2nd year of life surveys overestimated coverage by 2% compared to birth cohorts. Compared to mean coverage in infants, static coverage at 12 months was exaggerated by 7–8% for third doses of oral polio, pentavalent (Penta3) and pneumococcal conjugate vaccines, and by 24% for the measles vaccine. Surveys and administrative coverage also underestimated the proportion of the fully immunised child by 10–14%. For BCG, Penta3 and measles, timeliness was 23–44% higher in children born in a health facility but 20–37% lower in those who first attended during vaccine stock outs. Conclusions Standard coverage surveys in 12–23 month old children overestimate protection by ignoring timeliness, and survivor and recall biases. Where delayed vaccination is common, up-to-date coverage will give biased estimates of population immunity. Surveys and administrative methods also underestimate FIC prevalence. Better measurement of coverage and more sophisticated analyses are required to control vaccine preventable diseases

Distribution of 2,875 chest radiographs within the batches according to the vaccine period and repeat readings for intra-observer variability evaluation.

<p>Distribution of 2,875 chest radiographs within the batches according to the vaccine period and repeat readings for intra-observer variability evaluation.</p

Performance of primary readers interpreting 232 chest radiographs selected at random from among their concordant readings compared against the final readings of the arbitration panel.

<p>Performance of primary readers interpreting 232 chest radiographs selected at random from among their concordant readings compared against the final readings of the arbitration panel.</p

Inter-observer agreement between the two primary readers on pathological features and on radiologically-confirmed pneumonia against batch number.

<p>Inter-observer agreement between the two primary readers on pathological features and on radiologically-confirmed pneumonia against batch number.</p

Inter-observer variability of the primary readers on 2714 images.

<p>Inter-observer variability of the primary readers on 2714 images.</p

Flow chart showing selection of chest radiographs for the separate analyses.

<p>Fig 1 footnote: * Of 232 CXRs with agreement between the two primary readers, the arbitration panel agreed with the primary readers (on all radiological criteria) in 181 and there was at least one disagreement between primary readers and the arbitration panel in 51. For RCP, there was agreement between primary readers and the arbitration panel in 219; in 13 radiographs the arbitration panel found RCP but the primary readers did not.</p

Weighted sensitivity and specificity of the primary readers’ conclusions against the arbiters’ gold-standard.

<p>Weighted sensitivity and specificity of the primary readers’ conclusions against the arbiters’ gold-standard.</p