Appropriateness of clinical severity classification of new WHO childhood pneumonia guidance : a multi-hospital, retrospective, cohort study

Background: Management of pneumonia in many low-income and middle-income countries is based on WHO guidelines that classify children according to clinical signs that define thresholds of risk. We aimed to establish whether some children categorised as eligible for outpatient treatment might have a risk of death warranting their treatment in hospital. Methods: We did a retrospective cohort study of children aged 2–59 months admitted to one of 14 hospitals in Kenya with pneumonia between March 1, 2014, and Feb 29, 2016, before revised WHO pneumonia guidelines were adopted in the country. We modelled associations with inpatient mortality using logistic regression and calculated absolute risks of mortality for presenting clinical features among children who would, as part of revised WHO pneumonia guidelines, be eligible for outpatient treatment (non-severe pneumonia). Findings: We assessed 16 162 children who were admitted to hospital in this period. 832 (5%) of 16 031 children died. Among groups defined according to new WHO guidelines, 321 (3%) of 11 788 patients with non-severe pneumonia died compared with 488 (14%) of 3434 patients with severe pneumonia. Three characteristics were strongly associated with death of children retrospectively classified as having non-severe pneumonia: severe pallor (adjusted risk ratio 5·9, 95% CI 5·1–6·8), mild to moderate pallor (3·4, 3·0–3·8), and weight-for-age Z score (WAZ) less than −3 SD (3·8, 3·4–4·3). Additional factors that were independently associated with death were: WAZ less than −2 to −3 SD, age younger than 12 months, lower chest wall indrawing, respiratory rate of 70 breaths per min or more, female sex, admission to hospital in a malaria endemic region, moderate dehydration, and an axillary temperature of 39°C or more. Interpretation: In settings of high mortality, WAZ less than −3 SD or any degree of pallor among children with non-severe pneumonia was associated with a clinically important risk of death. Our data suggest that admission to hospital should not be denied to children with these signs and we urge clinicians to consider these risk factors in addition to WHO criteria in their decision making

AfriMTE and AfriCOMET : Empowering COMET to Embrace Under-resourced African Languages

Despite the progress we have recorded in scaling multilingual machine translation (MT) models and evaluation data to several under-resourced African languages, it is difficult to measure accurately the progress we have made on these languages because evaluation is often performed on n-gram matching metrics like BLEU that often have worse correlation with human judgments. Embedding-based metrics such as COMET correlate better; however, lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with a simplified MQM guideline for error-span annotation and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AfriCOMET, a COMET evaluation metric for African languages by leveraging DA training data from high-resource languages and African-centric multilingual encoder (AfroXLM-Roberta) to create the state-of-the-art evaluation metric for African languages MT with respect to Spearman-rank correlation with human judgments (+0.406)

AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languages

Despite the recent progress on scaling multilingual machine translation (MT) to several under-resourced African languages, accurately measuring this progress remains challenging, since evaluation is often performed on n-gram matching metrics such as BLEU, which typically show a weaker correlation with human judgments. Learned metrics such as COMET have higher correlation; however, the lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with simplified MQM guidelines for error detection and direct assessment (DA) scoring for 13 typologically diverse African languages. Furthermore, we develop AfriCOMET: COMET evaluation metrics for African languages by leveraging DA data from well-resourced languages and an African-centric multilingual encoder (AfroXLM-R) to create the state-of-the-art MT evaluation metrics for African languages with respect to Spearman-rank correlation with human judgments (0.441)