Background Diarrhoeal disease remains a leading cause of death among children mostly in low and middle- income countries. Factors contributing to disease severity are complex and there is currently no consensus on a scoring tool for use in community-based studies. Methods Data were collected during a passive surveillance system in an outpatient health facility in Lusaka, Zambia from March 2019 to July 2019. Diarrhea episodes were assessed for severity using an in-house severity scoring tool (CIDRZ) and previously published scores (Vesikari, Clark, CODA, and DHAKA). The CIDRZ score was constructed using fieldworker-reported clinical signs and exploratory factor analysis. We used precision-recall curves measuring severe diarrhoea (i.e., requiring intravenous rehydration or referred for hospital admission) to determine the best performing scores. Then, we used Cronbach's alpha to assess the scale's internal consistency. Finally, we used Cohen's kappa to assess agreement between the scores. Results Of 110 diarrhea episodes, 3 (3%) required intravenous rehydration or were referred for hospital admission. The precision-recall area under the curve of each score as a predictor of severe diarrhoea requiring intravenous rehydration or hospital admission was 0.26 for Vesikari, 0.18 for CODA, 0.24 for Clark, 0.59 for DHAKA, and 0.59 for CIDRZ. The CIDRZ scale had substantial reliability and performed similarly to the DHAKA score. Conclusions Diarrhoea severity scores focused on characteristics specific to dehydration status may better predict severe diarrhea among children in Lusaka. Aetiology-specific scoring tools may not be appropriate for use in community healthcare settings. Validation studies for the CIDRZ score in diverse settings and with larger sample sizes are warranted
