A smartphone app to screen for neonatal jaundice has a large

potential impact in reducing neonatal death and disability.

Our app, neoSCB, uses a colour measurement of the sclera

to make a screening decision. Although there are numerous

benefits of a smartphone-based approach, smartphone colour

measurement that is accurate and repeatable is a challenge.

Using data from a clinical setting in Ghana, we compare

sclera colour measurement using an ambient subtraction

method to sclera colour measurement using a standard colour

card method, and find they are comparable provided the subtracted signal-to-noise ratio (SSNR) is sufficient. Calculating

a screening decision metric via the colour card method gave

100% sensitivity and 69% specificity (n=87), while applying

the ambient subtraction method gave 100% sensitivity and

78% specificity (SSNR>3.5; n=50)

Brako, NO

Enweronu-Laryea, C

Leung, TS

MacDonald, LW

Meek, J

Nixon, M

Outlaw, F

English

UCL Discovery

Poster: Smartphone Colorimetry Using AmbientSubtraction: Application to Neonatal JaundiceScreening in GhanaFelix Outlawfelix.outlaw.15@ucl.ac.ukUCL Department of Medical Physicsand Biomedical EngineeringLondon, UKMiranda NixonUCL Department of Medical Physicsand Biomedical EngineeringLondon, UKNana Okai BrakoGreater Accra Regional HospitalAccra, GhanaLindsay W. MacDonaldUCL Department of CivilEnvironmental and GeomaticEngineeringLondon, UKJudith MeekThe Neonatal Care Unit, ElizabethGarrett Anderson Wing, UniversityCollege London HospitalsLondon, UKChristabel Enweronu-LaryeaDepartment of Child Health,University of Ghana School ofMedicine and Dentistry and Korle-BuTeaching HospitalAccra, GhanaTerence S. LeungUCL Department of Medical Physicsand Biomedical EngineeringLondon, UKABSTRACTA smartphone app to screen for neonatal jaundice has a largepotential impact in reducing neonatal death and disability.Our app, neoSCB, uses a colour measurement of the sclerato make a screening decision. Although there are numerousbenefits of a smartphone-based approach, smartphone colourmeasurement that is accurate and repeatable is a challenge.Using data from a clinical setting in Ghana, we comparesclera colour measurement using an ambient subtractionmethod to sclera colourmeasurement using a standard colourcard method, and find they are comparable provided the sub-tracted signal-to-noise ratio (SSNR) is sufficient. Calculatinga screening decision metric via the colour card method gave100% sensitivity and 69% specificity (n=87), while applyingthe ambient subtraction method gave 100% sensitivity and78% specificity (SSNR>3.5; n=50).Permission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contactthe owner/author(s).UbiComp/ISWC ’19 Adjunct, September 9–13, 2019, London, United Kingdom© 2019 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-6869-8/19/09.https://doi.org/10.1145/3341162.3343805CCS CONCEPTS• Human-centered computing → Ubiquitous and mo-bile computing systems and tools; Smartphones.KEYWORDScolorimetry, neonatal jaundice, screening, smartphoneACM Reference Format:Felix Outlaw, Miranda Nixon, Nana Okai Brako, Lindsay W. Mac-Donald, Judith Meek, Christabel Enweronu-Laryea, and Terence S.Leung. 2019. Poster: Smartphone Colorimetry Using Ambient Sub-traction: Application to Neonatal Jaundice Screening in Ghana. InAdjunct Proceedings of the 2019 ACM International Joint Conferenceon Pervasive and Ubiquitous Computing and the 2019 InternationalSymposium on Wearable Computers (UbiComp/ISWC ’19 Adjunct),September 9–13, 2019, London, United Kingdom. ACM, New York,NY, USA, 4 pages. https://doi.org/10.1145/3341162.33438051 INTRODUCTIONThe accurate quantification of colour is important in a widerange of disciplines, including several scientific and med-ical applications. Context may add constraints as to howcolour may be measured. In the least constrained case, aspectrophotometer can be used to measure the reflectanceof an object with high spectral resolution, for example, intesting the colour reproduction quality of printed materials.In other cases, the target may be harder to measure if it isdelicate, transient, or inaccessible. For example, the measure-ment of animal or animal habitat colours, which often mustUbiComp/ISWC ’19 Adjunct, September 9–13, 2019, London, United Kingdom Outlaw et al.be done in situ, under time pressure, and without disturbingthe subject. In such cases, digital cameras have advantagesof portability and rapid, contact-free measurement [1, 11].Medical uses of colorimetry include screening for condi-tions such as anaemia and jaundice [4, 6–10, 13]. Jaundice isa potentially life-threatening condition that affects the major-ity of newborns [2]. It is caused by an excessive accumulationof bilirubin, which manifests as a yellow discolouration ofthe skin and sclera.Existing approaches to screen for neonatal jaundice havesignificant limitations. The transcutaneous bilirubinometer(TcB) is a contact-based jaundice screening device that worksby measuring the reflectance of the skin at various wave-lengths. While effective, cost limits its use, especially in low-income countries. Subjective visual inspection is often theonly available option in these settings, though it is not areliable means of screening [2].We aim to detect dangerous cases of jaundice using asmartphone camera measurement of the sclera colour. Unlikeskin, the sclera does not contain melanin and haemoglobinchromophores, and so it is a more sensitive site to measurethe yellow discolouration from jaundice [6, 8]. However, mea-suring the eye rules out contact-based colour measurement.Jaundice can become severe days to weeks after birth, andso screening methods must be usable in a home environment.Jaundice-related disability and death is most common in Sub-Saharan Africa and South Asia, where it is estimated thatthree-quarters of the deaths occur [3]. This means there is aneed for a low-cost approach. These additional constraintsmake the smartphone an ideal candidate for a screening tool.Not only do smartphones allow rapid, objective, and contact-less measurement of colour, but they are also ubiquitous inlow-income regions and in the domestic setting worldwide.Some studies have shown promising results using a smart-phone to detect jaundice [4, 10, 12]. Our neoSCB app (neona-tal Scleral-Conjunctival Bilirubinometer) achieved a screen-ing performance comparable to a modern TcB [10]. BiliCamis a smartphone app that has been validated in a sample of530 newborns with impressive results [12]. BiliCam relies ona colour card to standardise a measurement of the skin, whileneoSCB is based on sclera colour and uses a flash/ no-flashimage pair to suppress the influence of ambient light.In this work we evaluate an ambient subtraction (AS)method of standardising colour measurement using a smart-phone and compare this approach to a colour card (CC) ap-proach. To do this we analyse image data collected as partof a two-year study in Ghana, the purpose of which is todevelop and test the neoSCB jaundice screening app.2 THEORYThe sensor RGB values recorded by a digital camera dependon the illuminating light and the camera as well as the objectproperties. The prevailing ambient light varies depending onthe image capture environment, and different smartphone de-vices do not have the same spectral responses. Accurate andrepeatable colour measurement must be device- and ambientlight-independent. This can be achieved by including a refer-ence colour card with an array of patches of known colour(defined in a reference colour space such as the CIE 1931XYZ colour space). The raw sensor RGB triplets recordedare mapped to XYZ values by a transform developed for thatimage. This simultaneously corrects for ambient light andcamera variability.The AS approach relies on taking two pictures in quicksuccession, one with an LED flash and one without. By sub-tracting the raw RGB values of the ambient-only image fromthe flash-and-ambient image, the RGB values as they wouldappear under only the LED flash are estimated. This stan-dardises the lighting environment, but the measurement isstill device-specific, as camera sensors and LED flashes varybetween smartphones. A colour card transform is used toachieve device-independent XYZ values. The advantage ofAS is that this transform need only be developed once perdevice rather than in each new capture instance.The camera signal post-subtraction may have a low signal-to-noise ratio if the ambient light dominates the smartphoneflash. To quantify this, we introduce the subtracted signal-to-noise ratio (SSNR). The noise is approximated as shot noise,given by the square root of the number of photon counts,I, for channel k, k={R,G,B}. Noise is added in quadraturebetween the flash (ambient-plus-flash) and no-flash (ambient-only) images to give Equation 1 for the SSNR.SSNR =sub . siдnalsub . noise=Iambient−plus−f lashk − Iambient−onlyk√Iambient−plus−f lashk + Iambient−onlyk(1)3 METHODSThe neoSCB app was used to capture flash/ no-flash imagepairs of the sclera of 87 newborns in the Greater Accra Re-gional Hospital. The smartphone device was a Samsung S8.An X-Rite ColorChecker Classic colour card was included ineach image. The images were taken to coincide with a rou-tine blood test for total serum bilirubin (TSB) concentration,the gold-standard for jaundice diagnosis. Ethical approvalfor this study was obtained from the Institutional ReviewBoard, College of Health Science, University of Ghana, theGhana Health Service Ethics Review Committee, and theUCL Research Ethics Committee. All data processing wasperformed using MATLAB R2018a.In this work, all RGB-to-XYZ transforms are developedusing a least-squares linear mapping based on measurementsPoster: Smartphone Colorimetry Using Ambient Subtraction... UbiComp/ISWC ’19 Adjunct, September 9–13, 2019, London, United Kingdomof the 24 colour patches of the ColorChecker Classic. Thepatch XYZ values were measured using an X-Rite Color-Munki Photo. Equation 2 shows how the transformationmatrix M is a function of 24x3 matrix of colour card XYZvalues H and a 24x3 matrix of recorded raw camera RGBvalues R.M = (RTR)−1RTH (2)Ambient Subtraction (AS) MethodCorresponding sclera regions of interest (ROI) were manu-ally drawn in the flash and no-flash images. The median RGBvalues were subtracted in linear raw space to arrive at an es-timate of flash-only RGB sclera values. These were convertedto XYZ values using a device-specific transform developedfor the Samsung S8 by imaging the X-Rite ColorCheckerunder its flash illumination.Colour Card (CC) MethodThe no-flash image was used to attain sclera XYZ valuesusing the CC method. The colour card was segmented auto-matically using ColorChecker Finder [5], and the raw RGBvalues of the card were extracted. From these an image-specific mapping to XYZ was developed and applied to themedian raw RGB sclera values.Scleral-Conjunctival Bilirubin (SCB) CalculationIn each case, the XYZ values were used to calculate xy chro-maticity values: x = X/(X + Y + Z ); y = Y/(X + Y + Z ). Amultiple linear regression of TSB against x and y producedour TSB estimation model, the output of which we call thescleral-conjunctival bilirubin (SCB), in analogy with the out-put from a transcutaneous bilirubinometer (TcB).4 RESULTSChromaticity ErrorIn comparing the chromaticity values from the AS approachto the established CC approach, we calculated the Euclideandistance in the two-dimensional chromaticity space as ourerror metric. Figure 1 plots the SSNR against the chromaticityerror. The correlation is -0.41 (p<0.01).A laboratory experiment determined an empirical SSNRthreshold of 3.5 above which chromaticity error did not de-crease further (tested on 148 patches from the Macbeth Col-orChecker DC card). Applying this threshold, which elim-inated 37 of the 87 data points, decreased the mean chro-maticity error by 34%.TSB PredictionFigure 2 shows the correlation between TSB and SCB. Thecorrelation is higher for the CC method (r=0.79; p<0.01) thanFigure 1: Scatter plot of subtracted signal-to-noise ratio(SSNR) against chromaticity error.the AS method (r=0.50; p<0.01). However, when the SSNRthreshold is used to discard image pairs with SSNR below3.5, the correlation becomes comparable between the twomethods for the remaining 50 subjects.Figure 2: Correlation between scleral-conjunctival bilirubin(SCB) and total serum bilirubin (TSB). Horizontal and ver-tical lines are decision and screening thresholds used, re-spectively. (A) Ambient subtractionmethod; (B) Colour cardmethod; (C) Ambient subtraction method, where SSNR<3.5;(D) Colour card method for same subjects as in (C).Table 1 shows the screening performance in identifyingneonateswith TSB levels above 250µmol, theNICE-recommendedthreshold for blood test referral in term infants [2]. The CCmethod and AS method have area under curve (AUC) val-ues for the receiver operating characteristic curve of 0.91UbiComp/ISWC ’19 Adjunct, September 9–13, 2019, London, United Kingdom Outlaw et al.Table 1: Results for ambient subtraction (AS) and colour card(CC)methods in predicting total serumbilirubin from sclerachromaticity. Screening performance statistics are given fora screening threshold of 250µmol, with decision thresholdsset to maximise Youden’s Index. *p<0.01.All subjects, n=87 SSNR>3.5, n=50AS CC AS CCCorrelation, r 0.50* 0.79* 0.78* 0.80*Sensitivity 0.84 1.00 1.00 0.85Specificity 0.81 0.69 0.78 0.81AUC 0.82 0.91 0.93 0.89and 0.82 respectively. Again, when only the high SSNR dataare considered, the performance of the AS method increasesgreatly. For the n=50 sample the AS method has a higherAUC (0.93) than the CC method (AUC=0.89).5 DISCUSSIONOur results show that a sclera colour-based jaundice screen-ing tool using ambient subtraction can deliver comparablescreening performance to one using a colour card, providedsignal-to-noise in the post-subtraction signal is sufficient.Ambient subtraction has some advantages over a colourcard method in the context of jaundice screening. Colourcards are not cheap and must be kept pristine to be useful.Distribution and maintenance of colour cards may representa barrier to adoption, particularly in low-income countries.Furthermore, especially when imaging the sclera rather thanthe skin, it is not trivial to capture an image that includesa sufficiently large sclera ROI as well as the entirety of thecolour card. A colour measurement protocol based on sub-traction requires only a one-time calibration.A limitation of the ambient subtraction approach is that itcan fail if there is movement between the first and second cap-tures. The capture sequence takes less than one second so thisproblem only affects a minority of cases. In future, for robust-ness, neoSCB will require several estimates to agree beforeproviding a result, and automatically recommend moving toa less bright area if SSNR is below the required threshold.ACKNOWLEDGMENTSThe authors would like to thank all the study participantsand their parents.This work is supported by the EPSRC-funded UCL Centrefor Doctoral Training in Medical Imaging (EP/L016478/1)and the Department of Health’s NIHR-funded BiomedicalResearch Centre at University College London Hospitals. Itis made possible through the generous support of the Sav-ing Lives at Birth partners: the United States Agency forInternational Development (USAID), the Government of Nor-way, the Bill & Melinda Gates Foundation, Grand ChallengesCanada, the UK Government, and the Korea InternationalCooperation Agency (KOICA). It was prepared by the au-thors and does not necessarily reflect the views of the SavingLives at Birth partners.REFERENCES[1] Derya Akkaynak, Tali Treibitz, Bei Xiao, Umut A Gürkan, Justine JAllen, Utkan Demirci, and Roger T Hanlon. 2014. Use of commercialoff-the-shelf digital cameras for scientific data acquisition and scene-specific color calibration. JOSA A 31, 2 (2014), 312–321.[2] Rachel C Amos, Hannah Jacob, and Wynne Leith. 2017. Jaundice innewborn babies under 28 days: NICE guideline 2016 (CG98). Archivesof Disease in Childhood-Education and Practice 102, 4 (2017), 207–209.[3] Vinod K Bhutani, Alvin Zipursky, Hannah Blencowe, Rajesh Khanna,Michael Sgro, Finn Ebbesen, Jennifer Bell, RintaroMori, TinaM Slusher,Nahed Fahmy, et al. 2013. Neonatal hyperbilirubinemia and Rhesusdisease of the newborn: incidence and impairment estimates for 2010at regional and global levels. Pediatric research 74, S1 (2013), 86.[4] Lilian De Greef, Mayank Goel, Min Joon Seo, Eric C Larson, James WStout, James A Taylor, and Shwetak N Patel. 2014. Bilicam: usingmobile phones to monitor newborn jaundice. In Proceedings of the2014 ACM International Joint Conference on Pervasive and UbiquitousComputing. ACM, 331–342.[5] Keigo Hirakawa. 2014. ColorChecker Finder. Retrieved June 02, 2019from http://campus.udayton.edu/~ISSL/software[6] Terence S Leung, Karan Kapur, Ashley Guilliam, Jade Okell, Bee Lim,Lindsay W MacDonald, and Judith Meek. 2015. Screening neonataljaundice based on the sclera color of the eye using digital photography.Biomedical optics express 6, 11 (2015), 4529–4538.[7] Terence S Leung, Felix Outlaw, Lindsay W MacDonald, and JudithMeek. 2019. Jaundice Eye Color Index (JECI): quantifying the yel-lowness of the sclera in jaundiced neonates with digital photography.Biomedical optics express 10, 3 (2019), 1250–1256.[8] Alex Mariakakis, Megan A Banks, Lauren Phillipi, Lei Yu, James Tay-lor, and Shwetak N Patel. 2017. Biliscreen: smartphone-based scleraljaundice monitoring for liver and pancreatic disorders. Proceedings ofthe ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies1, 2 (2017), 20.[9] Felix Outlaw, Judith Meek, Lindsay W MacDonald, and Terence SLeung. 2017. Screening for neonatal jaundice with a smartphone. InProceedings of the 2017 International Conference on Digital Health. ACM,241–242.[10] Felix Outlaw, Miranda Nixon, Oluwatobiloba Odeyemi, Lindsay WMacDonald, Judith Meek, and Terence S Leung. 2019. Smartphonescreening for neonatal jaundice via ambient-subtracted sclera chro-maticity: neoSCB app pilot study. BioRxiv (2019), 627034.[11] Martin Stevens, C Alejandro Párraga, Innes C Cuthill, Julian C Par-tridge, and Tom S Troscianko. 2007. Using digital photography tostudy animal coloration. Biological journal of the Linnean society 90, 2(2007), 211–237.[12] James A Taylor, JamesW Stout, Lilian de Greef, Mayank Goel, ShwetakPatel, Esther K Chung, Aruna Koduri, Shawn McMahon, Jane Dick-erson, Elizabeth A Simpson, et al. 2017. Use of a smartphone app toassess neonatal jaundice. Pediatrics 140, 3 (2017), e20170312.[13] Edward Jay Wang, William Li, Doug Hawkins, Terry Gernsheimer,Colette Norby-Slycord, and Shwetak N Patel. 2016. HemaApp: nonin-vasive blood screening of hemoglobin using smartphone cameras. InProceedings of the 2016 ACM International Joint Conference on Pervasiveand Ubiquitous Computing. ACM, 593–604.

Smartphone colorimetry using ambient subtraction: Application to neonatal jaundice screening in Ghana

https://discovery.ucl.ac.uk/10085333/1/Outlaw%20et%20al%20UBICOMP%202019.pdf

Smartphone colorimetry using ambient subtraction: Application to neonatal jaundice screening in Ghana

Abstract

Similar works

Full text

Available Versions

UCL Discovery