ÐSophisticated mathematical algorithms (such as differencing, thresholding,

aggregation and statistical analysis of skin colours) are used to compare successive

frames of computer-captured images of the face. From these, changes in state of the

eyes are determined and are used to detect blinks. A recognition performance of

83.74+0.03% is achieved over ®ve subjects with a low rate of false positives

2.71+0.01%. A logical decision rule identi®es purposeful blinks and applies them

to control either a custom-designed communication package or an external device

Burke, D.

de Paor, A

Ward, Tomas E.

English

MURAL - Maynooth University Research Archive Library

Medical & BiImage processing used to harnessblinking as a channel ofcommunication and control forphysically disabled peopleD. Burke1,2 T. Ward1,3 A. de Paor1,21Rehabilitation Engineering, National Rehabilitation Hospital, Dun Laoghaire, Co. Dublin, Ireland2National University of Ireland, Dublin, Ireland3National University of Ireland, Maynooth, IrelandAbstract—Sophisticated mathematical algorithms (such as differencing, thresholding,aggregation and statistical analysis of skin colours) are used to compare successiveframes of computer-captured images of the face. From these, changes in state of theeyes are determined and are used to detect blinks. A recognition performance of83.740.03% is achieved over five subjects with a low rate of false positives2.710.01%. A logical decision rule identifies purposeful blinks and applies themto control either a custom-designed communication package or an external device.Keywords—Image processing, Blinking, Rehabilitation, Locked-in syndrome, Commu-nicationMed. Biol. Eng. Comput., 2001, 39, 000–0001 IntroductionFOR SOME people with very severe physical disabilities (e.g.locked-in syndrome), eye movements may represent the onlyfeasible channel for communicating with others and forenvironmental control. There are effective non-contactsystems available to detect eye gaze based on infrared, e.g.Eyegaze System, CLEVELAND et al. (1992). However, thesesystems require specialist equipment and suffer from therequirement that the user must calibrate the system, whichnecessitates some level of cognition to be first established withdisabled patients. In contrast to marker-based systems(MIYAZAKI et al., 2000), the philosophy adopted here was todevelop an entirely non-contact, marker-free, and real-timesolution, using an inexpensive digital camera (‘a webcam’), adesktop PC (Pentium III, 500 MHz), and a judicious choice ofsophisticated techniques from digital image processing. Digitalimage processing is a very wide field but for present purposes,can be regarded as a collection of mathematical algorithmsapplied to computer stored visual images to extract features,which can be used to accomplish some desirable end. Bydispensing with the requirement for markers, the systempresents itself as unobtrusive and simple to use to boththerapist and patient alike. Based on a development of ideasdescribed in CLARKE et al. (1998), the mathematical techni-ques are summarised in the next section. We present the resultsof a reliability measure of the system in the subsequent section,and the paper concludes with a brief description of tworehabilitation applications.Correspondence should be addressed to D. Burke;email: david.burke@ucd.ieFirst received 8 December 2000 and in final form 2 March 2001MBEC online number: 20013579# IFMBE: 2001ological Engineering & Computing 2001, Vol. 392 MethodsThe basis of our blink detection method invokes the classicEuclidean distance formula to measure the colour differencebetween successive frames of images. Each pixel has a colourassociated with it represented by its discrete tristimulus values(red (R), green (G), and blue (B) parameters corresponding tointensities ranging from 0 to 255). The colour distance for a pixel(i, j) between frame n and nÿ 1 is calculated from:dijnRijnÿRijnÿ12GijnÿGijnÿ12BijnÿBijnÿ12qBy applying a threshold to the above equation, we can extractonly those pixels that have changed significantly in intensitybetween two successive frames, i.e. if dij[n]4dthreshold then setpixel (i, j) colour to blue say, otherwise set to white. Fig. 1illustrates the effect for a blink.Next, we determine the two aggregations of pixels having thelargest areas by using an algorithm that traces the perimeter of allgroups of adjacent pixels in the frame and reports the respectiveareas. Eyes are classified as detected when the horizontal andvertical separations between the largest two aggregations areverified to be between obvious anatomical bounds (assuming anapproximate vertical orientation of the head).Although the techniques presented thus far can be used todetect eye blinks, they still require some refinement to removefalse detections due to saccadic movements of the eyes, or in thecase of one patient, to filter the effect of nystagmus. Nystagmuscan also be problematic with infrared-based systems(CLEVELAND et al., 1992). To differentiate eye movementsfrom actual eye blinks, we make use of the fact that onaverage, pixels representing closed eyes will be predominatelyflesh coloured. On detection of the first blink (or a saccadic eyemovement as the case might be), we sample a rectangular area285surrounding the eyes (proportional to the inter-eye separation) togenerate a statistical histogram of flesh colours. To simplifycomputation, and remove brightness information, we normalisethe tristimulus values, producing the trichromatic coefficients(GONALES and WINTZ, 1987):r  RR G Bg  GR G Bb  BR G BSince r g  b  1 we can now evidently describe chromati-city by just two values, e.g. r and g. Fig. 2 illustrates an exampleof a histogram of flesh colours sampled in an area surroundingthe eyes.The brightness has been normalised and we are now viewingjust the chromatic (hue and saturation) information. Once thehistogram has been generated, the trichromatic coefficients ofpixels from subsequent frames are compared to the storeddistribution and the relative frequency of each measured chro-maticity is determined. Frequencies above a threshold arecharacterised as flesh colour. Fig. 3 illustrates the flesh colourfiltering mechanism (flesh colours are shown in black).For successive blinks, we compute the average probabilitythat the detected pair of pixel aggregates is flesh coloured (i.e.eyelids are closed) and probabilities above a certain value areclassed as true blinks.Fig. 1 Detection of blinks by the colour distance method1005000.40.200 0.20.4 0.60.81.relativefrequencyrgFig. 2 Distribution of the trichromatic coefficients of a flesh sample2863 ResultsFive volunteer subjects (two male, three female, aged between21 and 25) each performed a set of five experimental blockslasting two minutes each. This translates to 50 minutes (56562)of analysis. During each experimental block, subjects wereplaced in a seated position 0.5 m from the digital camera andasked to look in the direction of the camera while remainingreasonably still. A second camera recorded the subject’s face andits output was routed to a video=audio capture board in a separatePC. Each detected blink resulted in an audio track to togglebetween play and pause and this audio was recorded simulta-neously by the capture board. The captured audio and video werethen manually analysed offline and the fraction of true positivesand false positives determined for each experimental blockfor each subject (Table 1). Fig. 4 presents the results (meanand standard deviation) for each subject. Performance averagedover all subjects yields 83.74  0.03% for true positives and2.71  0.01% for false positives.Fig. 3 Detection of flesh colours0 1 2 3 4 5 600.10.20.30.40.50.60.70.80.91.0subject numberfractiontrue positivesfalse positivesFig. 4 Fraction of true positives and false positives for each subjectTable 1 ResultsTrial 1 Trial 2 Trial 3 Trial 4 Trial 5A D F A D F A D F A D F A D FSubject 1 54 48 0 48 39 2 48 42 0 45 38 1 57 52 1Subject 2 25 20 0 13 10 0 11 10 1 40 35 0 32 27 0Subject 3 11 9 1 23 19 2 16 15 0 12 8 0 21 19 0Subject 4 42 34 0 29 22 2 49 38 1 17 15 0 23 21 2Subject 5 33 27 0 34 27 2 33 26 0 36 30 0 33 29 3A: actual blinks; D: detected blinks; F: false positives.Medical & Biological Engineering & Computing 2001, Vol. 394 ConclusionsWe have harnessed the above techniques in two successfulrehabilitation applications to date. A component architecture hasbeen developed in which the blink detection system runs in aseparate process and interprocess communication (IPC) allowsdifferent applications to be effectively ‘plugged-in’. The firstapplication uses purposeful blinks to control the switchingaction of a CD player, ubiquitous in the average personalcomputer. The second application utilises blinks to control acustom-designed communication program, allowing users tobuild up sentences letter by letter, and produce synthesisedspeech. With each application, an adaptable logical decisionrule operates allowing different blink rates to trigger thecorresponding control action. Only the on-response is detectedsince this allows us to filter eye movements, which can some-times give erroneous results.We have presented preliminary results on the use of real-timedigital image processing in the implementation of a non-contacthuman–computer interface. The results are promising, indi-cating that this type of technology is sufficiently robust to beused in rehabilitation applications. Where large head movementsoccur, the system does not attempt to detect blinks (sincedetermining the position of the eyes becomes problematic)although this has not been an issue with the type of patientsthe system was built for. We are currently developing animproved set of algorithms to account for large head movementsby tracking the position of the eyes (by first locating the face bythe flesh colour filtering mechanism) and performing thedifference operation on the pixels marked as representing theeyes.Acknowledgments—The authors wish to thank the OccupationalTherapy Department at the National Rehabilitation Hospital,Ireland for constructive criticism and fruitful discussions.Medical & Biological Engineering & Computing 2001, Vol. 39ReferencesCLARKE, L., HARPER, P., and REILLY, R. (1998): ‘Video based gesturerecognition for augmentative communication’. TIDE ThirdCongress Papers, HelisinkiCLEVELAND, N. R., and DOYLE, P. G. (1992): ‘Eyegaze communica-tion system, How does it work? Who can use it?’. Closing thegap – 10th Annual Conference, Minneapolis, Minnesota, Oct.22–24, 1992GONALES, R. C., and WINTZ, P. (1987): ‘Digital image processing’,(Addison-Wesley), 2nd ednMIYAZAKI, S., ISHIDA, A., and KOMATSUZAKI, A. (2000): ‘A clinicallyoriented video-based system for quantification of eyelidmovements’, IEEE Trans. Biomed. Eng., 47, pp. 1088–1096Authors’ biographiesDAVID BURKE received his BE and MEngSc from the NationalUniversity of Ireland (NUI), Dublin, in 1997 and 1998, respectively.He is currently working towards a PhD at the NUI, Dublin. Hisresearch interests are in the areas of dynamical systems analysisapplied to biological modelling and Internet-based human-computerinteraction.TOMAS WARD received his BE, MEngSc and PhD degrees from theNational University of Ireland, Dublin, in 1994, 1996 and 1999,respectively. From 1999–2000 he taught at the Department of Com-puter Science at the National University of Ireland (NUI), Maynooth.Currently he teaches in the Department of Electronic Engineering atNUI Maynooth. His research lies in the areas of biological modellingand human-computer interaction.ANNRAOI DE PAOR is Professor of Electrical Engineering at theNational University of Ireland, Dublin. He founded and directs theEngineering Research Laboratory at the National Rehabilitation Hos-pital, Du´n Laoghaire. He has published over 200 papers on controltheory, biomedical engineering, renewable energy systems, engineer-ing education, classical electrical engineering, and geomagnetism.287

Image processing used to harness blinking as a channel of

communication and control for physically disabled people

http://mural.maynoothuniversity.ie/1286/1/TWburke_harry_blink.pdf

Image processing used to harness blinking as a channel of communication and control for physically disabled people

Abstract

Similar works

Full text

Available Versions

MURAL - Maynooth University Research Archive Library