Various disabilities restrict the ease with which individuals can operate electronic and ICT
devices. X10 is a system for home automation control and consequently lends itself for use by
disabled individuals, who particularly have mobility restrictions, to control a wide range of
devices although the resultant user interface can be cumbersome. The development of an
adequate user-centred interface/control which will allow such an individual easily to operate
multiple ICT devices is then a considerable challenge. The development of a technique that
utilises a user’s point of gaze to select a particular ICT device for subsequent operation,
thereby simplifying the user interface, is described. All ICT devices in the environment are
first digitally imaged from different angles to identify them to a computer imaging system.
Subsequently each device can be automatically recognised. The user’s eye movements are
recorded and their direction of gaze related in real time to the known 3D location of the
possible ICT devices so enabling device selection prior to operation. The development of the
technique and current ongoing research status are described

Alastair Gale (1258077)

Fangmin Shi (7153157)

Kevin Purdy (7149809)

English

Loughborough University Institutional Repository

   This item was submitted to Loughborough’s Institutional Repository by the author and is made available under the following Creative Commons Licence conditions.      For the full text of this licence, please go to: http://creativecommons.org/licenses/by-nc-nd/2.5/   X10 - are you looking at me?  Shi F., Gale, A.G., and Purdy K.P.  Applied Vision Research Centre, University of Loughborough, Loughborough, LE11 3UZ, UK  Tel: +44 (0)1509 635732, Email: a.g.gale@lboro.ac.uk Abstract Various disabilities restrict the ease with which individuals can operate electronic and ICT devices. X10 is a system for home automation control and consequently lends itself for use by disabled individuals, who particularly have mobility restrictions, to control a wide range of devices although the resultant user interface can be cumbersome.  The development of an adequate user-centred interface/control which will allow such an individual easily to operate multiple ICT devices is then a considerable challenge.  The development of a technique that utilises a user’s point of gaze to select a particular ICT device for subsequent operation, thereby simplifying the user interface, is described.  All ICT devices in the environment are first digitally imaged from different angles to identify them to a computer imaging system.  Subsequently each device can be automatically recognised.  The user’s eye movements are recorded and their direction of gaze related in real time to the known 3D location of the possible ICT devices so enabling device selection prior to operation.  The development of the technique and current ongoing research status are described.  Background Individuals with several types of disability can have very limited mobility with greatly restricted hand/arm movements coupled with little physical strength, e.g. ALS (Amyotrophic Lateral Sclerosis), spinal cord injury, Cerebral Palsy, Multiple Sclerosis, or brain injury.  Various dedicated interaction control systems can enable such individuals the freedom to move about in the environment, using a powered wheelchair, and also to operate multiple devices by means of simple controls.  Some individuals have locked-in syndrome, which is a condition in which a person is aware but unable to communicate or move due to complete paralysis of the body.  This condition typically results from some interruption of motor pathways in the brain usually by a stroke, trauma or a tumour. However they are able to understand sensory stimuli and so interaction, whether it be communication or direct control of electronic devices, is possible by recording and utilising their eye movements.  It is further argued here that eye gaze can provide an appropriate method of interaction and device control for a whole range of disabled people.  The Attentive Responsive Technology (ART) project is developing a system enabling a user’s eye gaze to select a device and offer it for subsequent direct control.   The speed of operation of an eye gaze based interface is an important issue in such a development, and generally it may be expected to be slower than other forms of interaction but the converse has been shown to be possible (Ware and Mikaelian, 1987).  For instance, with the Dasher communication system, developed by Ward and MacKay (2002), users can achieve good ‘typing’ speeds simply by using their eye gaze behaviour.  In Europe, the COGAIN Network of Excellence (2004) is investigating the development of appropriate communication devices for such individuals; here the question of the individual being able to directly operate electrical or electronic devices is addressed.    In order to understand the needs of the end user of the ART system a representative selection of disabled people have been questioned. Questions have included:  their current and anticipated future use of ICT devices; which ICT devices they operate most often; their living environments; current interface methods of controlling such devices;  their opinions about the potential for an ART eye-operated interface and specific questions concerning the selection, as opposed to the operation of devices.  This work will be reported separately.  Home automation technology In a typical home or office environment there exist numerous electrical devices which are controlled by physically operating some simple switch (e.g. room lights), or non-electrical devices which can be controlled electrically (e.g. curtains), as well as more complex ICT (Information and Communication Technologies) devices.  The latter typically have a much more sophisticated control mechanism, typified by the plethora of domestic remote controls (such as for TV, DVD, and Hi-Fi).  Such devices are endemic in modern life and can pose a considerable challenge for independent operation by a disabled person, particularly someone who has restricted mobility.  The issue of offering the disabled person some means of independently operating numerous ICT and other electrical devices is addressed here.  One approach to controlling a wide range of devices is to use home automation technology based on the X10 system.  The X10 electronic communication protocol has been widely used for over 20 years and permits home automation devices to be operated remotely. It works by a transmitter sending control signals over the usual electrical wiring in a building to receivers.  Electrical devices, instead of being plugged directly into the electrical mains supply, are plugged into these interface receivers which in turn are plugged into the electrical supply.  Each receiver interprets control signals sent by the transmitter and if these signals are meant for that particular device then that device is operated accordingly.  X10 allows simple remote operation of on/off electrical devices as well as more complex commands more useful for the  control any ICT device.  As a home automation system X10 has several advantages.  It uses the existing wiring infrastructure in the building and so additional wiring is not necessary.  X10 devices are quite cheap, easy to use and readily available.  These devices can be controlled directly from a remote control or a computer which effectively sends a radio signal encompassing up to 256 addresses transponder which then transmits the signals over the building’s electrical circuit.  Up to 256 addresses can be accessed by X10 and so a wide range of commands can be issued and many devices operated (see figure 1).  Like any approach, there are potential disadvantages which need to be considered when implementing an X10 system and these include the need to be aware of any separate electrical circuits within a house and the fact that potentially X10 signals may be swamped by some other signals on the building’s electrical supply, although there is equipment to overcome both of these obstacles.  In addition, the signal transmission speed can be slow which may be an issue in some situations where response speed is critical.  Insert figure 1 about here  Overall the simple design and affordability of X10 as a control system for electronic devices makes it very attractive as a means for individuals easily to control multiple ICT devices in their environments.  Additionally, it acts as a starting point to enable people with mobility problems to control the same devices by themselves.  More recently there has been the development of other home automation systems which can control ICT devices such as Z-Wave and Zigbee.  Both approaches utilise radio frequency (RF) to communicate between home automation devices.   Essentially a mesh communication network between the individual devices is set up by these systems so that if one device is out of range of the transmitter then other devices act as intermediaries to extend the RF range to that device.  A third approach is Insteon, which additionally works with X10 devices, thereby providing backwards compatibility and can control many more devices than the other two approaches.  Again, such control systems require a suitable interface for a disabled user.  If X10 is used to control devices then a suitable means of interfacing to this control system is required. Ostensibly if some form of graphical user interface (or other menu system) were to be used then the user could be presented with a complex menu of options where the initial selection is between a top layer representing all of the various different available devices.  Lower menu levels would then deal with the operating aspects of each device.  For instance operating a lamp would be a simple case of selecting the lamp followed by on/off control whereas operating a TV set would require much more complex options (figure 2).    Insert figure 2 about here   Such a menu system becomes inordinately complex as the number of separate devices expands and each one needs to be presented to the user.  A suitable Assistive Technologies (AT) approach can present such a complex interface and an internet based AT network (EASTIN, 2005) for people with disabilities has recently been set up which provides information on existing AT products, as covered by the ISO 9999 standard (2002), and their availability.   A common method is to display such menu items on a computer screen and record the user’s eye movements.  By gazing at a menu target area of the screen for a certain length of time the user will then initiate the selection of that menu option. Various commercial systems exist for monitoring user’s eye movements in such scenarios (examples are: ERICA, Quick Glance, VisionKey, Eyegaze Communication System, Visioboard, EagleEyes, and MyTobii).  The difficulties with such an approach are that the disabled user has to be physically in front of the computer monitor (unless a head mounted miniature display screen is used) and the recording accuracy of the eye movement system for that particular user is important so that the correct option is selected appropriately.  Accuracy of recording eye gaze is always an issue with real world eye movement recording and is especially important where the user may be unable to control head movements.   The key issue with such eye movement techniiques, however, is that whenever one’s eyes are open then effectively these are pointing ‘somewhere’, which can be measured by such approaches.  However, this does not necessarily mean that the user is gazing attentively at the recorded point on the screen and can give rise to unwanted command activations whenever the user looks at something – a problem known as the ‘Midas touch’ (Jacob, 1990).  The art of teasing out whether a user is attending to something or simply looking but attending to something quite different is a pervasive difficulty with all eye movement research.  Insert Figure 3 (a and b) about here For a disabled person, can such an interface/control system be simplified?  The approach developed here as part of the ART (Attention Responsive Technology) project is to utilise the user’s eye gaze to select potential ICT devices from within the environment (figure 3a) which enables a simple interface for that specific device that the user could then choose, or not, to operate (figure 3b).  The significant point here is leaving it to the user whether they wish to actually operate a device, thereby overcoming many of the issues relating to utilising eye gaze itself as a direct control system.  It is possible to use eye gaze as a selection device  - Wooding et al. (2002) demonstrated that eye gaze can be used easily to operate response ‘eye buttons’ using over 5,000 naïve people. System development In the ART system development both a head mounted (ASL 501) and a remote head free (Smarteye) system are used to monitor user’s eye gaze behaviour.   Both approaches are widely in use for eye tracking purposes and the system needs to be robust in both scenarios.  It is envisaged, however, that for usability reasons, a final system will have no need for attachment to the head of the user.    Current work utilises the head mounted system.  There are several potential issues concerning the actual deployment of such a head mounted system with a disabled user population which are not considered here.  However, the head mounted approach readily lends itself to the system development and many of the issues faced are also congruent with a remote head-free system.  insert Figure 4 about here   The overall approach (Gale, 2004) is illustrated in figure 4, although for the head mounted system approach the first determination of the position of the user’s eyes in the environment is not required.  There are four main stages:  1. identify all known ICT devices (or objects) in the user’s environment 2. determine user’s direction of gaze 3. if gaze falls on an ICT object then initiate the interface for that object. 4. operate interface and control system  Identifying ICT objects The identification of ICT objects, which are essentially 3D volumes in the environment, using digital cameras which produce 2D images is a considerable challenge:-    • Objects need to be identified accurately from different camera directions • Each object may be presented at different orientations as these are moved about the environment by people (e.g. a mobile phone may be picked up in one location but put down in another) • Environmental lighting (both natural and artificial) may alter producing varying object shadows.   • Objects may obscure one another or be overlaid with other materials. • The technique needs to be able to allow new ICT objects to be added to the environment.    Conventional object recognition methods, e.g. template matching or by use of 3D models would not be suitable here.  Instead, each ICT object is first presented to the imaging system at various rotations, the images processed and stored to build up a small dataset of ICT object representations.  This easily allows for the addition of new objects.  In examining the environment for the presence of possible objects, then Lowe’s (2004) SIFT (Scale Invariant Feature Transform) algorithm is applied and ICT objects are either recognised, together with a measured degree of accuracy, or not recognised (Shi et al., 2006).  User’s direction of gaze In the original ART research formulation it was planned that a head mounted eye movement system would produce a vector representing the direction of user eye gaze with respect to the head.  In conjunction with this a magnetic system (Ascension Flock of Birds®) would be used to map both the spatial co-ordinates of the user in the environment, as well as the user’s head direction (head orientation vector).  By combining the eye vector with the orientation vector then a 3D vector would result representing the user’s overall direction of gaze from the known 3D location of the user.  Mapping this vector in space to the 3D location of ICT objects would then allow identification of whether the user was gazing at an object or not.  A somewhat simpler approach is fully to utilise the output from the ASL eye movement system.  This device has a miniature video camera mounted centrally on the user’s forehead which produces a video image of the scene in front of the user.  Eye gaze direction is monitored by the system and then software superimposes this as a symbol over the scene video.  Applying the ICT object detection algorithms directly to this video image then yields information about ICT objects present in the scene in front of the user which can be compared directly with the location of the user’s gaze without further knowledge of the 3D layout of such objects in the environment.    Interface operation If the user is gazing at a known ICT object then the appropriate interface for that object can be initiated.  The user can then decide whether to operate the interface or not.  Just what constitutes an appropriate interface for each ICT object and for different disabled users is a key issue under investigation.  Discussion The development of a system, based on head mounted eye movement monitoring, is described as a development vehicle for identifying ICT objects in the environment.  End user feedback on current interfaces and control systems for operating ICT devices is forming the basis for the ongoing development of the ART interfaces.  The current status of this ongoing research is that user trials with non-disabled people are underway to examine the performance factors of the head mounted ART system.  These will then progress to user trials with disabled users in our laboratory environment.  Conclusion A new technique to enable disabled users control a wide range of both simple electronic and complex ICT devices by monitoring their eye gaze behaviour is under development.  The system has distinct advantages over conventional existing selection and control systems.     Acknowledgements This research is supported by the ESRC PACCIT (People At the Centre of Communication and Information Technology) Programme.  References  COGAIN Communication by Gaze Interaction (2004) EU IST Network of Excellence  http://www.cogain.org/    EASTIN - European Assistive Technologies Information Network (2005) - http://www.eastin.info/ Gale A.G. (2005).  Attention responsive technology and ergonomics. In P.D.Bust & P.T. McCabe (Eds.) Contemporary Ergonomics 2005, (pp273-276) London, Taylor & Francis.  ISO 9999: 2002 Standard on ‘Technical aids for persons with disabilities. Classification and Terminology’  Jacob, R. J. (1990). What You Look at is What You Get: Eye Movement-Based Interaction Techniques. In Human Factors in Computing Systems, CHI ’90 Conference Proceedings, ACM Press, 11–18.  Lowe D.G. (2004) Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision. 60, 2, 91-110  Shi F., Gale, A.G., and Purdy K.P. (in press). Sift Approach Matching Constraints for Real-Time Attention Responsive System.  In Proceedings of the Fifth Asia Pacific International Symposium on Information Technology.   Ward D.J. and MacKay D.J.C. (2002). Fast hands free writing by gaze direction. Nature, August 22.   Ware, C. & Mikaelian, H. H. (1987). An evaluation of an eye tracker as a device for computer input, in J. M. Carroll & P. P. Tanner, (Eds.) CHI + GI 1987 Conference Proceedings, SIGCHI Bulletin, ACM, pp. 183-188. Special Issue.   Wooding, D.S., Mugglestone, M.D., Purdy, K.J. and Gale, A.G. (2002). Eye movements of large populations: I. Implementation and performance of an autonomous public eye tracker. Behavior Research Methods, Instruments, & Computers, 34, 509-517  Figures  Figure 1         Figure 2          Figure 3                        Figure 4    Figure Legends  Figure 1. X10 control system   Figure 2.  Possible complex menu interface for all devices   Figure 3 ART approach  - (a) the user simply looks directly at the device and (b) only the selected device operations are enabled on the interface   Figure 4.  Overview flow diagram of the ART system    

X10 - are you looking at me?

https://repository.lboro.ac.uk/articles/X10_-_are_you_looking_at_me_/9404132/files/17020817.pdf

X10 - are you looking at me?

Abstract

Similar works

Full text

Available Versions

Loughborough University Institutional Repository