Living in modern times, people at home are greatly enjoying the convenience brought about by advanced technologies. With steadily increased home automation applications, it is becoming more and more popular for individuals to use one central control interface to set up and operate of all the audio, video and many household appliances in a home. However, such interfaces often are too complicated for people with a disability to operate. However, the technology has long been available to achieve Environmental Control (EC) for disabled people with limited mobility, which then helps them live with more independence.
This paper presents a specially designed EC system for use by people who have lost significant mobility but who have good control of their eye movements. Through attention responsive technology, a user will be able to perform either simple or complex operations of any electrical household appliance by directly gazing at it

Alastair G. Gale (7148963)

Fangmin Shi (7153157)

Kevin Purdy (7149809)

English

Loughborough University Institutional Repository

 The 2nd Conference on Communication by Gaze Interaction – COGAIN 2006: Gazing into the Future  September 4-5, 2006  1  Turin, Italy Direct Gaze-Based Environmental Controls Fangmin Shi, Alastair Gale and Kevin Purdy Applied Vision Research Centre Loughborough University, Loughborough, UK {f.shi, a.g.gale, k.j.purdy}@lboro.ac.uk  Keywords Eye-tracking, environmental control, computer vision Introduction Living in modern times, people at home are greatly enjoying the convenience brought about by advanced technologies. With steadily increased home automation applications, it is becoming more and more popular for individuals to use one central control interface to set up and operate of all the audio, video and many household appliances in a home. However, such interfaces often are too complicated for people with a disability to operate. However, the technology has long been available to achieve Environmental Control (EC) for disabled people with limited mobility, which then helps them live with more independence. This paper presents a specially designed EC system for use by people who have lost significant mobility but who have good control of their eye movements. Through attention responsive technology, a user will be able to perform either simple or complex operations of any electrical household appliance by directly gazing at it. Current Environmental Control Technologies Under Home Automation, the Environmental Control is a system that allows the remote operation of electrical devices in the home surroundings. Although such system can be used regardless of a users’ physical ability, it is primarily designed to achieve more independent living for people with disability. Using an appropriate interface, a disabled user can independently control the room lighting, home entertainment system, air conditioning, or even open/close curtains, doors and operate intercoms.  An EC system should address the following two technological issues: 1. the input/output interface for users to operate The simplest input is by pressing buttons on a remote control such as for a typical TV control. Other input devices operated by hand can be a touch screen, switch, or a joystick. More advanced are by means of voice recognition or even eye tracking. The latter is one of the main challenges in the current research.  2. what wireless technology to adopt There exist three different types: ultrasound, infrared (IR) and radio frequency (RF). Each type uses the resource known from their names. An ultrasound system is not cost effective. IR and RF are currently very popular, even though conventional RF controlling requires additional wiring. The increasingly developed type is RF communication transmitted over the AC home power supply.  This makes use of existing electric outlets in the environment but can also switch on/off the power supply through a special RF communication protocol. It is also the type that is employed in this current research.   The 2nd Conference on Communication by Gaze Interaction – COGAIN 2006: Gazing into the Future  September 4-5, 2006  2  Turin, Italy A control unit, an input device, together with electric appliances form the three main components of an EC system. A typical example of such system can be illustrated by the SRS series (http://www.srstechnology.co.uk/). These provide a range of solutions for use with either hand-held devices or integration onto a wheelchair. Both IR and new RF control are available. Furthermore, there are a variety of input choices such as integral keys, keypad or joystick; all of which have to be referred to using either a key display or a LCD screen for a confirmative selection.  R&D of Attention Responsive Technology for Environmental Control Such assistive systems can be very useful for many mobility-restricted people as the simple selection operation, which can be as straightforward as keeping the joystick moving in order to select from a complicated menu. However, for people suffering from severe disability due to diseases such as Amyotrophic Lateral Sclerosis or Cerebral Palsy, then their ability to operate a simple interface with even a few items can be very restricted.  In recent years, eye movement research has made significant progress in developing eye gaze based input techniques for interacting with computers. Instead of using a normal keyboard and a mouse, systems such as Dasher (Ward 2002) have been successful in achieving eye gaze based word typing. Eye tracking research also has applications in Environmental Control. The Eyegaze System (Cleveland 1994) is such an example. However, the collected eye gaze data of such systems are usually relative to a computer monitor. In operation, a user selects a device by looking at its representative icon on the screen. Therefore, the selection is indirect. In addition, it can be very tiring for a severely disabled user to go through such a content-rich menu. This motivates the investigation of a new alternative EC input technique for severely disabled users - hence the current research project – Attention Responsive Technology (ART). The aim of the project is to establish an EC approach by using an individual’s eye gaze as a direct input method. In use - a user gazes at a real household appliance and his/her gaze attention will turn into an action for the device. For instance, if a user wants to turn on a light, it can be automatically achieved currently by looking at the light in three-dimensional space for a second or less. This is achieved by applying the following methodologies: ● Computer vision: the user’s living environment and eye movements are monitored in real time. ● Object recognition: based upon the user’s gaze direction, any controllable devices are detected.   ● Device control: X10 technology enables the remote control by plugging in simple modules to outlets. ● Selection interface: simple ON/OFF menu on a touch sensitive screen or via simple switch selection. Computer vision and Object recognition A head mounted eye tracking system is firstly employed in the project. With the user wearing a head band, two fixed cameras (an eye camera tracing eye movements and a scene camera monitoring the front view) work together to give the output of eye pupil co-ordinates relative to their positions in the scene image. This needs the system calibration in advance. The prototype system setup is described in Shi et al. (2006-1). Certain criteria are also established to mark an eye gaze from a set of eye movement data obtained at the frequency of 50Hz (Shi et al., 2006-2). Upon any gaze, object recognition  using the SIFT approach (Lowe, 2004) is activated with a view to finding a match in the real time scene image with any pre-known device, these devices are imaged in advance and saved in the image database. Figure 1 (a.& b.) show the resultant images of a lamp when it is in the OFF and ON status, respectively. The highlighted squares are the found SIFT matches with reference to one of the database images. The ‘+’ signs are the detected eye gaze co-ordinates from the gaze analysis.  The 2nd Conference on Communication by Gaze Interaction – COGAIN 2006: Gazing into the Future  September 4-5, 2006  3  Turin, Italy The advantages of the SIFT approach to object matching are that it is not only invariant to scale and illumination but also stable in the case of occlusion. Figure 1 (c) is a correctly recognized result of a captured image when a visitor was partly blocking the fan.            a.  b.  c.  Figure 1. Objects captured by directly looking at them For severely disabled users, the continual use of a head mounted eye tracker can be problematic. Consequently, the use of a remote eye tracker - the Smart Eye system (http://www.smarteye.se) is being investigated. This consists of two or three cameras which form a computer vision system and this measures the user's head position and orientation as well as gaze direction. A research challenge is how best to place the scene camera in order to relate the eye gaze data to where they are looking in the recorded scene image. Device control X10 technology (http://www.simplyautomate.co.uk/GuideToX10.asp?Cat1=82) is employed in the ART remote control. X10 is a wireless communication protocol which allows short digital messages to be sent and received over existing wiring. Only simple and cheap X10 modules are needed to be plugged between each electric device and its mains power supply. Each device is assigned a unique address. Each command then addresses one or more devices with an action such as ON and OFF and the module will accordingly operate the device. Although over the past decade many other home automation technologies have come into use, X10 has been expanding because of its low cost and the key feature that it uses existing electrical wiring. Selection interface Although the ART project aims to develop a direct eye-operated control interface, the word ‘direct’ must be understood in terms of the eye tracking with reference to real devices placed in 3D space. As a matter of fact, a gaze at a real appliance will be confirmed in ART by introducing a simple pop-up menu with say only two items, ON or OFF, on a touch sensitive screen. This is also the main feature that distinguishes the ART system with other ‘always-on’ eye tracking control systems, which can trigger numerous wrong actions for random gazes. Depending on a user’s requirement, the touch screen can be replaced by any type of switch input. Results Currently the ART system is PC-based. It has a simulation interface as shown in Figure 2. The system has integrated the whole process including the system calibration, eye gaze data analysis, objection recognition, X10 control and selection, and intermediate result display with a list of gaze control history. An eye gaze can be obtained stably from 50 eye movement data points, consuming 1s (less time can be easily used). It takes less than 2s for determining if a gaze falls on any found object with the current Matlab-driven software.  The 2nd Conference on Communication by Gaze Interaction – COGAIN 2006: Gazing into the Future  Figure 2. Current ART user interface Conclusion and Future Plan The ART system based on the head mounted eye tracker has demonstrated the successful process of using eye gaze to address a control directly. The current tested household appliances are limited but can be easily extended to as many as a user requires. As a PC-based system with no need for extra electrical wiring, it can be very handy for the ART components to integrate into a wheelchair Environmental Control system using other control technologies such as Infra Red.     We are currently progressing to a remote eye system, which has low cost and does not need any attachment to the user.  However, the other modules discussed above will remain the same as the head mounted one. Although the SIFT approach to object recognition has shown its powerfulness, to apply a more stable 3D object recognition in a real environment with a complicated background and ambient light will still be the core next stage of the research programme.  Acknowledgements This research is funded by the UK ESRC PACCIT Programme and also part of the COGAIN network. References Cleveland N.R. (1994). Eyegaze human-computer interface for people with disabilities. First Automation Technology and Human Performance Conference, Washington, DC, April 7-8, 1994.  Lowe D.G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2, pp. 91-110. Shi, F., Gale, A.G. & Purdy, K.J. (2006). Eye-centric ICT control. In Bust P.D. & McCabe P.T. (Eds.) Contemporary Ergonomics 2006, pp. 215-218. Shi, F., Gale, A.G. & Purdy, K.J. (2006). Helping people with ICT device control by eye gaze. 10th International Conference on Computers for Helping People with Special Needs, Austria, to be presented on 12-14 July 2006. pp. 480-487. Ward, D. J. & MacKay, D. J. C. (2002). Fast hands-free writing by gaze direction. Nature, 418, 838  September 4-5, 2006  4  Turin, Italy 

Direct gaze based environmental controls

https://repository.lboro.ac.uk/articles/Direct_gaze_based_environmental_controls/9404144/files/17020829.pdf

Direct gaze based environmental controls

Abstract

Similar works

Full text

Available Versions

Loughborough University Institutional Repository