The Attention Responsive Technology (ART) system is designed to enable control of the environment by individuals for whom movement is difficult or undesirable. This paper reports additional development of the ART system through replacing its initial head-mounted eye-tracking technology with a remotely mounted tracking system. The new system can release the user from the need to wear any head-mounted equipment, thus improving user comfort and acceptability. Instead, eye tracking cameras and the scene camera are situated in a fixed position some small distance from the user; these then allow tracking of the user’s eye gaze and field of view, respectively. This system would suit many situations in which the user remains seated, for example, in a wheelchair or before a workstation onto which the cameras can be mounted

Alastair G. Gale (7148963)

Fangmin Shi (7153157)

Kevin Purdy (7149809)

English

Loughborough University Institutional Repository

EXPLORING EYE RESPONSIVE CONTROL – FROM A HEAD MOUNTED TO A REMOTE SYSTEM   Fangmin Shi, Alastair Gale and Kevin Purdy   Applied Vision Research Centre Loughborough University Loughborough LE11 3TU      The Attention Responsive Technology (ART) system is designed to enable control of the environment by individuals for whom movement is difficult or undesirable. This paper reports additional development of the ART system through replacing its initial head-mounted eye-tracking technology with a remotely mounted tracking system. The new system can release the user from the need to wear any head-mounted equipment, thus improving user comfort and acceptability. Instead, eye tracking cameras and the scene camera are situated in a fixed position some small distance from the user; these then allow tracking of the user’s eye gaze and field of view, respectively. This system would suit many situations in which the user remains seated, for example, in a wheelchair or before a workstation onto which the cameras can be mounted.   Introduction  Environmental control technologies in the field of Home Automation can play an important role in allowing disabled people to achieve a more independent lifestyle. The benefits for individuals with restricted movement can be great, but success is heavily dependent on the provision of a suitable user interface to enable interaction appropriate to their needs and capabilities. Therefore, research into eye movement control techniques is of particular interest to individuals with very limited mobility. We have previously reported our innovative design of the Attention Responsive Technology (ART) system, an eye-centric object controller which uses a head-mounted eye tracker and can enable an appropriate interface for control of any device, such as a lamp or a curtain, by directly gazing at it followed by a user configured input for initiation (Shi et al., 2006a). Compared with conventional environmental controls that utilise eye interaction with representative icons displayed on a 2-D computer screen our method has distinct advantages. Looking directly at the objects to be controlled is an extremely intuitive form of user interaction and by using this approach the system does not inherently need the user to sit incessantly before a monitor, making it suitable for use in a wider range of situations and by users with a range of disabilities. For example, it immediately removes the need to distinguish small icons or words on a monitor.   However, one drawback of the current system is that prolonged wearing of the eye tracker on the head can potentially cause user discomfort and fatigue over a period of time. The application of the ART system is such that many users would require its operation for such extended periods, as it will facilitate autonomy throughout their everyday life, so comfort is an important factor. To truly integrate the system into other assistive technologies, it would be hard to imagine that the user would like to wear the head-mounted eye tracker all the time. Another factor is that users with highly restricted mobility can often be understandably concerned about the impact new technology and situations may have upon them so, for reasons of user-acceptability and ease of interaction too, a less physically invasive system would be a distinct advantage. Eye tracking and assistive technologies are addressed within the COGAIN (2006) European research Network of Excellence which aims to empower users with disabilities to communicate with others, including computers and their environment, by means of their eye gaze. There exist a number of research projects, like ART, in the network that either develop or make use of remote eye trackers. For example, the Tobii system just requires its user to sit in front of the computer screen and the user can carry out many PC applications by eye gaze such as typing/editing documents, browsing the internet or even playing games.  To facilitate users of our system, there is clearly a need for developing a remote eye tracker for environmental control.   Attention Responsive Technology (ART)  The concept of the ART project is to analyse the user’s gaze behaviour while s/he is attending to a normal household appliance (or other controllable device) and to respond by offering appropriate controls necessary for them to execute any appropriate control function for that specific appliance only. The user control interface therefore remains simple in nature, offering greater scope for easy customisation as appropriate to the needs and abilities of the user.  For instance, when a user wants to switch on a fan, what s/he does is simply to look at the fan.  His/her eye gaze behaviour is recorded and a stable gaze point derived as shown in Figure 1(a). This gaze information is then analysed to determine whether or not it falls on any device (Figure 1b). A simple interface dialogue, illustrated in Figure 1(c), then appears on a display, asking for the user to make his/her control input (by various means) and the system then implements the control action. In order for the system to work appropriately a valid user calibration must first be performed (Shi, 2006a).                   a. Stable eye gaze captured b. Gaze on device detected c. Control initiated ON OFF ⊕ Figure 1. Typical stages of the ART system   Head-mounted eye tracking   The major development of the ART system using the head-mounted ASL eye tracker has been described in detail in previous papers (Shi et al., 2006a and Shi et al., 2006b). Essentially the user’s eye gaze is recorded by a head-mounted system and their gaze location within the field of view of a head-mounted scene camera (which images the scene in front of the user) determined.  If the user’s gaze falls upon a device automatically recognised from the scene camera image then the user is offered an appropriate control interface.    Remote eye tracking  Remote head and gaze tracking is achieved by using the Smart Eye Pro system. Up to five cameras can be placed in front of the user to cover a wide range of eye or head movements.  The system measures the user’s head position and line of gaze in real time. All the measurements are with referenced to a pre-defined three dimensional world coordinate system. The scene camera system is positioned such that it replicates the user's view of their environment and facilitates object recognition of controllable devices within their sight.  This configuration would suit many situations in which the user remains seated relative to a suitable mounting platform, for example, before a workstation or in a wheelchair on to which the three cameras can be fixed.  The current remote ART system development employs three cameras. These can be positioned completely independently from each other. However, in order to achieve the maximum system performance, each camera is oriented such that the user’s head is at the centre of their respective images while the user is seated in a natural position. Thus the system can accommodate different postures and heights of users simply by pointing the cameras at wherever their face happens to be. If the user moves around to some extent, or if multiple users must use the system without individual camera adjustment being practicable, it is possible to accommodate this by instead aiming the cameras to cover a range of likely movements and head positions, although some loss of accuracy or the need for more eye cameras may result. The relative camera position can then be calibrated by an automatic process using a calibration chart. This is possible even if the cameras are mounted on a mobile wheelchair, as it is their relative, rather than absolute, position that is important.  A World Coordinate System (WCS) needs to be defined in the system. Automatically, the origin can be either the first eye camera or the centre of a ‘chessboard’ calibration chart. Alternatively, any known real-world point can be marked as the WCS origin. In the case of a mobile platform such as a wheelchair upon which the cameras are mounted, the WCS will therefore be relative to, and so move about with, the wheelchair.  There is a difference in the method of determining the user’s eye gaze information between the remote and head-mounted trackers. The latter system detects the pupil outline and corneal reflection in the IR illuminated eye image, essentially determining gaze from the movement and position of the eyeball alone whereas, the Smart Eye tracker utilises the user’s profile, which is a virtual 3D model of the user’s head. For each new user a personal profile has to be created using a fairly automated procedure. This is a collection of ‘snapshots’ taken from different angles and with different cameras, utilising marked facial feature points such as eye centres, inner/outer eye corners, inner/outer eye brows, nostrils, mouth corners and ears. These marked features are also used as a template in  real time feature tracking, and head pose and  eye gaze direction are calculated based upon them.  Once calibration has been performed then the system remotely tracks the user’s eye gaze.  As with the head mounted approach the user’s eye gaze vector must be related to devices in the environment which are automatically detected by the ART system. To do this a scene camera needs to be integrated with the remote system.  Such a scene camera could be head mounted - although this revisits the possible disadvantages already discussed for the head mounted eye tracker. Therefore it could be mounted, in a slaved fashion, alongside the user’s head (Figure 2a) or mounted with the various system cameras (Figure 2b). If the scene camera is not head mounted it must still cover the scene which the user can reasonably view, or that part of the user’s scene over which eye position can successfully be tracked, whichever is the smaller.  Whilst the former position would more accurately map the user’s viewpoint on their world and so has advantages in relating this (and so the devices within it) to their point of gaze, the latter is mathematically convenient in locating the viewpoint close to the origin of the WCS.                               ??? - 3 eye tracking cameras???? ??Scene camera a. Scene camera by side of head b. Scene camera by the middle eye camera   Figure 2. Two ways of positioning the scene camera with the remote eye tracker  Suitable calibration then relates the scene image co-ordinate system and the already defined World Coordinate System which is made in a similar way to that performed with the ASL system (Shi et al., 2006a). That process is to take the image of the calibration targets, extract the coordinates of the targets by means of object recognition procedures, and then mathematically project the line of gaze in WCS to the scene image co-ordinate system.  After the system calibration is completed, all other operations of the object identification and the device initiation are largely identical to those used with the head mounted system. Figure 3 illustrates the result of such remote tracking, which shows that when the line of gaze falls on a device (identified here as a lamp while it was switched on). In this situation the user interface would enable a simple on/off control function for the user to operate if desired, though more complex automated objects would present more control options.           Figure 3. Object identified by the ART remote tracking system Discussion   The development of the remote ART implementation here has mainly focused on its system calibration, which is where the main differences and added complexity lie in comparison to the head-mounted ART system. The benefits that the remote system offers in terms of ease of use, comfort and acceptability to the user make this approach valuable, and so worthy of the added complexity during the system development. For many users who suffer from restricted movement, such improvements could be a crucial factor in the success of such a system.   The head mounted system was valuable as an initial development tool and in proving the concept of the project. However there remain some situations in which it would also be more appropriate, specifically where greater movement was possible or desirable.   Some users, who have limited body movement, may still be able to turn their head further than is desirable with a three-camera remote eye tracking system. As this may be one of the few actions available to them the ART system would need to accommodate this. Further work therefore may include widening of the field of view over which the system operates through extending the Smart Eye system and the range over which the scene camera (and therefore object recognition system) will perform. There are challenges in both of these, but again if successful it would reduce even further the circumstances in which users would need to wear a head-mounted tracking system.  The programming work to develop the current system is mainly done in Matlab, however, it also interfaces with eye tracking functions written in Microsoft Visual C++. Due to the system supplier’s limited release, part of the calibration process must rely on the system’s own interface software rather than being properly integrated into the ART system’s software. Consequently the remote system integration is not currently as complete as it was when using the ASL system and this must be overcome before a stand-alone system can be fully achieved. Once the remote tracking system is integrated properly into the ART system then user trials will be undertaken.    Conclusions The further development of the ART system to make use of remote eye tracking is described.  This offers increased usability advantages to end users however some technical issues need to be first addressed before full user trials can be undertaken.   Acknowledgement The research is funded by the PACCIT programme.   References  COGAIN, Communication by Gaze Interaction, 2006, http://www.cogain.org/Shi F., Gale A.G. & Purdy K.J. 2006a, Eye-centric ICT control. In Bust P.D. & McCabe P.T. (Eds.) Contemporary Ergonomics 2006, (Taylor and Francis, London), 215-218. Shi F., Gale A.G. & Purdy K.J. 2006b, Helping People with ICT Device Control by Eye Gaze, In Miesenberger K., Klaus J., Zagler W. & Karshmer A. (Eds.) Lecture Notes in Computer Science (Springer Verlag Berlin), pp 480-487.   

Exploring eye responsive control - from a head mounted to a remote system

https://repository.lboro.ac.uk/articles/Exploring_eye_responsive_control_-_from_a_head_mounted_to_a_remote_system/9403844/files/17020514.pdf

Exploring eye responsive control - from a head mounted to a remote system

Abstract

Similar works

Full text

Available Versions

Loughborough University Institutional Repository