Gender and gaze gesture recognition for human-computer interaction

© 2016 Elsevier Inc. The identification of visual cues in facial images has been widely explored in the broad area of computer vision. However theoretical analyses are often not transformed into widespread assistive Human-Computer Interaction (HCI) systems, due to factors such as inconsistent robustness, low efficiency, large computational expense or strong dependence on complex hardware. We present a novel gender recognition algorithm, a modular eye centre localisation approach and a gaze gesture recognition method, aiming to escalate the intelligence, adaptability and interactivity of HCI systems by combining demographic data (gender) and behavioural data (gaze) to enable development of a range of real-world assistive-technology applications. The gender recognition algorithm utilises Fisher Vectors as facial features which are encoded from low-level local features in facial images. We experimented with four types of low-level features: greyscale values, Local Binary Patterns (LBP), LBP histograms and Scale Invariant Feature Transform (SIFT). The corresponding Fisher Vectors were classified using a linear Support Vector Machine. The algorithm has been tested on the FERET database, the LFW database and the FRGCv2 database, yielding 97.7%, 92.5% and 96.7% accuracy respectively. The eye centre localisation algorithm has a modular approach, following a coarse-to-fine, global-to-regional scheme and utilising isophote and gradient features. A Selective Oriented Gradient filter has been specifically designed to detect and remove strong gradients from eyebrows, eye corners and self-shadows (which sabotage most eye centre localisation methods). The trajectories of the eye centres are then defined as gaze gestures for active HCI. The eye centre localisation algorithm has been compared with 10 other state-of-the-art algorithms with similar functionality and has outperformed them in terms of accuracy while maintaining excellent real-time performance. The above methods have been employed for development of a data recovery system that can be employed for implementation of advanced assistive technology tools. The high accuracy, reliability and real-time performance achieved for attention monitoring, gaze gesture control and recovery of demographic data, can enable the advanced human-robot interaction that is needed for developing systems that can provide assistance with everyday actions, thereby improving the quality of life for the elderly and/or disabled

Gaze-tracking-based interface for robotic chair guidance

This research focuses on finding solutions to enhance the quality of life for wheelchair users, specifically by applying a gaze-tracking-based interface for the guidance of a robotized wheelchair. For this purpose, the interface was applied in two different approaches for the wheelchair control system. The first one was an assisted control in which the user was continuously involved in controlling the movement of the wheelchair in the environment and the inclination of the different parts of the seat through the user’s gaze and eye blinks obtained with the interface. The second approach was to take the first steps to apply the device to an autonomous wheelchair control in which the wheelchair moves autonomously avoiding collisions towards the position defined by the user. To this end, the basis for obtaining the gaze position relative to the wheelchair and the object detection was developed in this project to be able to calculate in the future the optimal route to which the wheelchair should move. In addition, the integration of a robotic arm in the wheelchair to manipulate different objects was also considered, obtaining in this work the object of interest indicated by the user's gaze within the detected objects so that in the future the robotic arm could select and pick up the object the user wants to manipulate. In addition to the two approaches, an attempt was also made to estimate the user's gaze without the software interface. For this purpose, the gaze is obtained from pupil detection libraries, a calibration and a mathematical model that relates pupil positions to gaze. The results of the implementations have been analysed in this work, including some limitations encountered. Nevertheless, future improvements are proposed, with the aim of increasing the independence of wheelchair user

Rethinking Eye-blink: Assessing Task Difficulty through Physiological Representation of Spontaneous Blinking

Continuous assessment of task difficulty and mental workload is essential in improving the usability and accessibility of interactive systems. Eye tracking data has often been investigated to achieve this ability, with reports on the limited role of standard blink metrics. Here, we propose a new approach to the analysis of eye-blink responses for automated estimation of task difficulty. The core module is a time-frequency representation of eye-blink, which aims to capture the richness of information reflected on blinking. In our first study, we show that this method significantly improves the sensitivity to task difficulty. We then demonstrate how to form a framework where the represented patterns are analyzed with multi-dimensional Long Short-Term Memory recurrent neural networks for their non-linear mapping onto difficulty-related parameters. This framework outperformed other methods that used hand-engineered features. This approach works with any built-in camera, without requiring specialized devices. We conclude by discussing how Rethinking Eye-blink can benefit real-world applications.Comment: [Accepted version] In Proceedings of CHI Conference on Human Factors in Computing Systems (CHI '21), May 8-13, 2021, Yokohama, Japan. ACM, New York, NY, USA. 19 Pages. https://doi.org/10.1145/3411764.344557

Rethinking Eye-blink: Assessing Task Difficulty through Physiological Representation of Spontaneous Blinking

Continuous assessment of task difficulty and mental workload is essential in improving the usability and accessibility of interactive systems. Eye tracking data has often been investigated to achieve this ability, with reports on the limited role of standard blink metrics. Here, we propose a new approach to the analysis of eye-blink responses for automated estimation of task difficulty. The core module is a time-frequency representation of eye-blink, which aims to capture the richness of information reflected on blinking. In our first study, we show that this method significantly improves the sensitivity to task difficulty. We then demonstrate how to form a framework where the represented patterns are analyzed with multi-dimensional Long Short-Term Memory recurrent neural networks for their non-linear mapping onto difficulty-related parameters. This framework outperformed other methods that used hand-engineered features. This approach works with any built-in camera, without requiring specialized devices. We conclude by discussing how Rethinking Eye-blink can benefit real-world applications

A practical EMG-based human-computer interface for users with motor disabilities

In line with the mission of the Assistive Technology Act of 1998 (ATA), this study proposes an integrated assistive real-time system which affirms that technology is a valuable tool that can be used to improve the lives of people with disabilities . An assistive technology device is defined by the ATA as any item, piece of equipment, or product system, whether acquired commercially, modified, or customized, that is used to increase, maintain, or improve the functional capabilities of individuals with disabilities . The purpose of this study is to design and develop an alternate input device that can be used even by individuals with severe motor disabilities . This real-time system design utilizes electromyographic (EMG) biosignals from cranial muscles and electroencephalographic (EEG) biosignals from the cerebrum\u27s occipital lobe, which are transformed into controls for two-dimensional (2-D) cursor movement, the left-click (Enter) command, and an ON/OFF switch for the cursor-control functions . This HCI system classifies biosignals into mouse functions by applying amplitude thresholds and performing power spectral density (PSD) estimations on discrete windows of data. Spectral power summations are aggregated over several frequency bands between 8 and 500 Hz and then compared to produce the correct classification . The result is an affordable DSP-based system that, when combined with an on-screen keyboard, enables the user to fully operate a computer without using any extremities

Multimodality with Eye tracking and Haptics: A New Horizon for Serious Games?

The goal of this review is to illustrate the emerging use of multimodal virtual reality that can benefit learning-based games. The review begins with an introduction to multimodal virtual reality in serious games and we provide a brief discussion of why cognitive processes involved in learning and training are enhanced under immersive virtual environments. We initially outline studies that have used eye tracking and haptic feedback independently in serious games, and then review some innovative applications that have already combined eye tracking and haptic devices in order to provide applicable multimodal frameworks for learning-based games. Finally, some general conclusions are identified and clarified in order to advance current understanding in multimodal serious game production as well as exploring possible areas for new applications

An end-to-end review of gaze estimation and its interactive applications on handheld mobile devices

In recent years we have witnessed an increasing number of interactive systems on handheld mobile devices which utilise gaze as a single or complementary interaction modality. This trend is driven by the enhanced computational power of these devices, higher resolution and capacity of their cameras, and improved gaze estimation accuracy obtained from advanced machine learning techniques, especially in deep learning. As the literature is fast progressing, there is a pressing need to review the state of the art, delineate the boundary, and identify the key research challenges and opportunities in gaze estimation and interaction. This paper aims to serve this purpose by presenting an end-to-end holistic view in this area, from gaze capturing sensors, to gaze estimation workflows, to deep learning techniques, and to gaze interactive applications.PostprintPeer reviewe

Towards Computer-Assisted Regulation of Emotions

Tunteet ovat keskeinen ja erottamaton osa ihmisen toimintaa, ajattelua ja yksilöiden välistä vuorovaikutusta. Tunteet luovat perustan mielekkäälle, toimivalle ja tehokkaalle toiminnalle. Joskus tunteiden sävy tai voimakkuus voi kuitenkin olla epäedullinen henkilön tavoitteiden ja hyvinvoinnin kannalta. Tällöin taidokas tunteiden säätely voi auttaa saavuttamaan terveen ja menestyksellisen elämän. Väitöstyön tavoitteena oli muodostaa perusta tulevaisuuden tietokoneille, jotka auttavat säätelemään tunteita. Tietokoneiden tunneälyä on toistaiseksi kehitetty kahdella alueella: ihmisen tunnereaktioiden mittaamisessa ja tietokoneen tuottamissa tunneilmaisuissa. Viimeisimmät teknologiat antavat tietokoneille jo mahdollisuuden tunnistaa ja jäljitellä ihmisen tunneilmaisuja hyvinkin tarkasti. Väitöstyössä toimistotuoliin asennetuilla paineantureilla kyettiin huomaamattomasti havaitsemaan muutoksia kehon liikkeissä: osallistujat nojautuivat kohti heille esitettyjä tietokonehahmoja. Tietokonehahmojen esittämät kasvonilmeet ja kehollinen etäisyys vaikuttivat merkittävästi osallistujien tunne- ja tarkkaavaisuuskokemuksiin sekä sydämen, ihon hikirauhasten ja kasvon lihasten toimintaan. Tulokset osoittavat että keinotekoiset tunneilmaisut voivat olla tehokkaita henkilön kokemusten ja kehon toiminnan säätelyssä. Väitöstyössä laadittiin lopulta vuorovaikutteinen asetelma, jossa tunneilmaisujen automaattinen tarkkailu liitettiin tietokoneen tuottamien sosiaalisten ilmaisujen ohjaamiseen. Osallistujat pystyivät säätelemään välittömiä fysiologisia reaktioitaan ja tunnekokemuksiaan esittämällä tahdonalaisia kasvonilmeitä (mm. ikään kuin hymyilemällä) heitä lähestyvälle tietokonehahmolle. Väitöstyön tuloksia voidaan hyödyntää laajasti, muun muassa uudenlaisten, ihmisen luonnollisia vuorovaikutustapoja paremmin tukevien tietokoneiden suunnittelussa.Emotions are intimately connected with our lives. They are essential in motivating behaviour, for reasoning effectively, and in facilitating interactions with other people. Consequently, the ability to regulate the tone and intensity of emotions is important for leading a life of success and well-being. Intelligent computer perception of human emotions and effective expression of virtual emotions provide a basis for assisting emotion regulation with technology. State-of-the-art technologies already allow computers to recognize and imitate human social and emotional cues accurately and in great detail. For example, in the present work a regular looking office chair was used to covertly measure human body movement responses to artifical expressions of proximity and facial cues. In general, such artificial cues from visual agents were found to significantly affect heart, sweat gland, and facial muscle activities, as well as subjective experiences of emotion and attention. The perceptual and expressive capabilities were combined in a setup where a person regulated her or his more spontaneous reactions by either smiling or frowning voluntarily to a virtual humanlike character. These results highlight the potential of future emotion-sensitive technologies for creating supportive and even healthy interactions between humans and computers