Yawn analysis with mouth occlusion detection

tOne of the most common signs of tiredness or fatigue is yawning. Naturally, identification of fatiguedindividuals would be helped if yawning is detected. Existing techniques for yawn detection are centred onmeasuring the mouth opening. This approach, however, may fail if the mouth is occluded by the hand, as itis frequently the case. The work presented in this paper focuses on a technique to detect yawning whilstalso allowing for cases of occlusion. For measuring the mouth opening, a new technique which appliesadaptive colour region is introduced. For detecting yawning whilst the mouth is occluded, local binarypattern (LBP) features are used to also identify facial distortions during yawning. In this research, theStrathclyde Facial Fatigue (SFF) database which contains genuine video footage of fatigued individuals isused for training, testing and evaluation of the system

Analysis of yawning behaviour in spontaneous expressions of drowsy drivers

Driver fatigue is one of the main causes of road accidents. It is essential to develop a reliable driver drowsiness detection system which can alert drivers without disturbing them and is robust to environmental changes. This paper explores yawning behaviour as a sign of drowsiness in spontaneous expressions of drowsy drivers in simulated driving scenarios. We analyse a labelled dataset of videos of sleep-deprived versus alert drivers and demonstrate the correlation between hand-over-face touches, face occlusions and yawning. We propose that face touches can be used as a novel cue in automated drowsiness detection alongside yawning and eye behaviour. Moreover, we present an automatic approach to detect yawning based on extracting geometric and appearance features of both mouth and eye regions. Our approach successfully detects both hand-covered and uncovered yawns with an accuracy of 95%. Ultimately, our goal is to use these results in designing a hybrid drowsiness-detection system

Algorithm for Monitoring Head/Eye Motion for Driver Alertness with one Camera

Visual methods and systems are described for detecting alterness and vigilance of persons under the conditions of fatigue, lack of sleep, and exposure to mind altering substances such as alcohol or drugs. In particular, the invention can have particular application for truck drivers, bus drivers, train operators, pilots and watercraft controllers and stationary heavy equipment operators, and students and employees during either daytime or nighttime conditions. The invention robustly tracks a person\u27s head and facial features with a single on-board camera with a fully automatic system, that can intitalize automatically, and can reinitialize when it needs to and provide outputs in realtime. The system can classify rotation in all viewing directions, detects eye/mouth occlusion, detects eye blinking, and recovers the 3D (three dimensional) gaze of the eyes. In addition, the system is able to track both through occlusion like eye blinking and through occlusion like rotation. Outputs can be visual and sound alarms to the driver directly..

Development of Yawning Detection Algorithm for Normal Lighting Condition and IR Condition

Drowsiness monitoring system has been widely used in this current technology to monitor the driver’s state while driving. This paper presents a drowsiness detection method through the activity of yawning for both normal lighting condition and Infrared (IR) condition. Development of the algorithm consists of several steps. Initially, the detection of face and mouth implementing the Viola-Jones algorithm takes place. For IR condition, the mouth is detected by applying the geometrical measurements of a face. After the detection process is done, the tracking process for both face and mouth takes place utilizing the Kanade-Lucas-Tomasi (KLT) algorithm which is basically a point tracking algorithm. Based on the tracked mouth, the region of interest (ROI) is selected which is to be used as an input image in the image processing step in order to get a clearer image of the mouth. From the finalized mouth image in the preprocessing step, the properties of the image are further used in the yawning detection step. In the indication of yawning, the height of the mouth opening reading score is observed. The performance of the proposed method is tested on 5 subjects and achieved an overall accuracy of 98.89% for normal lighting condition and 95.29% for IR condition

Robust Modeling of Epistemic Mental States and Their Applications in Assistive Technology

This dissertation presents the design and implementation of EmoAssist: Emotion-Enabled Assistive Tool to Enhance Dyadic Conversation for the Blind . The key functionalities of the system are to recognize behavioral expressions and to predict 3-D affective dimensions from visual cues and to provide audio feedback to the visually impaired in a natural environment. Prior to describing the EmoAssist, this dissertation identifies and advances research challenges in the analysis of the facial features and their temporal dynamics with Epistemic Mental States in dyadic conversation. A number of statistical analyses and simulations were performed to get the answer of important research questions about the complex interplay between facial features and mental states. It was found that the non-linear relations are mostly prevalent rather than the linear ones. Further, the portable prototype of assistive technology that can aid blind individual to understand his/her interlocutor\u27s mental states has been designed based on the analysis. A number of challenges related to the system, communication protocols, error-free tracking of face and robust modeling of behavioral expressions /affective dimensions were addressed to make the EmoAssist effective in a real world scenario. In addition, orientation-sensor information from the phone was used to correct image alignment to improve the robustness in real life deployment. It was observed that the EmoAssist can predict affective dimensions with acceptable accuracy (Maximum Correlation-Coefficient for valence: 0.76, arousal: 0.78, and dominance: 0.76) in natural conversation. The overall minimum and maximum response-times are (64.61 milliseconds) and (128.22 milliseconds), respectively. The integration of sensor information for correcting the orientation has helped in significant improvement (16% in average) of accuracy in recognizing behavioral expressions. A user study with ten blind people shows that the EmoAssist is highly acceptable to them (Average acceptability rating using Likert: 6.0 where 1 and 7 are the lowest and highest possible ratings, respectively) in social interaction

A CNN-LSTM-based Deep Learning Approach for Driver Drowsiness Prediction

Abstract: The development of neural networks and machine learning techniques has recently been the cornerstone for many applications of artificial intelligence. These applications are now found in practically all aspects of our daily life. Predicting drowsiness is one of the most particularly valuable of artificial intelligence for reducing the rate of traffic accidents. According to earlier studies, drowsy driving is at responsible for 25 to 50% of all traffic accidents, which account for 1,200 deaths and 76,000 injuries annually. The goal of this research is to diminish car accidents caused by drowsy drivers. This research tests a number of popular deep learning-based models and presents a novel deep learning-based model for predicting driver drowsiness using a combination of convolutional neural networks (CNN) and Long-Short-Term Memory (LSTM) to achieve results that are superior to those of state-of-the-art methods. Utilizing convolutional layers, CNN has excellent feature extraction abilities, whereas LSTM can learn sequential dependencies. The National Tsing Hua University (NTHU) driver drowsiness dataset is used to test the model and compare it to several other current models as well as state-of-the-art models. The proposed model outperformed state-of-the-art models, with results up to 98.30% for training accuracy and 97.31% for validation accuracy

Facial Drowsiness Signs Detection Algorithm Using Image Processing Techniques For Various Lighting Condition

For the past few years, drowsiness signs detection systems have been developed as one of the initiative to reduce car crashes. However, various luminance intensities are one of the major problems in the development of a drowsiness signs detection system. This research studies the suitable image processing techniques to be implemented in a drowsiness signs detection algorithm for various lighting conditions. Four lighting conditions are proposed with the average range of 0 luminance value to 175 luminance value. In this project, the algorithm is developed based on four main algorithms which are the detection algorithm, the tracking algorithm, the preprocessing algorithm and the drowsiness signs analysis algorithm. Viola-Jones algorithm is utilized for face detection. Upon acquiring the face location, the knowledge-based method is implemented to locate the eye and the mouth. After that, Kanade Lucas Tomasi algorithm is applied for tracking purpose. Based on the tracked face and the tracked facial components, the region of interest is selected. Image processing techniques are applied to the eye region and the mouth region to fix the image intensity and to enhance the features of the image. In order to analyse the drowsiness signs portrayed by the eye and the mouth, the operation to determine the eye state and the mouth state is determined. The distance between eyelid is computed to determine the eye state. Meanwhile, the height of the mouth opening is computed to determine the mouth state. There are three drowsiness signs that are analysed for the eye region, namely, the eye blink count, the duration of the eye closure and the percentage of time that the eye is closed. As for the drowsiness sign in the mouth region, the yawning count is computed. This thesis presents a small-scale drowsiness signs database for four lighting conditions. The performance of the algorithm is validated by using the developed database under four luminance intensities and achieved promising results. The performance of the drowsiness signs detection algorithm is fully dependent on the performance of the eye state detection and the mouth state detection. For eye state detection, the proposed technique possessed an accuracy of 98.71 % for 0 luminance value, 97.10 % for 2 luminance value, 98.30 % for 5.2 luminance value and 98.8 % for 174.9 luminance value. As for mouth detection, the proposed technique possessed an accuracy of 99.45 % for 0 luminance value, 98.03 % for 2 luminance value, 99.6 for 5.2 luminance value and 99.7 % for 174.9 luminance value. The proposed technique yielded the overall accuracy of 98.22% for eye state detection and the overall accuracy of 99.23% for the mouth state detection. In conclusion, the proposed technique managed to yield high accuracy for four lighting conditions and could be improved for further research to be implemented in a real time environment

A CNN-LSTM-based Deep Learning Approach for Driver Drowsiness Prediction

Abstract: The development of neural networks and machine learning techniques has recently been the cornerstone for many applications of artificial intelligence. These applications are now found in practically all aspects of our daily life. Predicting drowsiness is one of the most particularly valuable of artificial intelligence for reducing the rate of traffic accidents. According to earlier studies, drowsy driving is at responsible for 25 to 50% of all traffic accidents, which account for 1,200 deaths and 76,000 injuries annually. The goal of this research is to diminish car accidents caused by drowsy drivers. This research tests a number of popular deep learning-based models and presents a novel deep learning-based model for predicting driver drowsiness using a combination of convolutional neural networks (CNN) and Long-Short-Term Memory (LSTM) to achieve results that are superior to those of state-of-the-art methods. Utilizing convolutional layers, CNN has excellent feature extraction abilities, whereas LSTM can learn sequential dependencies. The National Tsing Hua University (NTHU) driver drowsiness dataset is used to test the model and compare it to several other current models as well as state-of-the-art models. The proposed model outperformed state-of-the-art models, with results up to 98.30% for training accuracy and 97.31% for validation accuracy