The use of tongue protrusion gestures for video-based communication

We propose a system that tracks the mouth region in a video sequence and detects the occurrence of a tongue protrusion event. Assuming that the natural location of the tongue is inside the mouth, the tongue protrusion gesture is interpreted as an intentional communication sign that the user wishes to perform a task. The system operates in three steps: (1) mouth template segmentation, in which we initialize one template for the entire mouth, and one template for each of the left and right halves of the mouth; (2) mouth region tracking using the Normalized Correlation Coefficient (NCC), and (3) tongue protrusion event detection and interpretation. We regard the tongue protrusion transition as the event that begins when a minimum part of the tongue starts protruding from the mouth, and which ends when the protrusion is clearly visible. The left and right templates are compared to their corresponding halves for each new mouth image that has been tracked, and a left-NCC and a right-NCC are obtained for each part. By analyzing the NCCs during the tongue protrusion transition time, the left or right position of the protrusion, relative to the center of the mouth, is determined. We analyze our proposed communication method and demonstrate that it adapts easily to different users. The detection of this gesture can be used for instance as a dual-switch hand-free human-computer interface for granting control of a computer

Detection of tongue protrusion gestures from videos

We propose a system that, using video information, segments the mouth region from a face image and then detects the protrusion of the tongue from inside the oral cavity. Initially, under the assumption that the mouth is closed, we detect both mouth corners. We use a set of specifically oriented Gabor filters for enhancing horizontal features corresponding to the shadow existing between the upper and lower lips. After applying the Hough line detector, the extremes of the line that was found are regarded as the mouth corners. Detection rate for mouth corner localization is 85.33%. These points are then input to a mouth appearance model which fits a mouth contour to the image. By segmenting its bounding box we obtain a mouth template. Next, considering the symmetric nature of the mouth, we divide the template into right and left halves. Thus, our system makes use of three templates. We track the mouth in the following frames using normalized correlation for mouth template matching. Changes happening in the mouth region are directly described by the correlation value, i.e., the appearance of the tongue in the surface of the mouth will cause a decrease in the correlation coefficient through time. These coefficients are used for detecting the tongue protrusion. The right and left tongue protrusion positions will be detected by analyzing similarity changes between the right and left half-mouth templates and the currently tracked ones. Detection rates under the default parameters of our system are 90.20% for the tongue protrusion regardless of the position, and 84.78% for the right and left tongue protrusion positions. Our results demonstrate the feasibility of real-time tongue protrusion detection in vision-based systems and motivates further investigating the usage of this new modality in human-computer communication

Mouth region localization based on Gabor features and active appearance models

No abstract availabl