DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation

There is an undeniable communication barrier between deaf people and people with normal hearing ability. Although innovations in sign language translation technology aim to tear down this communication barrier, the majority of existing sign language translation systems are either intrusive or constrained by resolution or ambient lighting conditions. Moreover, these existing systems can only perform single-sign ASL translation rather than sentence-level translation, making them much less useful in daily-life communication scenarios. In this work, we fill this critical gap by presenting DeepASL, a transformative deep learning-based sign language translation technology that enables ubiquitous and non-intrusive American Sign Language (ASL) translation at both word and sentence levels. DeepASL uses infrared light as its sensing mechanism to non-intrusively capture the ASL signs. It incorporates a novel hierarchical bidirectional deep recurrent neural network (HB-RNN) and a probabilistic framework based on Connectionist Temporal Classification (CTC) for word-level and sentence-level ASL translation respectively. To evaluate its performance, we have collected 7,306 samples from 11 participants, covering 56 commonly used ASL words and 100 ASL sentences. DeepASL achieves an average 94.5% word-level translation accuracy and an average 8.2% word error rate on translating unseen ASL sentences. Given its promising performance, we believe DeepASL represents a significant step towards breaking the communication barrier between deaf people and hearing majority, and thus has the significant potential to fundamentally change deaf people's lives

Robust Signal Processing Techniques for Wearable Inertial Measurement Unit (IMU) Sensors

Activity and gesture recognition using wearable motion sensors, also known as inertial measurement units (IMUs), provides important context for many ubiquitous sensing applications including healthcare monitoring, human computer interface and context-aware smart homes and offices. Such systems are gaining popularity due to their minimal cost and ability to provide sensing functionality at any time and place. However, several factors can affect the system performance such as sensor location and orientation displacement, activity and gesture inconsistency, movement speed variation and lack of tiny motion information. This research is focused on developing signal processing solutions to ensure the system robustness with respect to these factors. Firstly, for existing systems which have already been designed to work with certain sensor orientation/location, this research proposes opportunistic calibration algorithms leveraging camera information from the environment to ensure the system performs correctly despite location or orientation displacement of the sensors. The calibration algorithms do not require extra effort from the users and the calibration is done seamlessly when the users present in front of an environmental camera and perform arbitrary movements. Secondly, an orientation independent and speed independent approach is proposed and studied by exploring a novel orientation independent feature set and by intelligently selecting only the relevant and consistent portions of various activities and gestures. Thirdly, in order to address the challenge that the IMU is not able capture tiny motion which is important to some applications, a sensor fusion framework is proposed to fuse the complementary sensor modality in order to enhance the system performance and robustness. For example, American Sign Language has a large vocabulary of signs and a recognition system solely based on IMU sensors would not perform very well. In order to demonstrate the feasibility of sensor fusion techniques, a robust real-time American Sign Language recognition approach is developed using wrist worn IMU and surface electromyography (EMG) sensors

Robust Signal Processing Techniques for Wearable Inertial Measurement Unit (IMU) Sensors

Activity and gesture recognition using wearable motion sensors, also known as inertial measurement units (IMUs), provides important context for many ubiquitous sensing applications including healthcare monitoring, human computer interface and context-aware smart homes and offices. Such systems are gaining popularity due to their minimal cost and ability to provide sensing functionality at any time and place. However, several factors can affect the system performance such as sensor location and orientation displacement, activity and gesture inconsistency, movement speed variation and lack of tiny motion information. This research is focused on developing signal processing solutions to ensure the system robustness with respect to these factors. Firstly, for existing systems which have already been designed to work with certain sensor orientation/location, this research proposes opportunistic calibration algorithms leveraging camera information from the environment to ensure the system performs correctly despite location or orientation displacement of the sensors. The calibration algorithms do not require extra effort from the users and the calibration is done seamlessly when the users present in front of an environmental camera and perform arbitrary movements. Secondly, an orientation independent and speed independent approach is proposed and studied by exploring a novel orientation independent feature set and by intelligently selecting only the relevant and consistent portions of various activities and gestures. Thirdly, in order to address the challenge that the IMU is not able capture tiny motion which is important to some applications, a sensor fusion framework is proposed to fuse the complementary sensor modality in order to enhance the system performance and robustness. For example, American Sign Language has a large vocabulary of signs and a recognition system solely based on IMU sensors would not perform very well. In order to demonstrate the feasibility of sensor fusion techniques, a robust real-time American Sign Language recognition approach is developed using wrist worn IMU and surface electromyography (EMG) sensors

American Sign Language Recognition System by Using Surface EMG Signal

Sign Language Recognition (SLR) system is a novel method that allows hard of hearing to communicate with general society. In this study, American Sign Language (ASL) recognition system was proposed by using the surface Electromyography (sEMG). The objective of this study is to recognize the American Sign Language alphabet letters and allow users to spell words and sentences. For this purpose, sEMG data are acquired from subject right forearm for twenty-seven American Sign Language gestures of twenty-six English alphabets and one for home position. Time and frequency domain (band power) information used in the feature extraction process. As a classification method, Support Vector Machine and Ensemble Learning algorithm were used and their performances are compared with tabulated results. In conclusion, the results of this study show that sEMG signal can be used for SLR systems

Using High Density EMG to Proportionally Control 3D Model of Human Hand

Control of human hand using surface electromyography (EMG) is already established in various mechanisms, but proportionally controlling magnitudes degrees of freedom (DOF) of humanoid hand model is still highly developed in recent years. This paper proposes another method to achieve a proportional estimation and control of human’s hand multiple DOFs. Gestures in the form of American Sign Language (ABCDFIKLOW) were chosen as the targets, of which ten alphabetical gestures were specifically used following their clarity on its 3D model. Then the dataset of the movements gestures was simultaneously recorded using High-density electromyography (HD-EMG) and motion capture system. Sensor placements were on intrinsic - extrinsic muscles for HD-EMG and finger joints for the motion capture system. To derive the proportional control in time series between both datasets (HD-EMG and kinematics data), neural network (NN) and k-Nearest Neighbour were used. The models produced around 70-95 % (R index) accuracy for the eleven DOFs in four healthy subjects’ hand. kNN’s performance was better than NN, even if the input features were reduced either using manual selections or principal component analysis (PCA). The time series controls could also identify most sign language gestures (9 of 10), with difficulty was given on O gesture. The false interpretation was because of nearly identical muscle’s EMG and kinematics data between O and C. This paper intends to extend its conference version [1] by adding more in-depth Results and Discussion along making other sections more comprehensive

STUDY OF HAND GESTURE RECOGNITION AND CLASSIFICATION

To recognize different hand gestures and achieve efficient classification to understand static and dynamic hand movements used for communications.Static and dynamic hand movements are first captured using gesture recognition devices including Kinect device, hand movement sensors, connecting electrodes, and accelerometers. These gestures are processed using hand gesture recognition algorithms such as multivariate fuzzy decision tree, hidden Markov models (HMM), dynamic time warping framework, latent regression forest, support vector machine, and surface electromyogram. Hand movements made by both single and double hands are captured by gesture capture devices with proper illumination conditions. These captured gestures are processed for occlusions and fingers close interactions for identification of right gesture and to classify the gesture and ignore the intermittent gestures. Real-time hand gestures recognition needs robust algorithms like HMM to detect only the intended gesture. Classified gestures are then compared for the effectiveness with training and tested standard datasets like sign language alphabets and KTH datasets. Hand gesture recognition plays a very important role in some of the applications such as sign language recognition, robotics, television control, rehabilitation, and music orchestration

EMG SIGNALS FOR FINGER MOVEMENT CLASSIFICATION BASED ON SHORT-TERM FOURIER TRANSFORM AND DEEP LEARNING

An interface based on electromyographic (EMG) signals is considered one of the central fields in human-machine interface (HCI) research with broad practical use. This paper presents the recognition of 13 individual finger movements based on the time-frequency representation of EMG signals via spectrograms. A deep learning algorithm, namely a convolutional neural network (CNN), is used to extract features and classify them. Two approaches to EMG data representations are investigated: different window segmentation lengths and reduction of the measured channels. The overall highest accuracy of the classification reaches 95.5% for a segment length of 300 ms. The average accuracy attains more than 90% by reducing channels from four to three