DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation

There is an undeniable communication barrier between deaf people and people with normal hearing ability. Although innovations in sign language translation technology aim to tear down this communication barrier, the majority of existing sign language translation systems are either intrusive or constrained by resolution or ambient lighting conditions. Moreover, these existing systems can only perform single-sign ASL translation rather than sentence-level translation, making them much less useful in daily-life communication scenarios. In this work, we fill this critical gap by presenting DeepASL, a transformative deep learning-based sign language translation technology that enables ubiquitous and non-intrusive American Sign Language (ASL) translation at both word and sentence levels. DeepASL uses infrared light as its sensing mechanism to non-intrusively capture the ASL signs. It incorporates a novel hierarchical bidirectional deep recurrent neural network (HB-RNN) and a probabilistic framework based on Connectionist Temporal Classification (CTC) for word-level and sentence-level ASL translation respectively. To evaluate its performance, we have collected 7,306 samples from 11 participants, covering 56 commonly used ASL words and 100 ASL sentences. DeepASL achieves an average 94.5% word-level translation accuracy and an average 8.2% word error rate on translating unseen ASL sentences. Given its promising performance, we believe DeepASL represents a significant step towards breaking the communication barrier between deaf people and hearing majority, and thus has the significant potential to fundamentally change deaf people's lives

Rotation Correction Method Using Depth-Value Symmetry of Human Skeletal Joints for Single RGB-D Camera System

Most red-green-blue and depth (RGB-D) motion-recognition technologies employ both depth and RGB cameras to recognize a user\u27s body. However, motion-recognition solutions using a single RGB-D camera struggle with rotation recognition depending on the device-user distance and field-of-view. This paper proposes a near-real-time rotational-coordinate-correction method that rectifies a depth error unique Microsoft Kinect by using the symmetry of the depth coordinates of the human body. The proposed method is most effective within 2 m, a key range in which the unique depth error of Kinect occurs, and is anticipated to be utilized in applications requiring low cost and fast installation. It could also be useful in areas such as media art that involve unspecified users because it does not require a learning phase. Experimental results indicate that the proposed method has an accuracy of 85.38%, which is approximately 12% higher than that of the reference installation method

Two Hand Gesture Based 3D Navigation in Virtual Environments

Natural interaction is gaining popularity due to its simple, attractive, and realistic nature, which realizes direct Human Computer Interaction (HCI). In this paper, we presented a novel two hand gesture based interaction technique for 3 dimensional (3D) navigation in Virtual Environments (VEs). The system used computer vision techniques for the detection of hand gestures (colored thumbs) from real scene and performed different navigation (forward, backward, up, down, left, and right) tasks in the VE. The proposed technique also allow users to efficiently control speed during navigation. The proposed technique is implemented via a VE for experimental purposes. Forty (40) participants performed the experimental study. Experiments revealed that the proposed technique is feasible, easy to learn and use, having less cognitive load on users. Finally gesture recognition engines were used to assess the accuracy and performance of the proposed gestures. kNN achieved high accuracy rates (95.7%) as compared to SVM (95.3%). kNN also has high performance rates in terms of training time (3.16 secs) and prediction speed (6600 obs/sec) as compared to SVM with 6.40 secs and 2900 obs/sec

Building a web-based continuous sign language translation application for the deaf and hard-of-hearing (DHH)

Applied project submitted to the Department of Computer Science and Information Systems, Ashesi University, in partial fulfillment of Bachelor of Science degree in Computer Science, May 2020Advancement in technology has brought about limitless possibilities. We have seen technology been applied to every aspect of our lives, from Medicine to the food we eat. This technology can also be applied to make life more comfortable for the Deaf and Hard-of Hearing (DHH) individuals. Assistive technologies have been built for individuals with various forms of disabilities. DHH individuals have seen assistive technologies in the form of mechanical hardware, and cybernetics that enables translation of sign language to verbal communication in various languages. However, these technologies are often expensive and difficult to afford. In this project, an automatic sign language translation model is implemented to help people without hearing impairment to communicate better with DHH individuals in order to better provide services for them.Ashesi Universit