Indian Sign Language Recognition through Hybrid ConvNet-LSTM Networks

Dynamic hand gesture recognition is a challenging task of Human-Computer Interaction (HCI) and Computer Vision. The potential application areas of gesture recognition include sign language translation, video gaming, video surveillance, robotics, and gesture-controlled home appliances. In the proposed research, gesture recognition is applied to recognize sign language words from real-time videos. Classifying the actions from video sequences requires both spatial and temporal features. The proposed system handles the former by the Convolutional Neural Network (CNN), which is the core of several computer vision solutions and the latter by the Recurrent Neural Network (RNN), which is more efficient in handling the sequences of movements. Thus, the real-time Indian sign language (ISL) recognition system is developed using the hybrid CNN-RNN architecture. The system is trained with the proposed CasTalk-ISL dataset. The ultimate purpose of the presented research is to deploy a real-time sign language translator to break the hurdles present in the communication between hearing-impaired people and normal people. The developed system achieves 95.99% top-1 accuracy and 99.46% top-3 accuracy on the test dataset. The obtained results outperform the existing approaches using various deep models on different datasets

Use of Key Points and Transfer Learning Techniques in Recognition of Handedness Indian Sign Language

The most expressive way of communication for individuals who have trouble speaking or hearing is sign language. Normal people are unable to comprehend sign language. As a result, communication barriers are put up. Majority of people are right-handed. Statistics say that, an average population of left-handed person in the world is about 10%, where they use left hand as their dominating hand. In case of hand written text recognition, if the text is written by left-handed or right-handed person, then there would not be any problem in recognition neither for human and nor for computer. But same thing is not true for sign language and its detection using computer. When the detection is performed using computer vision and if it falls into the category of detection by appearance, then it might not detect correctly. In machine and deep learning, if the model is trained using just one dominating hand, let’s say right hand, then the predictions can go wrong if same sign is performed by left-handed person. This paper addresses this issue. It takes into account the signs performed by any type of signer: left-handed, right-handed or ambidexter. In proposed work is on Indian Sign Language (ISL). Two models are trained: Model I, is trained on one dominating hand and Model II, is trained on both the hands. Model II gives correct predictions regardless of any type of signer. It recognizes alphabets and numbers in ISL. We used the concept of Key points and Transfer Learning techniques for implementation. Using this approach, models get trained quickly and we could achieve validation accuracy of 99%

Indian Sign Language Numbers Recognition using Intel RealSense Camera

The use of gesture based interaction with devices has been a significant area of research in the field of computer science since many years. The main idea of these kind of interactions is to ease the user experience by providing high degree of freedom and provide more interactive way of communication with the technology in a natural way. The significant areas of applications of gesture recognition are in video gaming, human computer interaction, virtual reality, smart home appliances, medical systems, robotics and several others. With the availability of the devices such as Kinect, Leap Motion and Intel RealSense cameras accessing the depth as well as color information has become available to the public with affordable costs. The Intel RealSense camera is a USB powered controller that can be supported with few hardware requirements such as Windows 8 and above. This is one such camera that can be used to track the human body information similar to the Kinect and Leap Motion. It was designed specifically to provide more minute information about the different parts of the human body such as face, hand etc. This camera was designed to give users more natural and intuitive interactions with the smart devices by providing some features such as creating 3D avatars, high quality 3D prints, high-quality graphic gaming visuals, virtual reality and others. The main aim of this study is to try to analyze hand tracking information and build a training model in order to decide if this camera is suitable for sign language. In this study, we have extracted the joint information of 22 joint labels per single hand .We trained the model to identify the Indian Sign Language(ISL) numbers from 0-9. Through this study we analyzed that multi-class SVM model showed higher accuracy of 93.5% when compared to the decision tree and KNN models

ISLTranslate: Dataset for Translating Indian Sign Language

Sign languages are the primary means of communication for many hard-of-hearing people worldwide. Recently, to bridge the communication gap between the hard-of-hearing community and the rest of the population, several sign language translation datasets have been proposed to enable the development of statistical sign language translation systems. However, there is a dearth of sign language resources for the Indian sign language. This resource paper introduces ISLTranslate, a translation dataset for continuous Indian Sign Language (ISL) consisting of 31k ISL-English sentence/phrase pairs. To the best of our knowledge, it is the largest translation dataset for continuous Indian Sign Language. We provide a detailed analysis of the dataset. To validate the performance of existing end-to-end Sign language to spoken language translation systems, we benchmark the created dataset with a transformer-based model for ISL translation.Comment: Accepted at ACL 2023 Findings, 8 Page

Hand Gesture Recognition using Depth Data for Indian Sign Language

It is hard for most people who are not familiar with a sign language to communicate without an interpreter. Thus, a system that transcribes symbols in sign languages into plain text can help with real-time communication, and it may also provide interactive training for people to learn a sign language. A sign language uses manual communication and body language to convey meaning. The depth data for five different gestures corresponding to alphabets Y, V, L, S, I was obtained from online database. Each segmented gesture is represented by its timeseries curve and feature vector is extracted from it. To recognise the class of input noisy hand shape, distance metric for hand dissimilarity measure, called Finger-Earth Mover’s Distance (FEMD) is used. As it only matches fingers while not the complete hand shape, it can distinguish hand gestures of slight differences better