Real Time Static and Dynamic Sign Language Recognition using Deep Learning

Sign language recognition systems are used for enabling communication between deaf-mute people and normal user. Spatial localization of the hands could be a challenging task when hands-only occupies 10% of the entire image. This is overcome by designing a real-time efficient system that is capable of performing the task of extraction, recognition, and classification within a single network with the use of a deep convolution network. The recognition is performed for static image dataset with a simple and complex background, dynamic video dataset. Static image dataset is trained and tested using a 2D deep-convolution neural network whereas dynamic video dataset is trained and tested using a 3D deep-convolution neural network. Spatial augmentation is done to increase the number of images of static dataset and key-frame extraction to extract the key-frames from the videos for dynamic dataset. To improve the system performance and accuracy Batch-Normalization layer is added to the convolution network. The accuracy is nearly 99% for dataset with a simple background, 92% for dataset with complex background, and 84% for the video dataset. By obtaining a good accuracy, the system is proved to be real-time efficient in recognizing and interpreting the sign language gestures

Computational Models for the Automatic Learning and Recognition of Irish Sign Language

This thesis presents a framework for the automatic recognition of Sign Language sentences. In previous sign language recognition works, the issues of; user independent recognition, movement epenthesis modeling and automatic or weakly supervised training have not been fully addressed in a single recognition framework. This work presents three main contributions in order to address these issues. The first contribution is a technique for user independent hand posture recognition. We present a novel eigenspace Size Function feature which is implemented to perform user independent recognition of sign language hand postures. The second contribution is a framework for the classification and spotting of spatiotemporal gestures which appear in sign language. We propose a Gesture Threshold Hidden Markov Model (GT-HMM) to classify gestures and to identify movement epenthesis without the need for explicit epenthesis training. The third contribution is a framework to train the hand posture and spatiotemporal models using only the weak supervision of sign language videos and their corresponding text translations. This is achieved through our proposed Multiple Instance Learning Density Matrix algorithm which automatically extracts isolated signs from full sentences using the weak and noisy supervision of text translations. The automatically extracted isolated samples are then utilised to train our spatiotemporal gesture and hand posture classifiers. The work we present in this thesis is an important and significant contribution to the area of natural sign language recognition as we propose a robust framework for training a recognition system without the need for manual labeling

Towards Subject Independent Sign Language Recognition : A Segment-Based Probabilistic Approach

Ph.DDOCTOR OF PHILOSOPH

Analysis and extension of hierarchical temporal memory for multivariable time series

Tesis doctoral inédita. Universidad Autónoma de Madrid, Escuela Politécnica Superior, junio de 201

Proceedings of the 7th Sound and Music Computing Conference

Proceedings of the SMC2010 - 7th Sound and Music Computing Conference, July 21st - July 24th 2010

Approaching real time dynamic signature verification from a systems and control perspective.

Student Number : 9901877H MSc Dissertation School of Electrical and Information Engineering Faculty of Engineering and the Built Environmentalgorithm. The origins of handwriting idiosyncrasies and habituation are explained using systems theory, and it is shown that the 2/3 power law governing biomechanics motion also applies to handwriting. This leads to the conclusion that it is possible to derive handwriting velocity profiles from a static image, and that a successful forgery of a signature is only possible in the event of the forger being able to generate a signature using natural ballistic motion. It is also shown that significant portion of the underlying dynamic system governing the generation of handwritten signatures can be inferred by deriving time segmented transfer function models of the x and y co-ordinate velocity profiles of a signature. The prototype algorithm consequently developed uses x and y components of pen-tip velocity profiles (vx[n] and vy[n]) to create signature representations based on autoregression-with-exogenous-input (ARX) models. Verification is accomplished using a similarity measure based on the results of a k-step ahead predictor and 5 complementary metrics. Using 350 signatures collected from 21 signers, the system’s false acceptance (FAR) and false rejection (FRR) rates were 2.19% and 27.05% respectively. This high FRR is a result of measurement inadequacies, and it is believed that the algorithm’s FRR is approximately 18%

Proceedings of the 2nd European conference on disability, virtual reality and associated technologies (ECDVRAT 1998)

The proceedings of the conferenc

Fast upper body pose estimation for human-robot interaction

This work describes an upper body pose tracker that finds a 3D pose estimate using video sequences obtained from a monocular camera, with applications in human-robot interaction in mind. A novel mixture of Ornstein-Uhlenbeck processes model, trained in a reduced dimensional subspace and designed for analytical tractability, is introduced. This model acts as a collection of mean-reverting random walks that pull towards more commonly observed poses. Pose tracking using this model can be Rao-Blackwellised, allowing for computational efficiency while still incorporating bio-mechanical properties of the upper body. The model is used within a recursive Bayesian framework to provide reliable estimates of upper body pose when only a subset of body joints can be detected. Model training data can be extended through a retargeting process, and better pose coverage obtained through the use of Poisson disk sampling in the model training stage. Results on a number of test datasets show that the proposed approach provides pose estimation accuracy comparable with the state of the art in real time (30 fps) and can be extended to the multiple user case. As a motivating example, this work also introduces a pantomimic gesture recognition interface. Traditional approaches to gesture recognition for robot control make use of predefined codebooks of gestures, which are mapped directly to the robot behaviours they are intended to elicit. These gesture codewords are typically recognised using algorithms trained on multiple recordings of people performing the predefined gestures. Obtaining these recordings can be expensive and time consuming, and the codebook of gestures may not be particularly intuitive. This thesis presents arguments that pantomimic gestures, which mimic the intended robot behaviours directly, are potentially more intuitive, and proposes a transfer learning approach to recognition, where human hand gestures are mapped to recordings of robot behaviour by extracting temporal and spatial features that are inherently present in both pantomimed actions and robot behaviours. A Bayesian bias compensation scheme is introduced to compensate for potential classification bias in features. Results from a quadrotor behaviour selection problem show that good classification accuracy can be obtained when human hand gestures are recognised using behaviour recordings, and that classification using these behaviour recordings is more robust than using human hand recordings when users are allowed complete freedom over their choice of input gestures