Detection of major ASL sign types in continuous signing for ASL recognition

In American Sign Language (ASL) as well as other signed languages, different classes of signs (e.g., lexical signs, fingerspelled signs, and classifier constructions) have different internal structural properties. Continuous sign recognition accuracy can be improved through use of distinct recognition strategies, as well as different training datasets, for each class of signs. For these strategies to be applied, continuous signing video needs to be segmented into parts corresponding to particular classes of signs. In this paper we present a multiple instance learning-based segmentation system that accurately labels 91.27% of the video frames of 500 continuous utterances (including 7 different subjects) from the publicly accessible NCSLGR corpus (Neidle and Vogler, 2012). The system uses novel feature descriptors derived from both motion and shape statistics of the regions of high local motion. The system does not require a hand tracker

Computational Models for the Automatic Learning and Recognition of Irish Sign Language

This thesis presents a framework for the automatic recognition of Sign Language sentences. In previous sign language recognition works, the issues of; user independent recognition, movement epenthesis modeling and automatic or weakly supervised training have not been fully addressed in a single recognition framework. This work presents three main contributions in order to address these issues. The first contribution is a technique for user independent hand posture recognition. We present a novel eigenspace Size Function feature which is implemented to perform user independent recognition of sign language hand postures. The second contribution is a framework for the classification and spotting of spatiotemporal gestures which appear in sign language. We propose a Gesture Threshold Hidden Markov Model (GT-HMM) to classify gestures and to identify movement epenthesis without the need for explicit epenthesis training. The third contribution is a framework to train the hand posture and spatiotemporal models using only the weak supervision of sign language videos and their corresponding text translations. This is achieved through our proposed Multiple Instance Learning Density Matrix algorithm which automatically extracts isolated signs from full sentences using the weak and noisy supervision of text translations. The automatically extracted isolated samples are then utilised to train our spatiotemporal gesture and hand posture classifiers. The work we present in this thesis is an important and significant contribution to the area of natural sign language recognition as we propose a robust framework for training a recognition system without the need for manual labeling

A novel set of features for continuous hand gesture recognition

Applications requiring the natural use of the human hand as a human–computer interface motivate research on continuous hand gesture recognition. Gesture recognition depends on gesture segmentation to locate the starting and end points of meaningful gestures while ignoring unintentional movements. Unfortunately, gesture segmentation remains a formidable challenge because of unconstrained spatiotemporal variations in gestures and the coarticulation and movement epenthesis of successive gestures. Furthermore, errors in hand image segmentation cause the estimated hand motion trajectory to deviate from the actual one. This research moves toward addressing these problems. Our approach entails using gesture spotting to distinguish meaningful gestures from unintentional movements. To avoid the effects of variations in a gesture’s motion chain code (MCC), we propose instead to use a novel set of features: the (a) orientation and (b) length of an ellipse least-squares fitted to motion-trajectory points and (c) the position of the hand. The features are designed to support classification using conditional random fields. To evaluate the performance of the system, 10 participants signed 10 gestures several times each, providing a total of 75 instances per gesture. To train the system, 50 instances of each gesture served as training data and 25 as testing data. For isolated gestures, the recognition rate using the MCC as a feature vector was only 69.6 % but rose to 96.0 % using the proposed features, a 26.1 % improvement. For continuous gestures, the recognition rate for the proposed features was 88.9 %. These results show the efficacy of the proposed method

Evaluation of RSL history as a tool for assistance in the development and evaluation of computer vision algorithms

A revision of Recognition Strategy Language (RSL), a domain-specific language for pattern recognition algorithm development, is in development. This language provides several tools for pattern recognition algorithm implementation and analysis, including composition of operations and a detailed history of those operations and their results. This research focuses on that history and shows that for some problems it provides an improvement over traditional methods of gathering information. When designing a pattern recognition algorithm, bookkeeping code in the form of copious logging and tracing code must be written and analyzed in order to test the effectiveness of procedures and parameters. The amount of data grows when dealing with video streams; new organization and searching tools need to be designed in order to manage the large volume of data. General purpose languages have techniques like Aspect Oriented Programming intended to address this problem, but a general approach is limited because it does not provide tools that are useful to only one problem domain. By incorporating support for this bookkeeping work directly into the language, RSL provides an improvement over the general approach in both development time and ability to evaluate the algorithm being designed for some problems. The utility of RSL is tested by evaluating the implementation process of a computer vision algorithm for recognizing American Sign Language (ASL). RSL history is examined in terms of its use in the development and evaluation stages of the algorithm, and the usefulness of the history is stated based on the benefit seen at each stage. RSL is found to be valuable for a portion of the algorithm involving distinct steps that provide opportunity for comparison. RSL was less beneficial for the dynamic programming portion of the algorithm. Compromises were made for performance reasons while implementing the dynamic programming solution and the inspection at every step of what amounts to a brute-force search was less informative. We suggest that this investigation could be continued by testing with a larger data set and by comparing this ASL recognition algorithm with another

Towards Subject Independent Sign Language Recognition : A Segment-Based Probabilistic Approach

Ph.DDOCTOR OF PHILOSOPH

A new framework for sign language recognition based on 3D handshape identification and linguistic modeling

Current approaches to sign recognition by computer generally have at least some of the following limitations: they rely on laboratory conditions for sign production, are limited to a small vocabulary, rely on 2D modeling (and therefore cannot deal with occlusions and off-plane rotations), and/or achieve limited success. Here we propose a new framework that (1) provides a new tracking method less dependent than others on laboratory conditions and able to deal with variations in background and skin regions (such as the face, forearms, or other hands); (2) allows for identification of 3D hand configurations that are linguistically important in American Sign Language (ASL); and (3) incorporates statistical information reflecting linguistic constraints in sign production. For purposes of large-scale computer-based sign language recognition from video, the ability to distinguish hand configurations accurately is critical. Our current method estimates the 3D hand configuration to distinguish among 77 hand configurations linguistically relevant for ASL. Constraining the problem in this way makes recognition of 3D hand configuration more tractable and provides the information specifically needed for sign recognition. Further improvements are obtained by incorporation of statistical information about linguistic dependencies among handshapes within a sign derived from an annotated corpus of almost 10,000 sign tokens

A Real-time Sign Language Recognition System for Hearing and Speaking Challengers

[[abstract]]Sign language is the primary means of communication between deaf people and hearing/speaking challengers. There are many varieties of sign language in different challenger community, just like an ethnic community within society. Unfortunately, few people have knowledge of sign language in our daily life. In general, interpreters can help us to communicate with these challengers, but they only can be found in Government Agencies, Hospital, and etc. Moreover, it is expensive to employ interpreter on personal behalf and inconvenient when privacy is required. It is very important to develop a robust Human Machine Interface (HMI) system that can support challengers to enter our society. A novel sign language recognition system is proposed. This system is composed of three parts. First, initial coordinate locations of hands are obtained by using joint skeleton information of Kinect. Next, we extract features from joints of hands that have depth information and translate handshapes. Then we train Hidden Markov Model-based Threshold Model by three feature sets. Finally, we use Hidden Markov Model-based Threshold Model to segment and recognize sign language. Experimental results show, average recognition rate for signer-dependent and signer-independent are 95% and 92%, respectively. We also find that feature sets including handshape can achieve better recognition result.[[sponsorship]]Asia-Pacific Education & Research Association[[conferencetype]]國際[[conferencedate]]20140711~20140713[[booktype]]紙本[[iscallforpapers]]Y[[conferencelocation]]普吉島, 泰

Data mining and modelling for sign language

Sign languages have received significantly less attention than spoken languages in the research areas of corpus analysis, machine translation, recognition, synthesis and social signal processing, amongst others. This is mainly due to signers being in a clear minority and there being a strong prior belief that sign languages are simply arbitrary gestures. To date, this manifests in the insufficiency of sign language resources available for computational modelling and analysis, with no agreed standards and relatively stagnated advancements compared to spoken language interaction research. Fortunately, the machine learning community has developed methods, such as transfer learning, for dealing with sparse resources, while data mining techniques, such as clustering can provide insights into the data. The work described here utilises such transfer learning techniques to apply neural language model to signed utterances and to compare sign language phonemes, which allows for clustering of similar signs, leading to automated annotation of sign language resources. This thesis promotes the idea that sign language research in computing should rely less on hand-annotated data thus opening up the prospect of using readily available online data (e.g. signed song videos) through the computational modelling and automated annotation techniques presented in this thesis