Toward Computational Understanding Of Sign Language

Abstract

In this paper, we describe some of the current issues in computational sign language processing. Despite the seeming similarities between computational spoken language and sign language processing, signed languages have intrinsic properties that pose some very difficult problems. These include a high level of simultaneous actions, the intersection between signs and gestures, and the complexity of modeling grammatical processes. Additional problems are posed by the difficulties that computers face in extracting reliable information on the hands and the face from video images. So far, no single research group or company has managed to tackle all the hard problems and produced a real working system for analysis and recognition. We present a summary of our research into sign language recognition and how it interacts with sign language linguistics. We propose solutions to some of the aforementioned problems, and also discuss what problems are still unsolved. In addition, we summarize the current state of the art in our sign language recognition and facial expression analysis frameworks. © 2008 IOS Press. All rights reserved.202109119Athitsos, V., Sclaroff, S., Estimating 3d hand pose ftorn a cluttered image (2003) IEEE International Conference on Computer Vision, pp. 432-439. , ICCVBattison, R., (1978) Lexical borrowing in American Sign Language, , Linstok Press, Silver Spring, NM, Reprinted as Analyzing Signs, in: C. Lucas and C. Valli, Linguistics of American Sign Language, 1995, 19-58B. Bauer and K.-F. Kraiss, Towards an automatic sign language recognition system using subunits, in: Gesture and Sign Language in Human-Computer Interaction, 1. Wachsmuth and T. Sowa, eds, International Gesture Workshop, 2298 of Lecture Notes in Artificial Intelligence, Springer, 2001, pp. 64-75Bauer, B., Kraiss, K.-F., Video-based sign recognition using self-organizing subunits (2002) International Conference on Pattern RecognitionBrentari, D., (1998) A Prosodic Model of Sign Language Phonology, Language, Speech, and Communication, , MIT Press, Cambridge, MAU. Canzler and K.-F. Kraiss, Person-adaptive facial feature analysis for an advanced wheelchair user-interface, in: Conference on Mechatronics & Robotics, (3), P. Drews, ed., Sascha Eysoldt Verlag, 2004, pp. 871-876Cootes, T.F., Edwards, G.J., Taylor, C.J., Active appearance models (2001) IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI, 23 (6), pp. 681-685Cootes, T.F., Taylor, C.J., Active shape models - their training and application (1995) Computer Wsion and Image Understanding (CVIU), 61 (1), pp. 38-59DeCarlo, D., Metaxas, D., Adjusting shape parameters using model-based optical flow residuals (2002) IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 24 (6), pp. 814-823. , JuneGoldenstein, S., Vogler, C., Metaxas, D., Statistical Cue Integration in DAG Deformable Models (2003) IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 25 (7), pp. 801-813Goldenstein, S., Vogler, C., Metaxas, D., 3D facial tracking from corrupted movie sequences (2004) IEEE Conference in Computer Vision and Pattern Recognition, , CVPRGoldenstein, S., Vogler, C., Velho, L., Adaptive deformable models for graphics and vision (2005) Computer Graphics Forum (CGF), 2, pp. 729-741. , December(1996) Automatic Speech and Speaker Recognition, Advanced Topics, , C.-H. Lee, F.K. Soong and K.K. Paliwal, eds, Kluwer Academic Publishers, Boston, MALiang, R.-H., Ouhyoung, M., A real-time continuous gesture recognition system for sign language (1998) Proceedings of the Third International Conference on Automatic Face and Gesture Recognition, pp. 558-565. , Nara, JapanLiddell, S., Indicating verbs and pronouns: Pointing away from agreement (2000) The Signs of Language Revisited, pp. 303-320. , K. Emmorey and H. Lane, eds, Lawrence ErlbaumLiddell, S.K., Johnson, R.E., The phonological base (1989) Sign Language Studies, 64, pp. 195-277. , American Sign LanguageNeidle, C., Kegl, J., MacLaughlin, D., Bahan, B., Lee, R.G., (2000) The Sytax of American Sign Language, Language, Speech, and Communication, , MIT Press, Cambridge, MassachusettsNeidle, C., Sclaroff, S., Data collected, , http://www.bu.edu/asllrp/ncslgr.html, at the National Center for Sign Language and Gesture Resources, Boston University, under the supervision of C. Neidle and S. Sclaroff. Available online at, 2002Ong, A., Ranganath, S., Automatic sign language analysis: A survey and the future beyond lexical meaning (2005) IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 27 (6), pp. 873-891Rabiner, L.R., A tutorial on Hidden Markov Models and selected applications inspeech recognition (1989) Proceedings of the IEEE, 77 (2), pp. 257-286Rabiner, L.R., Juang, B.-H., (1993) Fundamentals of Speech Recognition, , Prentice-Hall, Inc, Englewoods Cliffs, NJSandler, W., Phonological Representation of the Sign: Linearity and Nonlinearity in American Sign Language (1989) Publications in Language Sciences, (32). , Foris Publications, DordrechtW. Sandler, Representing Handshapes, in: International Review of Sign Linguistics, W.H. Edmondson and R. Wilbur, eds, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1996, (1)(5), pp. 115-158Stokoe, W.C., Sign Language Structure: An Outline of the Visual Communication System of the American Deaf (1960) Studies in Linguistics, , Occasional Papers 8. Linstok Press, Silver Spring, MD, RevisedTaub, S., Galvan, D., Patterns of conceptual encoding in ASL motion descriptions (2001) Sign Language Studies, 2 (1), pp. 175-200Tsechpenakis, G., Metaxas, D., Neidle, C., Learning-based coupling of discreteand continuous trackers (2006) Computer Vision and Image Understanding (CVIU), 104 (2-3), pp. 140-156Valli, C., Lucas, C., Mulrooney, K., (2007) Linguistics of American Sign Language: An Introduction, , Gallaudet University Press, Washington DCVogler, C., Goldenstein, S., Stolfi, J., Pavlovic, V., Metaxas, D., Outlier rejection in high-dimensional deformable models (2007) Image and Vision Computing, 25 (3), pp. 274-284Vogler, C., Li, Z., Kanaujia, A., Goldenstein, S., Metaxas, D., The best of both worlds: Combining 3D deformable models with Active Shape Models (2007) IEEE International Conference on Computer Wsion, , ICCVVogler, C., Metaxas, D., Parallel hidden Markov models for American Sign Language recognition (1999) IEEE International Conference on Computer Vision (ICCV), pp. 116-122. , Kerkyra, GreeceVogler, C., Metaxas, D., Breaking down signs into phonemes (1999) Lecture Notes in Artificial Intelligence, 1739, pp. 211-224. , Toward scalability in ASL recognition:, Gesture-Based Communication in Human-Computer Interaction, A. Braffort, R. Gherbi, S. Gibet, J. Richardson and D. Teil, eds, of, SpringerVogler, C., Metaxas, D., A framework for recognizing the simultaneous aspects of American Sign Language (2001) Computer Vision and Image Understanding (CVIU), 81 (81), pp. 358-384Vogler, C., Metaxas, D., Handshapes and movements: Multiple-channel ASL recognition (2004) Lecture Notes in Artificial Intelligence, 2915, pp. 247-258. , Proceedings of the Gesture Workshop, G. Volpe et al, ed, of, SpringerWang, C., Gao, W., Ma, J., A real-time large vocabulary recognition system for Chinese Sign Language (2002) Lecture Notes in Artificial Intelligence, 2298, pp. 86-95. , I. Wachsmuth and T. Sowa, eds, Springe