4 research outputs found
3D face tracking and multi-scale, spatio-temporal analysis of linguistically significant facial expressions and head positions in ASL
Essential grammatical information is conveyed in signed languages by clusters of events involving facial expressions and movements of the head and upper body. This poses a significant challenge for computer-based sign language recognition. Here, we present new methods for the recognition of nonmanual grammatical markers in American Sign Language (ASL) based on: (1) new 3D tracking methods for the estimation of 3D head pose and facial expressions to determine the relevant low-level features; (2) methods for higher-level analysis of component events (raised/lowered eyebrows, periodic head nods and head shakes) used in grammatical markings—with differentiation of temporal phases (onset, core, offset, where appropriate), analysis of their characteristic properties, and extraction of corresponding features; (3) a 2-level learning framework to combine lowand high-level features of differing spatio-temporal scales. This new approach achieves significantly better tracking and recognition results than our previous methods
ASL video Corpora & Sign Bank: resources available through the American Sign Language Linguistic Research Project (ASLLRP)
The American Sign Language Linguistic Research Project (ASLLRP) provides Internet access to
high-quality ASL video data, generally including front and side views and a close-up of the face.
The manual and non-manual components of the signing have been linguistically annotated using
SignStream®. The recently expanded video corpora can be browsed and searched through the Data
Access Interface (DAI 2) we have designed; it is possible to carry out complex searches. The data
from our corpora can also be downloaded; annotations are available in an XML export format. We
have also developed the ASLLRP Sign Bank, which contains almost 6,000 sign entries for lexical
signs, with distinct English-based glosses, with a total of 41,830 examples of lexical signs (in addition
to about 300 gestures, over 1,000 fingerspelled signs, and 475 classifier examples). The Sign Bank is
likewise accessible and searchable on the Internet; it can also be accessed from within SignStream®
(software to facilitate linguistic annotation and analysis of visual language data) to make
annotations more accurate and efficient. Here we describe the available resources. These data have
been used for many types of research in linguistics and in computer-based sign language recognition
from video; examples of such research are provided in the latter part of this article.Published versio
Computer-based tracking, analysis, and visualization of linguistically significant nonmanual events in American Sign Language (ASL)
Our linguistically annotated American Sign Language (ASL) corpora have formed a basis for research to automate detection by
computer of essential linguistic information conveyed through facial expressions and head movements. We have tracked head position
and facial deformations, and used computational learning to discern specific grammatical markings. Our ability to detect, identify, and
temporally localize the occurrence of such markings in ASL videos has recently been improved by incorporation of (1) new techniques
for deformable model-based 3D tracking of head position and facial expressions, which provide significantly better tracking accuracy
and recover quickly from temporary loss of track due to occlusion; and (2) a computational learning approach incorporating 2-level
Conditional Random Fields (CRFs), suited to the multi-scale spatio-temporal characteristics of the data, which analyses not only
low-level appearance characteristics, but also the patterns that enable identification of significant gestural components, such as
periodic head movements and raised or lowered eyebrows. Here we summarize our linguistically motivated computational approach
and the results for detection and recognition of nonmanual grammatical markings; demonstrate our data visualizations, and discuss the
relevance for linguistic research; and describe work underway to enable such visualizations to be produced over large corpora and
shared publicly on the Web
NEW shared & interconnected ASL resources: SignStream® 3 Software; DAI 2 for web access to linguistically annotated video corpora; and a sign bank
2017 marked the release of a new version of SignStream® software, designed to facilitate linguistic analysis of ASL video. SignStream® provides an intuitive interface for labeling and time-aligning manual and non-manual components of the signing. Version 3 has many new features. For example, it enables representation of morpho-phonological information, including display of handshapes. An expanding ASL video corpus, annotated through use of SignStream®, is shared publicly on the Web. This corpus (video plus annotations) is Web-accessible—browsable, searchable, and downloadable—thanks to a new, improved version of our Data Access Interface: DAI 2. DAI 2 also offers Web access to a brand new Sign Bank, containing about 10,000 examples of about 3,000 distinct signs, as produced by up to 9 different ASL signers. This Sign Bank is also directly accessible from within SignStream®, thereby boosting the efficiency and consistency of annotation; new items can also be added to the Sign Bank. Soon to be integrated into SignStream® 3 and DAI 2 are visualizations of computer-generated analyses of the video: graphical display of eyebrow height, eye aperture, an