Detection of major ASL sign types in continuous signing for ASL recognition

In American Sign Language (ASL) as well as other signed languages, different classes of signs (e.g., lexical signs, fingerspelled signs, and classifier constructions) have different internal structural properties. Continuous sign recognition accuracy can be improved through use of distinct recognition strategies, as well as different training datasets, for each class of signs. For these strategies to be applied, continuous signing video needs to be segmented into parts corresponding to particular classes of signs. In this paper we present a multiple instance learning-based segmentation system that accurately labels 91.27% of the video frames of 500 continuous utterances (including 7 different subjects) from the publicly accessible NCSLGR corpus (Neidle and Vogler, 2012). The system uses novel feature descriptors derived from both motion and shape statistics of the regions of high local motion. The system does not require a hand tracker

A new framework for sign language recognition based on 3D handshape identification and linguistic modeling

Current approaches to sign recognition by computer generally have at least some of the following limitations: they rely on laboratory conditions for sign production, are limited to a small vocabulary, rely on 2D modeling (and therefore cannot deal with occlusions and off-plane rotations), and/or achieve limited success. Here we propose a new framework that (1) provides a new tracking method less dependent than others on laboratory conditions and able to deal with variations in background and skin regions (such as the face, forearms, or other hands); (2) allows for identification of 3D hand configurations that are linguistically important in American Sign Language (ASL); and (3) incorporates statistical information reflecting linguistic constraints in sign production. For purposes of large-scale computer-based sign language recognition from video, the ability to distinguish hand configurations accurately is critical. Our current method estimates the 3D hand configuration to distinguish among 77 hand configurations linguistically relevant for ASL. Constraining the problem in this way makes recognition of 3D hand configuration more tractable and provides the information specifically needed for sign recognition. Further improvements are obtained by incorporation of statistical information about linguistic dependencies among handshapes within a sign derived from an annotated corpus of almost 10,000 sign tokens

Spatial-orientation priming impedes rather than facilitates the spontaneous control of hand-retraction speeds in patients with Parkinson's disease.

BACKGROUND: Often in Parkinson's disease (PD) motor-related problems overshadow latent non-motor deficits as it is difficult to dissociate one from the other with commonly used observational inventories. Here we ask if the variability patterns of hand speed and acceleration would be revealing of deficits in spatial-orientation related decisions as patients performed a familiar reach-to-grasp task. To this end we use spatial-orientation priming which normally facilitates motor-program selection and asked whether in PD spatial-orientation priming helps or hinders performance. METHODS: To dissociate spatial-orientation- and motor-related deficits participants performed two versions of the task. The biomechanical version (DEFAULT) required the same postural- and hand-paths as the orientation-priming version (primed-UP). Any differences in the patients here could not be due to motor issues as the tasks were biomechanically identical. The other priming version (primed-DOWN) however required additional spatial and postural processing. We assessed in all three cases both the forward segment deliberately aimed towards the spatial-target and the retracting segment, spontaneously bringing the hand to rest without an instructed goal. RESULTS AND CONCLUSIONS: We found that forward and retracting segments belonged in two different statistical classes according to the fluctuations of speed and acceleration maxima. Further inspection revealed conservation of the forward (voluntary) control of speed but in PD a discontinuity of this control emerged during the uninstructed retractions which was absent in NC. Two PD groups self-emerged: one group in which priming always affected the retractions and the other in which only the more challenging primed-DOWN condition was affected. These PD-groups self-formed according to the speed variability patterns, which systematically changed along a gradient that depended on the priming, thus dissociating motor from spatial-orientation issues. Priming did not facilitate the motor task in PD but it did reveal a breakdown in the spatial-orientation decision that was independent of the motor-postural path

Priming Experiment to increase the cognitive load of a simple reach-to-grasp task.

<p>(A) Rods rendered in three dimensions were presented on the computer screen at 1 of 5 possible locations (shown here at the center location) at the four corners and at the center of the monitor. Color indicated the target speed (red-slow and green-fast). Speed was also labeled at the center of the cylinder. The target orientation could be horizontal or vertical. Because of redundancy in the degrees of freedom at multiple joints of the arm, each one of these oriented cylinders affords more than one arm-hand orientation. Subjects were free to choose the final orientation in the DEFAULT condition. (B) The primed condition instructed the subjects to use a particular target orientation while matching the hand-held cylinder to the simulated cylinder on the screen. The subjects were instructed to pick the orientation as though they were going grab the cup and drink from it. This instruction evoked a precise arm-hand orientation that was generally different from the DEFAULT one chosen by the subject in the first block. The primed-UP case required the same orientation as the DEFAULT but the primed-DOWN case required mental rotation to align the hand to the cup as if “picking it up to drink from it”. This orientation cue evoked rotations at the arm joints and at the hand that were unambiguously different from the DEFAULT and primed-UP cases. (C) An example of a DEFAULT arm-hand orientation evoked by the cylinder oriented vertically and positioned at the center of the screen. (D) The same canonical orientation of the cylinder evokes a very different arm posture and a different hand orientation during primed-DOWN.</p

Similar biomechanical constraints for DEFAULT and primed-UP in vertical and horizontal cases contrast with different biomechanical demands between primed-UP and primed-DOWN.

<p>Traces are the averaged projection of the degrees of freedom of the arm along the dimensions relevant to the task goals and the dimensions incidental to the goals. These averages are taken across 100 trials for a typical representative and for 100 frames. Red continuous lines are task-relevant DoF forward. Blue are task-relevant DoF retractions and black are task-incidental DoF traces. Dashed lines are the standard deviation from the mean traces. During forward segments and retractions in the same loop the recruitment, release and balance of the degrees of freedom of the arm tend to markedly change as a function of task complexity.</p

Self-emerging clusters and patient subtypes based on speed maxima (m/s).

<p>(A) Representative NC and patients from PD1 and PD2 subgroups grouped the speed maxima differently in the forward and retracting motions as a function of cognitive load condition. The NC maintained consistent separation across conditions in both portions of the pointing gesture yet patients consistently performed worse in the primed cases even though primed-UP was biomechanically equivalent to DEFAULT. PD1 performed the worst with no distinction in the primed cases. Slopes changed systematically with cognitive loads even for biomechanically similar DEFAULT and primed-UP motions. (B) Self emerging subtypes of misclassified trials from the blind clustering <i>k-means</i> separated exactly as the <i>p-value</i> statistics had predicted (see Methods and Results for details).</p

Normalized frequency distribution of the maximum speed and maximum acceleration values from the retracting primed motions in two groups of patients with PD.

<p>(A) Arm trajectories (shoulder, elbow, wrist, and hand) during the primed-UP condition similar to the DEFAULT case. The initial and final arm postures corresponding to the trajectories towards two randomly selected positions are superimposed. Patients in the PD1 group had a nearly symmetric distribution of speed maxima in the primed-UP condition where their retracting motions could no longer differentiate between the randomly instructed speeds. The group of PD2 was comprised of patients whose retracting motions could, on average, differentiate between instructed fast or slow speeds during the easier primed-UP cases. Their distribution of maximum speed values was skewed.</p

Effects of priming on the movement trajectories at the wrist in typical patient within PD1 group using two different levels of speeds randomly cued.

<p>(A) DEFAULT forward motion trajectories with corresponding speed profiles for slow (red) and fast (green) cases. Inset shows the actual stimuli on the screen priming the subject to match the orientation of the rod on the screen (vertical in this case) with the hand-held rod. Top are the forward paths and bottom are the retracting motions. (B) Primed-UP cases evoked similar final orientations as the DEFAULT condition. The inset shows the priming cup with handle next to the original rod. (C) Primed-DOWN condition changed the trajectories in both the forward and retracting cases. The inset shows the priming condition where the arm and hand underwent complex rotations. The instruction was to match the orientation of the rod on the screen as if the hand were to gasp the handle of the cup to drink from it. Notice the dramatic differences in trajectories for all target positions. Retracting speed profiles in both primed-DOWN and primed-UP condition in PD1 group do not have statistically significant differences for instructed fast and slow speeds.</p

Examples of sub-types of Parkinson’s disease as a function of site of Lewy Bodies (LB) appearance in sub-cortical and cortical structures, density of LB and time course of spread.

<p>In all cases the presence of LB eventually leads to various stages of dementia and various subtypes of PD.</p

Autism: the micro-movement perspective

The current assessment of behaviors in the inventories to diagnose autism spectrum disorders (ASD) focus on observation and discrete categorizations. Behaviors require movements, yet measurements of physical movements are seldom included. Their inclusion however, could provide an objective characterization of behavior to help unveil interactions between the peripheral and the central nervous systems. Such interactions are critical for the development and maintenance of spontaneous autonomy, self-regulation and voluntary control. At present, current approaches cannot deal with the heterogeneous, dynamic and stochastic nature of development. Accordingly, they leave no avenues for real-time or longitudinal assessments of change in a coping system continuously adapting and developing compensatory mechanisms. We offer a new unifying statistical framework to reveal re-afferent kinesthetic features of the individual with ASD. The new methodology is based on the non-stationary stochastic patterns of minute fluctuations (micro-movements) inherent to our natural actions. Such patterns of behavioral variability provide re-entrant sensory feedback contributing to the autonomous regulation and coordination of the motor output. From an early age, this feedback supports centrally driven volitional control and fluid, flexible transitions between intentional and spontaneous behaviors. We show that in ASD there is a disruption in the maturation of this form of proprioception. Despite this disturbance, each individual has unique adaptive compensatory capabilities that we can unveil and exploit to evoke faster and more accurate decisions. Measuring the kinesthetic re-afference in tandem with stimuli variations we can detect changes in their micro-movements indicative of a more predictive and reliable kinesthetic percept. Our methods address the heterogeneity of ASD with a personalized approach grounded in the inherent sensory-motor abilities that the individual has already developed