An Effective and Efficient Method for Detecting Hands in Egocentric Videos for Rehabilitation Applications

Objective: Individuals with spinal cord injury (SCI) report upper limb function as their top recovery priority. To accurately represent the true impact of new interventions on patient function and independence, evaluation should occur in a natural setting. Wearable cameras can be used to monitor hand function at home, using computer vision to automatically analyze the resulting videos (egocentric video). A key step in this process, hand detection, is difficult to do robustly and reliably, hindering deployment of a complete monitoring system in the home and community. We propose an accurate and efficient hand detection method that uses a simple combination of existing detection and tracking algorithms. Methods: Detection, tracking, and combination methods were evaluated on a new hand detection dataset, consisting of 167,622 frames of egocentric videos collected on 17 individuals with SCI performing activities of daily living in a home simulation laboratory. Results: The F1-scores for the best detector and tracker alone (SSD and Median Flow) were 0.90$\pm$0.07 and 0.42$\pm$0.18, respectively. The best combination method, in which a detector was used to initialize and reset a tracker, resulted in an F1-score of 0.87$\pm$0.07 while being two times faster than the fastest detector alone. Conclusion: The combination of the fastest detector and best tracker improved the accuracy over online trackers while improving the speed of detectors. Significance: The method proposed here, in combination with wearable cameras, will help clinicians directly measure hand function in a patient's daily life at home, enabling independence after SCI.Comment: 7 pages, 3 figures, 5 table

Detecting Hands in Egocentric Videos: Towards Action Recognition

Recently, there has been a growing interest in analyzing human daily activities from data collected by wearable cameras. Since the hands are involved in a vast set of daily tasks, detecting hands in egocentric images is an important step towards the recognition of a variety of egocentric actions. However, besides extreme illumination changes in egocentric images, hand detection is not a trivial task because of the intrinsic large variability of hand appearance. We propose a hand detector that exploits skin modeling for fast hand proposal generation and Convolutional Neural Networks for hand recognition. We tested our method on UNIGE-HANDS dataset and we showed that the proposed approach achieves competitive hand detection results

Designing an Egocentric Video-Based Dashboard to Report Hand Performance Measures for Outpatient Rehabilitation of Cervical Spinal Cord Injury

Background: Functional use of the upper extremities (UEs) is a top recovery priority for individuals with cervical spinal cord injury (cSCI), but the inability to monitor recovery at home and limitations in hand function outcome measures impede optimal recovery. Objectives: We developed a framework using wearable cameras to monitor hand use at home and aimed to identify the best way to report information to clinicians. Methods: A dashboard was iteratively developed with clinician (n = 7) input through focus groups and interviews, creating low-fidelity prototypes based on recurring feedback until no new information emerged. Affinity diagramming was used to identify themes and subthemes from interview data. User stories were developed and mapped to specific features to create a high-fidelity prototype. Results: Useful elements identified for a dashboard reporting hand performance included summaries to interpret graphs, a breakdown of hand posture and activity to provide context, video snippets to qualitatively view hand use at home, patient notes to understand patient satisfaction or struggles, and time series graphing of metrics to measure trends over time. Conclusion: Involving end-users in the design process and breaking down user requirements into user stories helped identify necessary interface elements for reporting hand performance metrics to clinicians. Clinicians recognized the dashboard's potential to monitor rehabilitation progress, provide feedback on hand use, and track progress over time. Concerns were raised about the implementation into clinical practice, therefore further inquiry is needed to determine the tool's feasibility and usefulness in clinical practice for individuals with UE impairments

Separability of neural responses to standardised mechanical stimulation of limbs

Abstract Considerable scientific and technological efforts are currently being made towards the development of neural prostheses. Understanding how the peripheral nervous system responds to electro-mechanical stimulation of the limb, will help to inform the design of prostheses that can restore function or accelerate recovery from injury to the sensory motor system. However, due to differences in experimental protocols, it is difficult, if not impossible, to make meaningful comparisons between different peripheral nerve interfaces. Therefore, we developed a low-cost electronic system to standardise the mechanical stimulation of a rat’s hindpaw. Three types of mechanical stimulations, namely, proprioception, touch and nociception were delivered to the limb and the electroneurogram signals were recorded simultaneously from the sciatic nerve with a 16-contact cuff electrode. For the first time, results indicate separability of neural responses according to stimulus type as well as intensity. Statistical analysis reveal that cuff contacts placed circumferentially, rather than longitudinally, are more likely to lead to higher classification rates. This flexible setup may be readily adapted for systematic comparison of various electrodes and mechanical stimuli in rodents. Hence, we have made its electro-mechanical design and computer programme available onlin

Rehabilitation of hand function after spinal cord injury using a novel handgrip device: a pilot study

BackgroundActivity-based therapy (ABT) for patients with spinal cord injury (SCI), which consists of repetitive use of muscles above and below the spinal lesion, improves locomotion and arm strength. Less data has been published regarding its effects on hand function. We sought to evaluate the effects of a weekly hand-focused therapy program using a novel handgrip device on grip strength and hand function in a SCI cohort.MethodsPatients with SCI were enrolled in a weekly program that involved activities with the MediSens (Los Angeles, CA) handgrip. These included maximum voluntary contraction (MVC) and a tracking task that required each subject to adjust his/her grip strength according to a pattern displayed on a computer screen. For the latter, performance was measured as mean absolute accuracy (MAA). The Spinal Cord Independence Measure (SCIM) was used to measure each subject's independence prior to and after therapy.ResultsSeventeen patients completed the program with average participation duration of 21.3 weeks. The cohort included patients with American Spinal Injury Association (ASIA) Impairment Scale (AIS) A (n = 12), AIS B (n = 1), AIS C (n = 2), and AIS D (n = 2) injuries. The average MVC for the cohort increased from 4.1 N to 21.2 N over 20 weeks, but did not reach statistical significance. The average MAA for the cohort increased from 9.01 to 21.7% at the end of the study (p = .02). The cohort's average SCIM at the end of the study was unchanged compared to baseline.ConclusionsA weekly handgrip-based ABT program is feasible and efficacious at increasing hand task performance in subjects with SCI

A real-time system for biomechanical analysis of human movement and muscle function

Mechanical analysis of movement plays an important role in clinical management of neurological and orthopedic conditions. There has been increasing interest in performing movement analysis in real-time, to provide immediate feedback to both therapist and patient. However, such work to date has been limited to single-joint kinematics and kinetics. Here we present a software system, named human body model (HBM), to compute joint kinematics and kinetics for a full body model with 44 degrees of freedom, in real-time, and to estimate length changes and forces in 300 muscle elements. HBM was used to analyze lower extremity function during gait in 12 able-bodied subjects. Processing speed exceeded 120 samples per second on standard PC hardware. Joint angles and moments were consistent within the group, and consistent with other studies in the literature. Estimated muscle force patterns were consistent among subjects and agreed qualitatively with electromyography, to the extent that can be expected from a biomechanical model. The real-time analysis was integrated into the D-Flow system for development of custom real-time feedback applications and into the gait real-time analysis interactive lab system for gait analysis and gait retraining. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1007/s11517-013-1076-z) contains supplementary material, which is available to authorized users

Imaging fascicular organization of rat sciatic nerves with fast neural electrical impedance tomography

Imaging compound action potentials (CAPs) in peripheral nerves could help avoid side effects in neuromodulation by selective stimulation of identified fascicles. Existing methods have low resolution, limited imaging depth, or are invasive. Fast neural electrical impedance tomography (EIT) allows fascicular CAP imaging with a resolution of <200 µm, <1 ms using a non-penetrating flexible nerve cuff electrode array. Here, we validate EIT imaging in rat sciatic nerve by comparison to micro-computed tomography (microCT) and histology with fluorescent dextran tracers. With EIT, there are reproducible localized changes in tissue impedance in response to stimulation of individual fascicles (tibial, peroneal and sural). The reconstructed EIT images correspond to microCT scans and histology, with significant separation between the fascicles (p < 0.01). The mean fascicle position is identified with an accuracy of 6% of nerve diameter. This suggests fast neural EIT can reliably image the functional fascicular anatomy of the nerves and so aid selective neuromodulation

Grasp Analysis in the Home Environment as a Measure of Hand Function After Cervical Spinal Cord Injury

Background: Following a spinal cord injury, regaining hand function is a top priority. Current hand assessments are conducted in clinics, which may not fully represent real-world hand function. Grasp strategies used in the home environment are an important consideration when examining the impact of rehabilitation interventions. Objective: The main objective of this study is to investigate the relationship between grasp use at home and clinical scores. Method: We used a previously collected dataset in which 21 individuals with spinal cord injuries (SCI) recorded egocentric video while performing activities of daily living in their homes. We manually annotated 4432 hand-object interactions into power, precision, intermediate, and non-prehensile grasps. We examined the distributions of grasp types used and their relationships with clinical assessments. Results: Moderate to strong correlations were obtained between reliance on power grasp and the Spinal Cord Independence Measure III (SCIM; P &lt;.05), the upper extremity motor score (UEMS; P &lt;.01), and the Graded Redefined Assessment of Strength Sensibility and Prehension (GRASSP) Prehension (P &lt;.01) and Strength (P &lt;.01). Negative correlations were observed between the proportion of non-prehensile grasping and SCIM (P &lt;.05), UEMS (P &lt;.05), and GRASSP Prehension (P &lt;.01) and Strength (P &lt;.01). Conclusion: The types of grasp types used in naturalistic activities at home are related to upper limb impairment after cervical SCI. This study provides the first direct demonstration of the importance of hand grasp analysis in the home environment

Egocentric video: a new tool for capturing hand use of individuals with spinal cord injury at home

Abstract Background Current upper extremity outcome measures for persons with cervical spinal cord injury (cSCI) lack the ability to directly collect quantitative information in home and community environments. A wearable first-person (egocentric) camera system is presented that aims to monitor functional hand use outside of clinical settings. Methods The system is based on computer vision algorithms that detect the hand, segment the hand outline, distinguish the user’s left or right hand, and detect functional interactions of the hand with objects during activities of daily living. The algorithm was evaluated using egocentric video recordings from 9 participants with cSCI, obtained in a home simulation laboratory. The system produces a binary hand-object interaction decision for each video frame, based on features reflecting motion cues of the hand, hand shape and colour characteristics of the scene. Results The output from the algorithm was compared with a manual labelling of the video, yielding F1-scores of 0.74 ± 0.15 for the left hand and 0.73 ± 0.15 for the right hand. From the resulting frame-by-frame binary data, functional hand use measures were extracted: the amount of total interaction as a percentage of testing time, the average duration of interactions in seconds, and the number of interactions per hour. Moderate and significant correlations were found when comparing these output measures to the results of the manual labelling, with ρ = 0.40, 0.54 and 0.55 respectively. Conclusions These results demonstrate the potential of a wearable egocentric camera for capturing quantitative measures of hand use at home