Jointly Learning Energy Expenditures and Activities using Egocentric Multimodal Signals

Physiological signals such as heart rate can provide valuable information about an individual’s state and activity. However, existing work on computer vision has not yet explored leveraging these signals to enhance egocentric video understanding. In this work, we propose a model for reasoning on multimodal data to jointly predict activities and energy expenditures. We use heart rate signals as privileged self-supervision to derive energy expenditure in a training stage. A multitask objective is used to jointly optimize the two tasks. Additionally, we introduce a dataset that contains 31 hours of egocentric video augmented with heart rate and acceleration signals. This study can lead to new applications such as a visual calorie counter

Physical Activity Comparison Between Body Sides in Hemiparetic Patients Using Wearable Motion Sensors in Free-Living and Therapy: A Case Series

Background: Physical activity (PA) is essential in stroke rehabilitation of hemiparetic patients to avoid health risks, and moderate to vigorous PA could promote patients' recovery. However, PA assessments are limited to clinical environments. Little is known about PA in unguided free-living. Wearable sensors could reveal patients' PA during rehabilitation, and day-long long-term measurements over several weeks might reveal recovery trends of affected and less-affected body sides.Methods: We investigated PA in an observation study during outpatient rehabilitation in a day-care center. PA of affected and less-affected body sides, including upper and lower limbs were derived using wearable motion sensors. In this analysis we focused on PA during free-living and clinician guided therapies, and investigated differences between body-sides. Linear regressions were used to estimate metabolic equivalents for each limb at comparable scale. Non-parametric statistics were derived to quantify PA differences between body sides.Results: We analyzed 102 full-day movement data recordings from eleven hemiparetic patients during individual rehabilitation periods up to 79 days. The comparison between free-living and clinician guided therapy showed on average 16.1 % higher PA in the affected arm during therapy and 5.3 % higher PA in the affected leg during therapy. Average differences between free-living and therapy in the less-affected side were below 4.5 %.Conclusion: We analyzed PA of patients with a hemiparesis in two distinct rehabilitation settings, including free-living and clinician guided therapies over several weeks and compared MET values of affected and less-affected body sides. In particular, we investigated PA using individual regression models for each limb. We demonstrated that wearable motion sensors provide insights in patient's PA during rehabilitation. Although, no clear PA trends were found, our analysis showed patients' tendency to sedentary behavior, confirming previous lab study results. Our PA analysis approach could be used beyond clinical rehabilitation to devise personalized patient and limb-specific exercise recommendations in future remote rehabilitation