Markerless Motion Capture via Convolutional Neural Network

A human motion capture system can be defined as a process that digitally records the movements of a person and then translates them into computer-animated images. To achieve this goal, motion capture systems usually exploit different types of algorithms, which include techniques such as pose estimation or background subtraction: this latter aims at segmenting moving objects from the background under multiple challenging scenarios. Recently, encoder-decoder-type deep neural networks designed to accomplish this task have reached impressive results, outperforming classical approaches. The aim of this thesis is to evaluate and discuss the predictions provided by the multi-scale convolutional neural network FgSegNet_v2, a deep learning-based method which represents the current state-of-the-art for implementing scene-specific background subtraction. In this work, FgSegNet_v2 is trained and tested on BBSoF S.r.l. dataset, extending its scene- specific use to a more general application in several environments

A Continuous Grasp Representation for the Imitation Learning of Grasps on Humanoid Robots

Models and methods are presented which enable a humanoid robot to learn reusable, adaptive grasping skills. Mechanisms and principles in human grasp behavior are studied. The findings are used to develop a grasp representation capable of retaining specific motion characteristics and of adapting to different objects and tasks. Based on the representation a framework is proposed which enables the robot to observe human grasping, learn grasp representations, and infer executable grasping actions

Learning Action Maps of Large Environments via First-Person Vision

When people observe and interact with physical spaces, they are able to associate functionality to regions in the environment. Our goal is to automate dense functional understanding of large spaces by leveraging sparse activity demonstrations recorded from an ego-centric viewpoint. The method we describe enables functionality estimation in large scenes where people have behaved, as well as novel scenes where no behaviors are observed. Our method learns and predicts "Action Maps", which encode the ability for a user to perform activities at various locations. With the usage of an egocentric camera to observe human activities, our method scales with the size of the scene without the need for mounting multiple static surveillance cameras and is well-suited to the task of observing activities up-close. We demonstrate that by capturing appearance-based attributes of the environment and associating these attributes with activity demonstrations, our proposed mathematical framework allows for the prediction of Action Maps in new environments. Additionally, we offer a preliminary glance of the applicability of Action Maps by demonstrating a proof-of-concept application in which they are used in concert with activity detections to perform localization.Comment: To appear at CVPR 201

Neuron-level dynamics of oscillatory network structure and markerless tracking of kinematics during grasping

Oscillatory synchrony is proposed to play an important role in flexible sensory-motor transformations. Thereby, it is assumed that changes in the oscillatory network structure at the level of single neurons lead to flexible information processing. Yet, how the oscillatory network structure at the neuron-level changes with different behavior remains elusive. To address this gap, we examined changes in the fronto-parietal oscillatory network structure at the neuron-level, while monkeys performed a flexible sensory-motor grasping task. We found that neurons formed separate subnetworks in the low frequency and beta bands. The beta subnetwork was active during steady states and the low frequency network during active states of the task, suggesting that both frequencies are mutually exclusive at the neuron-level. Furthermore, both frequency subnetworks reconfigured at the neuron-level for different grip and context conditions, which was mostly lost at any scale larger than neurons in the network. Our results, therefore, suggest that the oscillatory network structure at the neuron-level meets the necessary requirements for the coordination of flexible sensory-motor transformations. Supplementarily, tracking hand kinematics is a crucial experimental requirement to analyze neuronal control of grasp movements. To this end, a 3D markerless, gloveless hand tracking system was developed using computer vision and deep learning techniques. 2021-11-3

Markerless Analysis of Upper Extremity Kinematics during Standardized Pediatric Assessment

Children with hemiplegic cerebral palsy experience reduced motor performance in the affected upper extremity and are typically evaluated based on degree of functional impairment using activity-based assessments such as the Shriners Hospitals for Children Upper Extremity Evaluation (SHUEE), a validated clinical measure, to describe performance prior to and following rehabilitative or surgical interventions. Evaluations rely on subjective therapist scoring techniques and lack sensitivity to detect change. Objective clinical motion analysis systems are an available but time-consuming and cost-intensive alternative, requiring uncomfortable application of markers to the patient. There is currently no available markerless, low-cost system that quantitatively assesses upper extremity kinematics to improve sensitivity of evaluation during standardized task performance. A motion analysis system was developed, using Microsoft Kinect hardware to track motion during broad arm and subtle hand and finger movements. Algorithms detected and recorded skeletal position and calculated angular kinematics. Lab-developed articulating hand model and elbow fixation devices were used to evaluate accuracy, intra-trial, and inter-trial reliability of the Kinect platform. Results of technical evaluation indicate reasonably accurate detection and differentiation between hand and arm positions. Twelve typically-developing adolescent subjects were tested to characterize and evaluate performance scores obtained from the SHUEE and Kinect motion analysis system. Feasibility of the platform was determined in terms of kinematics and as an enhancement of quantitative kinematic reporting to the SHUEE, and a population mean of typically developing subject kinematics obtained for future development of performance scoring algorithms. The system was observed to be easily operable and clinically effective in subject testing. The Kinect motion analysis platform developed to quantify upper extremity motion during standardized tasks is a low-cost, portable, accurate, and reliable system in kinematic reporting, and has demonstrated quality of results in both technical evaluation of the system and a study of its applicability to standardized task-based evaluation, but has hardware and software limitations which will be resolved in future improvements of the system. The SHUEE benefits from improved quantitative data, and the Kinect system provides enhanced sensitivity in clinical upper extremity analysis for children with hemiplegic cerebral palsy

Ambient Intelligence for Next-Generation AR

Next-generation augmented reality (AR) promises a high degree of context-awareness - a detailed knowledge of the environmental, user, social and system conditions in which an AR experience takes place. This will facilitate both the closer integration of the real and virtual worlds, and the provision of context-specific content or adaptations. However, environmental awareness in particular is challenging to achieve using AR devices alone; not only are these mobile devices' view of an environment spatially and temporally limited, but the data obtained by onboard sensors is frequently inaccurate and incomplete. This, combined with the fact that many aspects of core AR functionality and user experiences are impacted by properties of the real environment, motivates the use of ambient IoT devices, wireless sensors and actuators placed in the surrounding environment, for the measurement and optimization of environment properties. In this book chapter we categorize and examine the wide variety of ways in which these IoT sensors and actuators can support or enhance AR experiences, including quantitative insights and proof-of-concept systems that will inform the development of future solutions. We outline the challenges and opportunities associated with several important research directions which must be addressed to realize the full potential of next-generation AR.Comment: This is a preprint of a book chapter which will appear in the Springer Handbook of the Metavers