A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives

Extracting behavioral measurements non-invasively from video is stymied by the fact that it is a hard computational problem. Recent advances in deep learning have tremendously advanced predicting posture from videos directly, which quickly impacted neuroscience and biology more broadly. In this primer we review the budding field of motion capture with deep learning. In particular, we will discuss the principles of those novel algorithms, highlight their potential as well as pitfalls for experimentalists, and provide a glimpse into the future.Comment: Review, 21 pages, 8 figures and 5 boxe

Foot and ankle functional morphology in anthropoid primates and Miocene hominoids

Locomotion is essential for survival in many taxa. It also varies greatly among organisms, including primates. Studying locomotor diversity in extant and fossil primates requires an understanding of form-function relationships. This is particularly true in the foot and ankle, as the foot directly contacts the substrate and tarsals are well-represented in the fossil record. Morphological differences alone provide limited aid when inferring locomotion from fossil tarsals in the absence of in vivo biomechanical consideration. This dissertation takes a three-step approach to analyze both in vivo rotations in the foot and ankle as well as morphological variation in tarsal form in extant anthropoid primates and Miocene hominoids and will provide important new data from a poorly understood anatomical region. The amount of talocrural, subtalar, and transverse tarsal rotations among the tibia, calcaneus, and navicular were visualized and quantified during the gait cycles using biplanar fluoroscopy and 3D scans of marked bones, a method known as x-ray reconstruction of moving morphology (XROMM) in rhesus macaques (Macaca mulatta). This study supported previous hypotheses that the midfoot break occurs distal to the cuboid, demonstrated the predominance of plantarflexion/dorsiflexion at the talocrural joint on a flat surface, quantified conjunct rotation at the subtalar joint, showed evidence that the transverse tarsal joint does not function as a single joint complex. Geometric morphometric techniques were used to describe and quantify shape differences in isolated tarsals of extant anthropoid primates. PCA and M/ANOVA analyses were run on a Procrustes-fit landmarks taken on broad range of anthropoid tali (n = 241), calcanei (n = 230), cuboids (n = 282), and naviculars (n = 254). In addition to the typical geometric morphometric techniques, the interlandmark distances that accounted for the greatest amount of variation in this sample were isolated and plotted against centroid size. Phylogenetically controlled generalized least squares analysis revealed which of these measurements were related to locomotion. The relative orientation of the posterior subtalar facet on the talus, talar neck length, calcaneal tuber height, calcaneal anterior length, cuboid length, and navicular anteroposterior length were the morphologies that best separated based on differences in locomotion. The same landmarks were taken on 16 Miocene hominoid tarsals in order to infer foot function based on tarsal form. The geometric morphometric technique of the extant sample allowed for subsetted analyses for incomplete fossils. Early Miocene taxa Ekembo, Proconsul, and Rangwapithecus shared common bony features that suggest that they were generally above branch quadrupeds. Nacholapithecus showed a mixed or varied locomotor behavior. Oreopithecus was shown to not be bipedal, as previously hypothesized, but rather was suspensory. This dissertation provided the first ever quantification of intertarsal and talocrural rotations in anthropoid primate feet and ankles and an analysis of how rotations within and among joints are related. It also provided a quantification of shape differences in tarsals of extant anthropoid primates and fossil Miocene hominoids. Together, the in vivo biomechanics and morphometrics provide insight into form function relationships as well as a foundation for future studies of primate locomotor diversity

Image analysis platforms for exploring genetic and neuronal mechanisms regulating animal behavior

An important aim of neuroscience is to understand how gene interactions and neuronal networks regulate animal behavior. The larvae of the marine annelid Platynereis dumerilii provide a convenient system for such integrative studies. These larvae exhibit a wide range of behaviors, including phototaxis, chemotaxis and gravitaxis and at the same time exhibit relatively simple nervous system organization. Due to its small size and transparent body, the Platynereis larva is compatible with whole-body light microscopic imaging following tissue staining protocols. It is also suitable for serial electron microscopic imaging and subsequent neuronal connectome reconstruction. Despite advances in imaging techniques, automated computational tools for large data analysis are not well-established in Platynereis. In the current work, I developed image analysis software for exploring genetic and nervous system mechanisms modulating Platynereis behavior. Exploring gene expression patterns Current labeling and imaging techniques restrict the number of gene expression patterns that can be labelled and visualized in a single specimen, which hinders the study of behaviors driven by multi-molecular interactions. To address this problem, I employed image registration to generate a gene expression atlas that integrates gene expression information from multiple specimens in a common reference space. The gene expression atlas was used to investigate mechanisms regulating larval locomotion, settlement and phototaxis in Platynereis. The atlas can assist in the identification of inter-individual and inter-species variations in gene expression. To provide a representation convenient for exploring gene expression patterns, I created a model of the atlas using 3D graphics software, which enabled convenient data visualization and efficient data storage and sharing. Exploring neuronal networks regulating behavior Neuronal circuitry can be reconstructed from the images obtained from electron microscopy, which resolves very fine structures such as neuron morphology or synapses. The amount of data resulting from electron microscopy and the complexity of neuronal networks represent a significant challenge for manual analysis. To solve this problem, I developed the NeuroDetective software, which models a neuronal circuitry and analyzes the information flow within it. The software combines the advantages of 3D visualization and graph analysis software by integrating neuron morphology and spatial distribution together with synaptic connectivity. NeuroDetective allowed studying the neuronal circuitry responsible for phototaxis in Platynereis larvae, revealing the connections and the neurons important for the network functionality. NeuroDetective facilitated the establishment of a relationship between the function and the structure of the neuronal circuitry in Platynereis phototaxis. Integrating gene expression patterns with neuronal connectivity Neuronal circuitry and its associated modulating biomolecules, such as neurotransmitters and neuropeptides, are thought to be the main factors regulating animal behavior. Therefore it was important to integrate both genetic and neuronal information in order to fully understand how biomolecules in conjunction with neuronal anatomy elicit certain animal behavior. To resolve the difference in specimen preparation for gene expression versus electron microscopy preparations, I developed an image registration procedure to match the signals from these two different datasets. This method enabled the integration the spatial distribution of specific modulators into the analysis of neuronal networks, leading to an improved understanding of the genetic and neuronal mechanisms modulating behavior in Platynereis

Shape Dynamical Models for Activity Recognition and Coded Aperture Imaging for Light-Field Capture

Classical applications of Pattern recognition in image processing and computer vision have typically dealt with modeling, learning and recognizing static patterns in images and videos. There are, of course, in nature, a whole class of patterns that dynamically evolve over time. Human activities, behaviors of insects and animals, facial expression changes, lip reading, genetic expression profiles are some examples of patterns that are dynamic. Models and algorithms to study these patterns must take into account the dynamics of these patterns while exploiting the classical pattern recognition techniques. The first part of this dissertation is an attempt to model and recognize such dynamically evolving patterns. We will look at specific instances of such dynamic patterns like human activities, and behaviors of insects and develop algorithms to learn models of such patterns and classify such patterns. The models and algorithms proposed are validated by extensive experiments on gait-based person identification, activity recognition and simultaneous tracking and behavior analysis of insects. The problem of comparing dynamically deforming shape sequences arises repeatedly in problems like activity recognition and lip reading. We describe and evaluate parametric and non-parametric models for shape sequences. In particular, we emphasize the need to model activity execution rate variations and propose a non-parametric model that is insensitive to such variations. These models and the resulting algorithms are shown to be extremely effective for a wide range of applications from gait-based person identification to human action recognition. We further show that the shape dynamical models are not only effective for the problem of recognition, but also can be used as effective priors for the problem of simultaneous tracking and behavior analysis. We validate the proposed algorithm for performing simultaneous behavior analysis and tracking on videos of bees dancing in a hive. In the last part of this dissertaion, we investigate computational imaging, an emerging field where the process of image formation involves the use of a computer. The current trend in computational imaging is to capture as much information about the scene as possible during capture time so that appropriate images with varying focus, aperture, blur and colorimetric settings may be rendered as required. In this regard, capturing the 4D light-field as opposed to a 2D image allows us to freely vary viewpoint and focus at the time of rendering an image. In this dissertation, we describe a theoretical framework for reversibly modulating {4D} light fields using an attenuating mask in the optical path of a lens based camera. Based on this framework, we present a novel design to reconstruct the {4D} light field from a {2D} camera image without any additional refractive elements as required by previous light field cameras. The patterned mask attenuates light rays inside the camera instead of bending them, and the attenuation recoverably encodes the rays on the {2D} sensor. Our mask-equipped camera focuses just as a traditional camera to capture conventional {2D} photos at full sensor resolution, but the raw pixel values also hold a modulated {4D} light field. The light field can be recovered by rearranging the tiles of the {2D} Fourier transform of sensor values into {4D} planes, and computing the inverse Fourier transform. In addition, one can also recover the full resolution image information for the in-focus parts of the scene

THE PEDIATRIC FLAT FOOT: PRE AND POST SURGICAL CORRECTION 3D KINEMATICS DATA

Introduction: aim of this study was to establish normality parameters and analyze 3D kinematic data before and after surgical correction of the pediatric flexible flat foot Materials and methods: study population was composed of 2 groups: 10 children (20 feet, 5M/5F)without any disorders of the foot were evaluated to obtain normal reference data; 20 children with bilateral flexible flatfoot candidate to bilateral surgical correction (40 feet, 13M/7F) The RFM -3D kinematics protocol was used. Clinical, radiographic and instrumental evaluation were performed preoperatively and at 12 months by the same surgeon An arthroereisis of the subtalar joint was performed by the same surgeon. Patients were divided in 3 groups:1:normality;2:before surgery;3: after surgery. For all the variables and for the three planes of the space comparison between groups were performed. Results: 3D rotational joint variables and planar angles were defined for normality, before and after sur-gery at the upright standing position. Differences were observed: hind foot , frontal plane; Chopart Joint ,transverse plane; Lisfanc Joint, frontal/transverse planes; ratio between 1rst and 2nd metatarsal, transverse plane; 2nd and 5th metatarsal versus ground respectively, sagittal plane; MLA, transverse plane Discussion/conclusions:: different variables, normalized after correction, suggest that surgery performed at the hind foot can also improves mid foot pronation, increases the medial longitudinal arch and im-proves ratio between metatarsal bones, allowing to quantify changes that clinical and radiological evaluation cannot provide. The pediatric foot is similar to the adults and pediatric flexible flat foot could be corrected surgically, even if painless

Development of a Probabilistic Chimpanzee Glenohumeral Model: Implications for Human Function

Modern human shoulder function is affected by the evolutionary adaptations that have occurred to ensure survival and prosperity of the species. Robust examination of behavioral shoulder performance and injury risk can be holistically improved through an interdisciplinary approach that integrates anthropology and biomechanics. Anthropological research methods have attempted to resolve gaps in human shoulder evolution, while biomechanics research has attempted to explain the musculoskeletal function of the modern human shoulder. Coordination of these two fields can allow different perspectives to contribute to a more complete interpretation of, not only how the modern human shoulder is susceptible to specific injuries, but also why. How the modern human shoulder arose from a, likely, weight-bearing, arboreal past to its modern form, and what this has meant for modern behaviors, is not well understood. Despite a weight-bearing, locomotor ancestral usage, the modern human upper extremity is highly fatigable in repetitive, low to moderate force tasks, such as overhead reaching. The closest living human relative, modern chimpanzees, has retained an arboreal, locomotor upper extremity. Interdisciplinary comparative research on humans and chimpanzees could lead to greater insight on modern human shoulder function. The purpose of this research was to explore the modern human capacity for ancestral, brachiating behaviors and resultant injury mechanisms through comparative experimental, computational modeling and probabilistic modeling approaches with chimpanzees. The first study experimentally explored the modern human ability to perform a horizontal bimanual arm-suspensory climbing task. EMG of 12 upper extremity muscles and motion capture of the arm and thorax were monitored in experienced and inexperienced climbers. Results were also compared to previously published or collected data on chimpanzees performing an analogous task. While all human climbers used a high proportion of their muscular reserve to perform the task, experienced climbers had moderately reduced muscle activity in most muscles, particularly during phasic shifts of the climb cycle and moderately more efficient kinematics. Climbing kinematics and muscle activity were very similar between humans and chimpanzees. However, chimpanzees appear to have a different utility of the posterior deltoid, suggesting an influence of their habitual arboreal behaviors. The second and third studies describe the development of a deterministic chimpanzee musculoskeletal glenohumeral model. Study 2 developed geometric parameters of chimpanzee shoulder rhythm and glenoid stability ratios for the construction of a chimpanzee glenohumeral model. The chimpanzee glenohumeral model of Study 3 was designed to parallel an existing human glenohumeral model, enabling comparative analyses. The chimpanzee glenohumeral model consists of three modules – an external torque module, musculoskeletal geometric module, and an internal muscle force prediction module. Together, these modules use postural kinematics, subject specific anthropometrics and hand forces to estimate joint reaction forces and moments, subacromial space dimensions, and muscle and tissue forces. Using static postural data from Study 1, predicted muscle forces and subacromial space were compared between chimpanzees and humans during an overhead, weight-bearing climbing task. Compared to chimpanzees, the human model predicted a 2mm narrower subacromial space, deltoid muscle forces that were often double those of chimpanzees and a strong reliance on infraspinatus and teres minor (60-100% maximal force) over other rotator cuff muscles. Finally, the deterministic chimpanzee and human glenohumeral models were expanded in Study 4 to a probabilistic analysis of rotator cuff function between species. Using probabilistic software and the same postural climbing inputs, both models had anthropologically relevant musculoskeletal features perturbed in a series of Monte Carlo simulations – muscle origins and insertions, glenoid inclination and glenoid stability – to determine if rotator cuff muscle force prediction distributions would converge between species. Human rotator cuff muscle behavior did not converge with chimpanzees using probabilistic simulation. The human model continued to predict strong dependence on infraspinatus and teres minor, with 99% confidence intervals of [0-100]% maximal force, over supraspinatus and subscapularis, with confidence intervals of [0-5]% maximal force. Chimpanzee rotator cuff confidence intervals were typically between [0-40]% maximal force, with median force for all four rotator cuff muscles typically 5-20% maximal force. While perturbation of muscle origins and insertions had the greatest effect on muscle force output distributions, no musculoskeletal variation notably modified human climbing performance. Structural musculoskeletal differences between species dictated differences in glenohumeral function. The results from all studies indicate susceptibility for the fatigue-induced initiation of subacromial impingement syndrome and rotator cuff pathology in modern humans during overhead and repetitive tasks. Lower muscle absolute PCSA in humans, combined with a laterally oriented glenohumeral joint and laterally projecting acromion reduced the capacity for overhead postures and weight-bearing postures. These evolutionary differences may have been vestigial consequences, concurrent with necessary adaptions for important, evolutionary human-centric behaviors such as throwing. However, they have influenced the high rates of rotator cuff pathology in humans compared to closely related primates. The present work represents an important first step toward a broad scope of future research. Interdisciplinary computational modeling offers an evolving and improving alternative to traditional methods to study human evolution and function. Computational and probabilistic simulations can be expanded to numerous other biomechanical and evolutionary queries. The results of this thesis are a promising initial step to examining the evolutionary structural connection to biomechanical human function through comparative computational modeling