Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs

We address the problem of making human motion capture in the wild more practical by using a small set of inertial sensors attached to the body. Since the problem is heavily under-constrained, previous methods either use a large number of sensors, which is intrusive, or they require additional video input. We take a different approach and constrain the problem by: (i) making use of a realistic statistical body model that includes anthropometric constraints and (ii) using a joint optimization framework to fit the model to orientation and acceleration measurements over multiple frames. The resulting tracker Sparse Inertial Poser (SIP) enables 3D human pose estimation using only 6 sensors (attached to the wrists, lower legs, back and head) and works for arbitrary human motions. Experiments on the recently released TNT15 dataset show that, using the same number of sensors, SIP achieves higher accuracy than the dataset baseline without using any video data. We further demonstrate the effectiveness of SIP on newly recorded challenging motions in outdoor scenarios such as climbing or jumping over a wall.Comment: 12 pages, Accepted at Eurographics 201

Learning to Extract Motion from Videos in Convolutional Neural Networks

This paper shows how to extract dense optical flow from videos with a convolutional neural network (CNN). The proposed model constitutes a potential building block for deeper architectures to allow using motion without resorting to an external algorithm, \eg for recognition in videos. We derive our network architecture from signal processing principles to provide desired invariances to image contrast, phase and texture. We constrain weights within the network to enforce strict rotation invariance and substantially reduce the number of parameters to learn. We demonstrate end-to-end training on only 8 sequences of the Middlebury dataset, orders of magnitude less than competing CNN-based motion estimation methods, and obtain comparable performance to classical methods on the Middlebury benchmark. Importantly, our method outputs a distributed representation of motion that allows representing multiple, transparent motions, and dynamic textures. Our contributions on network design and rotation invariance offer insights nonspecific to motion estimation

Motion magnification in coronal seismology

We introduce a new method for the investigation of low-amplitude transverse oscillations of solar plasma non-uniformities, such as coronal loops, individual strands in coronal arcades, jets, prominence fibrils, polar plumes, and other contrast features, observed with imaging instruments. The method is based on the two-dimensional dual tree complex wavelet transform (DT$\mathbb{C}$WT). It allows us to magnify transverse, in the plane-of-the-sky, quasi-periodic motions of contrast features in image sequences. The tests performed on the artificial data cubes imitating exponentially decaying, multi-periodic and frequency-modulated kink oscillations of coronal loops showed the effectiveness, reliability and robustness of this technique. The algorithm was found to give linear scaling of the magnified amplitudes with the original amplitudes provided they are sufficiently small. Also, the magnification is independent of the oscillation period in a broad range of the periods. The application of this technique to SDO/AIA EUV data cubes of a non-flaring active region allowed for the improved detection of low-amplitude decay-less oscillations in the majority of loops.Comment: Accepted for publication in Solar Physic

Local Visual Microphones: Improved Sound Extraction from Silent Video

Sound waves cause small vibrations in nearby objects. A few techniques exist in the literature that can extract sound from video. In this paper we study local vibration patterns at different image locations. We show that different locations in the image vibrate differently. We carefully aggregate local vibrations and produce a sound quality that improves state-of-the-art. We show that local vibrations could have a time delay because sound waves take time to travel through the air. We use this phenomenon to estimate sound direction. We also present a novel algorithm that speeds up sound extraction by two to three orders of magnitude and reaches real-time performance in a 20KHz video.Comment: Accepted to BMVC 201

Overcoming Bandwidth Limitations in Wireless Sensor Networks by Exploitation of Cyclic Signal Patterns: An Event-triggered Learning Approach

Wireless sensor networks are used in a wide range of applications, many of which require real-time transmission of the measurements. Bandwidth limitations result in limitations on the sampling frequency and number of sensors. This problem can be addressed by reducing the communication load via data compression and event-based communication approaches. The present paper focuses on the class of applications in which the signals exhibit unknown and potentially time-varying cyclic patterns. We review recently proposed event-triggered learning (ETL) methods that identify and exploit these cyclic patterns, we show how these methods can be applied to the nonlinear multivariable dynamics of three-dimensional orientation data, and we propose a novel approach that uses Gaussian process models. In contrast to other approaches, all three ETL methods work in real time and assure a small upper bound on the reconstruction error. The proposed methods are compared to several conventional approaches in experimental data from human subjects walking with a wearable inertial sensor network. They are found to reduce the communication load by 60–70%, which implies that two to three times more sensor nodes could be used at the same bandwidth

Modulated Oscillations in Three Dimensions

The analysis of the fully three-dimensional and time-varying polarization characteristics of a modulated trivariate, or three-component, oscillation is addressed. The use of the analytic operator enables the instantaneous three-dimensional polarization state of any square-integrable trivariate signal to be uniquely defined. Straightforward expressions are given which permit the ellipse parameters to be recovered from data. The notions of instantaneous frequency and instantaneous bandwidth, generalized to the trivariate case, are related to variations in the ellipse properties. Rates of change of the ellipse parameters are found to be intimately linked to the first few moments of the signal's spectrum, averaged over the three signal components. In particular, the trivariate instantaneous bandwidth---a measure of the instantaneous departure of the signal from a single pure sinusoidal oscillation---is found to contain five contributions: three essentially two-dimensional effects due to the motion of the ellipse within a fixed plane, and two effects due to the motion of the plane containing the ellipse. The resulting analysis method is an informative means of describing nonstationary trivariate signals, as is illustrated with an application to a seismic record.Comment: IEEE Transactions on Signal Processing, 201

Muscle Synergies Facilitate Computational Prediction of Subject-Specific Walking Motions.

Researchers have explored a variety of neurorehabilitation approaches to restore normal walking function following a stroke. However, there is currently no objective means for prescribing and implementing treatments that are likely to maximize recovery of walking function for any particular patient. As a first step toward optimizing neurorehabilitation effectiveness, this study develops and evaluates a patient-specific synergy-controlled neuromusculoskeletal simulation framework that can predict walking motions for an individual post-stroke. The main question we addressed was whether driving a subject-specific neuromusculoskeletal model with muscle synergy controls (5 per leg) facilitates generation of accurate walking predictions compared to a model driven by muscle activation controls (35 per leg) or joint torque controls (5 per leg). To explore this question, we developed a subject-specific neuromusculoskeletal model of a single high-functioning hemiparetic subject using instrumented treadmill walking data collected at the subject's self-selected speed of 0.5 m/s. The model included subject-specific representations of lower-body kinematic structure, foot-ground contact behavior, electromyography-driven muscle force generation, and neural control limitations and remaining capabilities. Using direct collocation optimal control and the subject-specific model, we evaluated the ability of the three control approaches to predict the subject's walking kinematics and kinetics at two speeds (0.5 and 0.8 m/s) for which experimental data were available from the subject. We also evaluated whether synergy controls could predict a physically realistic gait period at one speed (1.1 m/s) for which no experimental data were available. All three control approaches predicted the subject's walking kinematics and kinetics (including ground reaction forces) well for the model calibration speed of 0.5 m/s. However, only activation and synergy controls could predict the subject's walking kinematics and kinetics well for the faster non-calibration speed of 0.8 m/s, with synergy controls predicting the new gait period the most accurately. When used to predict how the subject would walk at 1.1 m/s, synergy controls predicted a gait period close to that estimated from the linear relationship between gait speed and stride length. These findings suggest that our neuromusculoskeletal simulation framework may be able to bridge the gap between patient-specific muscle synergy information and resulting functional capabilities and limitations

Estimation Of Multiple Local Orientations In Image Signals

Local orientation estimation can be posed as the problem of finding the minimum grey level variance axis within a local neighbourhood. In 2D image signals, this corresponds to the eigensystem analysis of a 22-tensor, which yields valid results for single orientations. We describe extensions to multiple overlaid orientations, which may be caused by transparent objects, crossings, bifurcations, corners etc. Multiple orientation detection is based on the eigensystem analysis of an appropriately extended tensor, yielding so-called mixed orientation parameters. These mixed orientation parameters can be regarded as another tensor built from the sought individual orientation parameters. We show how the mixed orientation tensor can be decomposed into the individual orientations by finding the roots of a polynomial. Applications are, e.g., in directional filtering and interpolation, feature extraction for corners or crossings, and signal separation