Event-based Human Pose Tracking by Spiking Spatiotemporal Transformer

Event camera, as an emerging biologically-inspired vision sensor for capturing motion dynamics, presents new potential for 3D human pose tracking, or video-based 3D human pose estimation. However, existing works in pose tracking either require the presence of additional gray-scale images to establish a solid starting pose, or ignore the temporal dependencies all together by collapsing segments of event streams to form static event frames. Meanwhile, although the effectiveness of Artificial Neural Networks (ANNs, a.k.a. dense deep learning) has been showcased in many event-based tasks, the use of ANNs tends to neglect the fact that compared to the dense frame-based image sequences, the occurrence of events from an event camera is spatiotemporally much sparser. Motivated by the above mentioned issues, we present in this paper a dedicated end-to-end sparse deep learning approach for event-based pose tracking: 1) to our knowledge this is the first time that 3D human pose tracking is obtained from events only, thus eliminating the need of accessing to any frame-based images as part of input; 2) our approach is based entirely upon the framework of Spiking Neural Networks (SNNs), which consists of Spike-Element-Wise (SEW) ResNet and a novel Spiking Spatiotemporal Transformer; 3) a large-scale synthetic dataset is constructed that features a broad and diverse set of annotated 3D human motions, as well as longer hours of event stream data, named SynEventHPD. Empirical experiments demonstrate that, with superior performance over the state-of-the-art (SOTA) ANNs counterparts, our approach also achieves a significant computation reduction of 80% in FLOPS. Furthermore, our proposed method also outperforms SOTA SNNs in the regression task of human pose tracking. Our implementation is available at https://github.com/JimmyZou/HumanPoseTracking_SNN and dataset will be released upon paper acceptance

Action2Motion: Conditioned Generation of 3D Human Motions

Action recognition is a relatively established task, where givenan input sequence of human motion, the goal is to predict its ac-tion category. This paper, on the other hand, considers a relativelynew problem, which could be thought of as an inverse of actionrecognition: given a prescribed action type, we aim to generateplausible human motion sequences in 3D. Importantly, the set ofgenerated motions are expected to maintain itsdiversityto be ableto explore the entire action-conditioned motion space; meanwhile,each sampled sequence faithfully resembles anaturalhuman bodyarticulation dynamics. Motivated by these objectives, we followthe physics law of human kinematics by adopting the Lie Algebratheory to represent thenaturalhuman motions; we also propose atemporal Variational Auto-Encoder (VAE) that encourages adiversesampling of the motion space. A new 3D human motion dataset, HumanAct12, is also constructed. Empirical experiments overthree distinct human motion datasets (including ours) demonstratethe effectiveness of our approach.Comment: 13 pages, ACM MultiMedia 202

Distinctive Stress-Stiffening Responses of Regenerated Silk Fibroin Protein Polymers under Nanoscale Gap Geometries: Effect of Shear on Silk Fibroin-Based Materials

Interfacial dynamics, assembly processes, and changes in nanostructures and mechanical properties of Bombyx mori silk fibroin (SF) proteins under varying degrees of nanoconfinement without and with lateral shear are investigated. When only compressive confinement forces were applied, SF proteins adsorbed on the surfaces experienced conformational changes following the Alexander-de Gennes theory of polymer brushes. By contrast, when SF proteins were exposed to a simultaneous nanoconfinement and shear, remarkable changes in interaction forces were observed, displaying the second order phase transitions, which are attributed to the formation of SF micelles and globular superstructural aggregates via hierarchical assembly processes. The resultant nanostructured SF aggregates show several folds greater elastic moduli than those of SF films prepared by drop-casting and compression-only and even degummed SF fibers. Such a striking improvement in mechanical strength is ascribed to a directional organization of β-sheet nanocrystals, effectively driven by nanoconfinement and shear stress-induced stiffing and ordering mechanisms

Effect of Morphology/Structure on the Phase Behavior and Nonlinear Rheological Properties of NR/SBR Blends

The evolution of the morphology/structure and the nonlinear viscoelasticity of rubber blends under large amounts of strain are key scientific issues for the design and manufacture of rubber blends. The rheological responses of natural rubber/styrene-butadiene rubber (NR/SBR) blends are traced over a wide range of blend compositions to gain an insight into the effect of blend morphology on their nonlinear viscoelasticity. We also prepare NR + SBR physical blends without melt mixing to distinguish the contributions of composition and blend morphology to the viscoelastic response. The microscopic heterogeneous gel-like structure of NR/SBR blends may remarkably weaken their strain softening and improve their modulus hysteretic recovery under large strain, which may be attributed to the heterogeneous microscopic deformation for the NR and SBR phases. Furthermore, additional elastic contribution resulted from the increasing interfacial energy of domain deformation. This may provide some new insights into the effect of blend morphology on the Payne effect of rubber blends