Force-Aware Interface via Electromyography for Natural VR/AR Interaction

While tremendous advances in visual and auditory realism have been made for virtual and augmented reality (VR/AR), introducing a plausible sense of physicality into the virtual world remains challenging. Closing the gap between real-world physicality and immersive virtual experience requires a closed interaction loop: applying user-exerted physical forces to the virtual environment and generating haptic sensations back to the users. However, existing VR/AR solutions either completely ignore the force inputs from the users or rely on obtrusive sensing devices that compromise user experience. By identifying users' muscle activation patterns while engaging in VR/AR, we design a learning-based neural interface for natural and intuitive force inputs. Specifically, we show that lightweight electromyography sensors, resting non-invasively on users' forearm skin, inform and establish a robust understanding of their complex hand activities. Fuelled by a neural-network-based model, our interface can decode finger-wise forces in real-time with 3.3% mean error, and generalize to new users with little calibration. Through an interactive psychophysical study, we show that human perception of virtual objects' physical properties, such as stiffness, can be significantly enhanced by our interface. We further demonstrate that our interface enables ubiquitous control via finger tapping. Ultimately, we envision our findings to push forward research towards more realistic physicality in future VR/AR.Comment: ACM Transactions on Graphics (SIGGRAPH Asia 2022

A model-based approach to predict muscle synergies using optimization: application to feedback control

This Document is Protected by copyright and was first published by Frontiers. All rights reserved. it is reproduced with permission.This paper presents a new model-based method to define muscle synergies. Unlike the conventional factorization approach, which extracts synergies from electromyographic data, the proposed method employs a biomechanical model and formally defines the synergies as the solution of an optimal control problem. As a result, the number of required synergies is directly related to the dimensions of the operational space. The estimated synergies are posture-dependent, which correlate well with the results of standard factorization methods. Two examples are used to showcase this method: a two-dimensional forearm model, and a three-dimensional driver arm model. It has been shown here that the synergies need to be task-specific (i.e., they are defined for the specific operational spaces: the elbow angle and the steering wheel angle in the two systems). This functional definition of synergies results in a low-dimensional control space, in which every force in the operational space is accurately created by a unique combination of synergies. As such, there is no need for extra criteria (e.g., minimizing effort) in the process of motion control. This approach is motivated by the need for fast and bio-plausible feedback control of musculoskeletal systems, and can have important implications in engineering, motor control, and biomechanics.The authors wish to thank the Natural Sciences and Engineering Research Council of Canada (NSERC) for funding this study

Simplified Hand Configuration for Object Manipulation

This work is focused on obtaining realistic human hand models that are suitable for manipulation tasks. Firstly, a 24 DOF kinematic model of the human hand is defined. This model is based on the human skeleton. Intra-finger and inter-finger constraints have been included in order to improve the movement realism. Secondly, two simplified hand descriptions (9 and 6 DOF) have been developed according to the constraints predefined. These simplified models involve some errors in reconstructing the hand posture. These errors are calculated with respect to the 24 DOF model and evaluated according to the hand gestures. Finally, some criteria are defined by which to select the hand description best suited to the features of the manipulation task

A myoelectric digital twin for fast and realistic modelling in deep learning

Muscle electrophysiology has emerged as a powerful tool to drive human machine interfaces, with many new recent applications outside the traditional clinical domains, such as robotics and virtual reality. However, more sophisticated, functional, and robust decoding algorithms are required to meet the fine control requirements of these applications. Deep learning has shown high potential in meeting these demands, but requires a large amount of high-quality annotated data, which is expensive and time-consuming to acquire. Data augmentation using simulations, a strategy applied in other deep learning applications, has never been attempted in electromyography due to the absence of computationally efficient models. We introduce a concept of Myoelectric Digital Twin - highly realistic and fast computational model tailored for the training of deep learning algorithms. It enables simulation of arbitrary large and perfectly annotated datasets of realistic electromyography signals, allowing new approaches to muscular signal decoding, accelerating the development of human-machine interfaces

BIOMECHANICS OF SPORTS - SELECTED EXAMPLES OF SUCCESSFUL APPLICATIONS AND FUTURE PERSPECTIVES

The performance criteria of physical activities, especially those of sport disciplines, can usually be defined in biomechanico-mathematical terms. This implies that sufficiently complex models of the human neuromusculoskeletal system can be used for the simulation and analysis of sports motions and, at least in principle, for the biomechanical optimization of the performance in the various sport disciplines. It is frequently forgotten that biomechanical optimization, in the widest sense, is the ultimate goal of the majority of all endeavors in sports biomechanics, even if this may not always be obvious. Examples of the successful generation of biomechanical models include adequate models of the human skeletal and muscular subsystem, and the creation of a functional racket-hand-arm system model for simulating tennis strokes. Simultaneously, anthropometrico-computational and dynamometric methods were developed for determining respectively the subject-specific segmental and myodynamic parameter sets. The models and methods just mentioned will be illustrated during the oral presentation. As regards the practical applications of the biomechanical modelling approach to sports, some selected examples also to be presented are: the complete optimization of a kicking motion; the successful computer simulation and analysis of a rock ' n roll Betterini somersault in connection with an accident requiring a biomechanical expert opinion; the development of an objective biomechanical method for testing the quality criteria of tennis rackets; the quantification of the variability of repeated sports motions; and investigations into the validity and reliability of vertical jumping performance testing methods. Needless to say that appropriate biomechanical models of the human neuromusculoskeletal system are indispensable in theoretical studies such as the demonstration of the comparatively high insensivity of skeletal motions to neural control perturbations. Considering the current state of the art it would appear that contemporary biomechanics of sports is still too pre-occupied with measurement, data collection, and the subsequent phenomenological description of an observed event instead of asking the (much more difficult) question concerning the causes and fundamental mechanisms underlying the observed phenomenon. The mere measurement and description of the ground reaction forces during the release phase of the javelin throw, for instance, without relating their significance to the musculoskeletal factors that determine the throwing distance, is meaningless and constitutes a futile exercise. As a future trend in sport biomechanics, the utilization of models for performance optimization may be expected to gain increasing importance

Evaluation of a subject-specific, torque-driven computer simulation model of one-handed tennis backhand ground strokes

A torque-driven, subject-specific 3-D computer simulation model of the impact phase of one-handed tennis backhand strokes was evaluated by comparing performance and simulation results. Backhand strokes of an elite subject were recorded on an artificial tennis court. Over the 50-ms period after impact, good agreement was found with an overall RMS difference of 3.3° between matching simulation and performance in terms of joint and racket angles. Consistent with previous experimental research, the evaluation process showed that grip tightness and ball impact location are important factors that affect postimpact racket and arm kinematics. Associated with these factors, the model can be used for a better understanding of the eccentric contraction of the wrist extensors during one-handed backhand ground strokes, a hypothesized mechanism of tennis elbow

Articular human joint modelling

Copyright @ Cambridge University Press 2009.The work reported in this paper encapsulates the theories and algorithms developed to drive the core analysis modules of the software which has been developed to model a musculoskeletal structure of anatomic joints. Due to local bone surface and contact geometry based joint kinematics, newly developed algorithms make the proposed modeller different from currently available modellers. There are many modellers that are capable of modelling gross human body motion. Nevertheless, none of the available modellers offer complete elements of joint modelling. It appears that joint modelling is an extension of their core analysis capability, which, in every case, appears to be musculoskeletal motion dynamics. It is felt that an analysis framework that is focused on human joints would have significant benefit and potential to be used in many orthopaedic applications. The local mobility of joints has a significant influence in human motion analysis, in understanding of joint loading, tissue behaviour and contact forces. However, in order to develop a bone surface based joint modeller, there are a number of major problems, from tissue idealizations to surface geometry discretization and non-linear motion analysis. This paper presents the following: (a) The physical deformation of biological tissues as linear or non-linear viscoelastic deformation, based on spring-dashpot elements. (b) The linear dynamic multibody modelling, where the linear formulation is established for small motions and is particularly useful for calculating the equilibrium position of the joint. This model can also be used for finding small motion behaviour or loading under static conditions. It also has the potential of quantifying the joint laxity. (c) The non-linear dynamic multibody modelling, where a non-matrix and algorithmic formulation is presented. The approach allows handling complex material and geometrical nonlinearity easily. (d) Shortest path algorithms for calculating soft tissue line of action geometries. The developed algorithms are based on calculating minimum ‘surface mass’ and ‘surface covariance’. An improved version of the ‘surface covariance’ algorithm is described as ‘residual covariance’. The resulting path is used to establish the direction of forces and moments acting on joints. This information is needed for linear or non-linear treatment of the joint motion. (e) The final contribution of the paper is the treatment of the collision. In the virtual world, the difficulty in analysing bodies in motion arises due to body interpenetrations. The collision algorithm proposed in the paper involves finding the shortest projected ray from one body to the other. The projection of the body is determined by the resultant forces acting on it due to soft tissue connections under tension. This enables the calculation of collision condition of non-convex objects accurately. After the initial collision detection, the analysis involves attaching special springs (stiffness only normal to the surfaces) at the ‘potentially colliding points’ and motion of bodies is recalculated. The collision algorithm incorporates the rotation as well as translation. The algorithm continues until the joint equilibrium is achieved. Finally, the results obtained based on the software are compared with experimental results obtained using cadaveric joints

Predictive Simulation of Reaching Moving Targets Using Nonlinear Model Predictive Control

This Document is Protected by copyright and was first published by Frontiers. All rights reserved. it is reproduced with permission.This article investigates the application of optimal feedback control to trajectory planning in voluntary human arm movements. A nonlinear model predictive controller (NMPC) with a finite prediction horizon was used as the optimal feedback controller to predict the hand trajectory planning and execution of planar reaching tasks. The NMPC is completely predictive, and motion tracking or electromyography data are not required to obtain the limb trajectories. To present this concept, a two degree of freedom musculoskeletal planar arm model actuated by three pairs of antagonist muscles was used to simulate the human arm dynamics. This study is based on the assumption that the nervous system minimizes the muscular effort during goal-directed movements. The effects of prediction horizon length on the trajectory, velocity profile, and muscle activities of a reaching task are presented. The NMPC predictions of the hand trajectory to reach fixed and moving targets are in good agreement with the trajectories found by dynamic optimization and those from experiments. However, the hand velocity and muscle activations predicted by NMPC did not agree as well with experiments or with those found from dynamic optimization.The authors would like to thank the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Canada Research Chairs program for financial support of this research

3D modeling of the human upper limb including the biomechanics of joints, muscles and soft tissues

The challenge in virtual human modeling is to achieve the representation of the main human characteristics with as much realism as possible. Such achievements would allow the simulation and/or analysis of many virtual situations involving humans. Simulation is especially useful to derive information from the models so as to predict and/or reproduce the behaviors that would be observed in real situations. Computer methods in visualization and simulation have thus great potential for advances in medicine. The processes of strength generation and motion coordination are some of these phenomena for which there is still much remaining to be understood. The human shoulder is also probably the articulation of the human body which deserves more than any other to be named "terra incognita". Investigations towards the biomechanical modeling and simulation of the human upper limb are therefore presented in this study. It includes thorough investigation into the musculoskeletal anatomy and biomechanics of the human upper limb, into the biomechanical constitutive modeling of muscles and soft tissues, and into the nonlinear continuum mechanics and numerical methods, especially the incremental finite element methods, necessary for their simulation. On this basis, a 3-D biomechanical musculoskeletal human upper limb model has been designed using the Visible Human Data provided by the U.S. National Library of Medicine, and applied to the dynamic musculoskeletal simulation of the human upper limb. This research has been achieved in the context of the EU ESPRIT Project CHARM, whose objective has been to develop a comprehensive human animation resource database and a set of software tools allowing the modeling of the human complex musculoskeletal system and the simulation of its dynamics, including the finite element simulation of soft tissue deformation and muscular contraction. An investigation towards the application of this knowledge for the realistic modeling and animation of the upper limb in computer animation is then presented. The anatomical and biomechanical modeling of the scapulo-thoracic constraint and the shoulder joint sinus cones are proposed and applied to the realistic animation, using inverse kinematics, of a virtual skeleton and an anatomic musculoskeletal body model

Estimation of Phantom Arm Mechanics About Four Degrees of Freedom After Targeted Muscle Reinnervation

The intuitive control of bionic arms requires estimation of amputee's phantom arm movements from residual muscle bio-electric signals. The functional use of myoelectric arms relies on the ability of controlling large sets of degrees of freedom (>3 DOFs) spanning elbow, forearm, and wrist joints. This would assure optimal hand orientation in any environment. As part of this paper we recorded high-density electromyograms with >190 electrodes from the residual stump of a trans-humeral amputee who underwent targeted muscle reinnervation. We employed clustering to determine eight spatially separated sub-sets of channels sampling electromyograms associated to the actuation of four phantom arm DOFs. We created a large-scale musculoskeletal model of the phantom arm encompassing 33 musculo-tendon units. For the first time, this enabled the accurate electromyography-driven simulation of complex phantom joint rotations about elbow flexion-extension, forearm pronation-supination, wrist flexion-extension, and radial-ulnar deviation. These results support the potential for a new class of bionic limbs that are controlled as natural extensions of the body, an important step toward next-generation prosthetics that mimic human biological functionality and robustness