Force-Aware Interface via Electromyography for Natural VR/AR Interaction

While tremendous advances in visual and auditory realism have been made for virtual and augmented reality (VR/AR), introducing a plausible sense of physicality into the virtual world remains challenging. Closing the gap between real-world physicality and immersive virtual experience requires a closed interaction loop: applying user-exerted physical forces to the virtual environment and generating haptic sensations back to the users. However, existing VR/AR solutions either completely ignore the force inputs from the users or rely on obtrusive sensing devices that compromise user experience. By identifying users' muscle activation patterns while engaging in VR/AR, we design a learning-based neural interface for natural and intuitive force inputs. Specifically, we show that lightweight electromyography sensors, resting non-invasively on users' forearm skin, inform and establish a robust understanding of their complex hand activities. Fuelled by a neural-network-based model, our interface can decode finger-wise forces in real-time with 3.3% mean error, and generalize to new users with little calibration. Through an interactive psychophysical study, we show that human perception of virtual objects' physical properties, such as stiffness, can be significantly enhanced by our interface. We further demonstrate that our interface enables ubiquitous control via finger tapping. Ultimately, we envision our findings to push forward research towards more realistic physicality in future VR/AR.Comment: ACM Transactions on Graphics (SIGGRAPH Asia 2022

Neuromotor Control of the Hand During Smartphone Manipulation

The primary focus of this dissertation was to understand the motor control strategy used by our neuromuscular system for the multi-layered motor tasks involved during smartphone manipulation. To understand this control strategy, we recorded the kinematics and multi-muscle activation pattern of the right limb during smartphone manipulation, including grasping with/out tapping, movement conditions (MCOND), and arm heights. In the first study (chapter 2), we examined the neuromuscular control strategy of the upper limb during grasping with/out tapping executed with a smartphone by evaluating muscle-activation patterns of the upper limb during different movement conditions (MCOND). There was a change in muscle activity for MCOND and segments. We concluded that our neuromuscular system generates the motor strategy that would allow smartphone manipulation involving grasping and tapping while maintaining MCOND by generating continuous and distinct multi-muscle activation patterns in the upper limb muscles. In the second study (chapter 3), we examined the muscle activity of the upper limb when the smartphone was manipulated at two arm heights: shoulder and abdomen to understand the influence of the arm height on the neuromuscular control strategy of the upper limb. Some muscles showed a significant effect for ABD, while some muscle showed a significant effect for SHD. We concluded that the motor control strategy was influenced by the arm height as there were changes in the shoulder and elbow joint angles along with the muscular activity of the upper limb. Further, shoulder position helped in holding the head upright while abdomen reduced the moment arm and moment and ultimately, muscle loading compared to the shoulder. Overall, our neuromuscular system generates motor command by activating a multi-muscle activation pattern in the upper limb, which would be dependent upon the task demands such as grasping with/out tapping, MCOND, and arm heights. Similarly, our neuromuscular system does not appear to increase muscle activation when there is a combined effect of MCOND and arm heights. Instead, it utilizes a simple control strategy that would select an appropriate muscle and activate them based on the levels of MCOND and arm heights

Tekstiilielektrodit electromyografiamittauksissa

This thesis investigates the use of conductive textiles as the material of surface electromyography electrodes especially in the case of hand gesture recognition. Several conductive textile materials are used to design electrode setups integrated in sleeve to measure an 8-channel EMG signal around forearm. The electric properties of the electrodes are investigated and the EMG signal is used in a data classification experiment to assess the performance of the textile electrodes in gesture recognition compared to conventional medical grade electrodes. The design and manufacturing process is also described in order to underline all benefits and challenges associated with the materials and methods. This thesis shows that textile electrodes are a viable option for conventional wet gel electrodes in gesture recognition applications.Tämä diplomityö tutkii johtavien tekstiilien soveltuvuutta pintaelektrodien materiaaliksi erityisesti mitattaessa lihasaktiivisuussignaalia (EMG) kyynärvarren lihaksista käden eleiden tunnistamista varten. Työssä mitataan elektrodien sähköisiä ominaisuuksia ja verrataan niitä perinteisiin kliinisiin pintaelektrodeihin. Diplomityö esittää myös tekstiilien ja työtapojen vahvuuksia ja heikkouksia, jotka ilmenivät prototyyppien suunnittelu- ja valmistusprosessin aikana. Tulokset osoittavat, että tekstiilielektrodit soveltuvat erittäin hyvin käden eleiden tunnistamiseen pyrkiviin mittauksiin

Applications of the PowerGlove

The hand is important in many daily life activities. During aging, quality of fine motor control of hand and fingers is decreasing. Also motor symptoms of the hand are important to define for instance the neurological state of a Parkinson’s disease patient. Although objective and reliable measurement of hand and finger dynamics is of interest, current measurement systems are limited. This paper describes the application of the PowerGlove, a new measurement system based on miniature inertial and magnetic sensors, to study the finger interdependency in healthy elderly and objectively quantify hand motor symptoms in Parkinson’s disease. Results of pilot experiments in young healthy subjects are shown to evaluate the feasibility of the applications

An electromyographic assessment pilot study on the reliability of the forearm muscles during multi-planar maximum voluntary contraction grip and wrist articulation in young males

BACKGROUND: Electromyographic systems are widely used in scientific and clinical practice. The reproducibility and reliability of these measures are crucial when conducting scientific research and collecting experimental data.OBJECTIVE: To test the reliability of surface electromyography signals from both the Flexor Digitorum Superficialis (FDS) and Extensor Carpi Radialis Brevis (ECRB) muscles of both the left and right arms during an individual, static multi-planar maximum voluntary contraction handgrip task using the Myon 320 system (Myon AG, Switzerland).METHODS: Eight right-handed male participants performed two maximal handgrip tests in five separate wrist positions using both hands. Muscle activity was recorded from both forearms. Reliability was measured using the Standard Error of Measurement (SEM), Coefficient of Variation (CV)and Intra-class correlation coefficients. Wrist joint position correlations within and between theFDS and ECRB muscle activities were also analysed.RESULTS: Absolute reliability was shown across all positions for both hands with CV and SEM recorded at below 10%. The output measures indicate that the Myon 320 system (Myon AG, Switzerland) produces good to fair reliability when assessing forearm muscle activity. Correlations in the left FDS muscles were negative. Correlations between the left ECRB and left FDS muscles were variable but positive between the right ECRB and right FDS muscles.CONCLUSIONS: The data sets retrieved from all participants were reliably evaluated. Wrist position correlations within and between the FDS and ECRB muscles may have been influenced by hand dominance. The findings demonstrate that the methods and systems outlined in this study can be used reliably in future research

Evidence of a Double Pulse Muscle Activation Strategy in Drummers’ Trunk and Upper Limb Muscles During High-velocity Cymbal Crashes

Background: Playing the drum kit is a physically and cognitively demanding task, and skilled drummers share many such attributes with elite athletes. The ‘double pulse’ muscle activation (DPMA) pattern is a motor control strategy that has been observed in athletes of sports involving ballistic movements (e.g., baseball, golf, Mixed Martial Arts), and is believed to function to increase force transfer to the target. Objective: This study examined the muscle activation patterns of highly skilled drummers for evidence of a DPMA during high-velocity cymbal crashes. Methods: Five drummers were instrumented with electromyography electrodes on the right latissimus dorsi, triceps brachii, erector spinae, rectus abdominis, deltoideus posterior (DP), teres major, extensor carpi radialis, and flexor carpi ulnaris muscles. Six trials of data were collected, including a resting baseline, three maximum voluntary exertions (MVE) consisting of maximal effort cymbal crashes, a drumming pattern that included multiple crashes, and a ‘free-play’ trial. Results: The DPMA waveform was observed in all trials, but only those observed during the MVE trials were confirmed to coincide with the crashing movement via video analysis. The DP muscle – which functions to extend the shoulder joint to crash the stick on to the cymbal – exhibited confirmed DPMAs the most frequently. Conclusion: The extent to which drummers use the DPMA to produce high-velocity cymbal crashes within authentic playing conditions is inconclusive and needs further examination. Future study of the DPMA phenomenon in drummers would benefit from the addition of 3-dimensional motion capture to further understand the purpose of the muscle contractions of the DPMA

Limits to temporal synchronization in fundamental hand and finger actions

Coordinated movement is critical not only to sports technique and performance but to daily living and as such represents a fundamental area of research. Coordination requires being able to produce the right actions at the right time and has to incorporate perception, cognition, and forceful neuro-muscular interaction with the environment. Coordinated movements of the hands and fingers are some of the most complex activities undertaken where continuous learning and adaptation take place, but the temporal variability of the most basic movement components is still unknown. This thesis investigates the extent of temporal variability in the execution of four different simple hand and finger coordination tasks, with the purpose to find the various intrinsic temporal variability which limit the ability to coordinate the hands in space and time. Study one showed that in a synchronized bi-lateral two finger tapping test (<<1 cm movement to target) the best participant had a temporaltiming variability of 4.8 ms whereas the largest time variability could be as high as 24.8 ms. No obvious improvement was found after transfer practice, whereas the average time variability for asynchronized tapping decreased from 62.1 ms to 30.3 ms after instructed practice indicating a likely change in task grouping. Study two showed that in a unilateral thumb-index finger pinch and release test, the largest mean timing variability was 12 ms for pinching irrespective of performing the task in a slow alert manner or at a faster speed. However, the mean temporal variability for release was only 6.3 ms when the task was performed in a more alert manner and indicates that release is more accurately controlled temporally than grip. Study three suggested that in a unilateral sagittal plane throwing action of the lower arm and hand, that elbow and wrist coordination for dynamic index finger tip location was better with a radial-ulnar deviation, darts-type, throwing action than a wrist flexor-extensor type action, basketball free throw type action (the mean variability was 37.5 ms and 27.2 ms, respectively). Study four compared the variability in bi-lateral finger tapping between voluntary tapping and involuntary finger contraction tapping. Electrically stimulated neural contractions had significantly lower force onset variability than voluntary or direct magnetic stimulation of muscles (6 ms, 9.5 ms, and 10.3 ms for electrically stimulated, voluntary and Transcranial Magnetic Stimulation stimulated contraction). This work provides a comprehensive analysis of the temporal variability in various fundamental digital movement tasks that can aid with the understanding of basic human coordination in sporting, daily living and clinical areas

The Influence of Dopamine Replacement on Movement Impairments During Bimanual Coordination in Parkinson’s Disease (PD)

The purpose of the current thesis was to investigate the influence of dopamine replacement on performance during bimanual coordination in individuals with Parkinson’s disease (PD) There has been conflicting research on the cause of movement impairments such as coordination deficits, slowed switching and upper limb freezing that occur during coordinated movements It is unclear whether decreased function of the dopaminergic system after withdrawal from dopamine replacement is responsible for these deficits Healthy age-matched control participants were compared to PD participants in two experiments to determine the movement impairments that occurred during three-dimensional wrist flexion-extension bimanual coordination as a result of PD. In addition, individuals with PD were compared without (‘off’) and with (‘on’) dopamine replacement in both experiments to determine whether modulation of the dopaminergic system influenced coordinated movements. In Experiment 1, continuous bimanual coordination was performed in m-phase (simultaneous wrist flexion and extension) and anti-phase (flexion of one wrist while extending other wrist) with movements externally paced with increasing across seven cycle frequencies (0.75 to 2 Hz). Visual feedback was also manipulated in one of three sensory conditions no vision, normal vision or augmented vision. Visual feedback, phase and cycle frequency manipulation was performed to determine whether other deficits (e.g. sensory and/or attentional deficits) may influence coordinated movements Despite reduced amplitude of movements in both limbs of individuals with PD (PD ‘off’), coordination deficits were not observed in PD compared to healthy control participants. In addition, there was an increased occurrence of upper limb freezing (ULF) when cycle frequency demand was greater Dopamine replacement did increase the amplitude of movements in individuals with PD but did not influence coordination performance or the occurrence of ULF. In Experiment 2, coordinated movements were initiated in either m-phase or antiphase and participants were required to voluntarily switch to the other phase pattern when an auditory cue was presented Trials were performed at one of two cycle frequencies (1 or 2 Hz) and one of two sensory conditions (no vision or normal vision) to determine whether other deficits (e.g. sensory and/or attentional deficits) may influence coordinated movement. In addition, a separate block of trials were performed in anti-phase coordination with an auditory cue that did not require a switch Non-switching trials were included to investigate whether the presence of a distracting cue could evoke ULF comparable to when switching between movements was required PD ‘off’ participants demonstrated slower switching, more delayed responses and deficits in coordination performance when compared to healthy control participants. The increased demand of cycle frequency particularly when initiating anti-phase coordination, after voluntary switching and with the presence of the auditory cue without switching contributed to a large occurrence of ULF in individuals with PD. Dopamine replacement improved the ability to switch between phase patterns but had no overall influence on coordination performance or the occurrence of ULF. Overall, the results of the current thesis demonstrated that dopamine replacement can improve motor symptoms during coordinated movements (e g hypometna and bradykinesia) but does not contribute to coordination performance or ULF in individuals with PD. As a consequence, it was concluded that coordination deficits and ULF are not caused by the dysfunctional dopaminergic system but rather associated to secondary impairment caused by PD. The movement impairments caused by secondary dysfunction of PD were proposed to be associated with increased attentional demands and possible executive dysfunction related to fronto-stnatal pathways that cannot be modulated by dopamine replacement. Thus, treatment of complex movement impairments such as coordination deficits and ULF may benefit from rehabilitation or non-dopamine therapies that focus on the global dysfunction caused by PD

A musculoskeletal model of the human hand to improve human-device interaction

abstract: Multi-touch tablets and smart phones are now widely used in both workplace and consumer settings. Interacting with these devices requires hand and arm movements that are potentially complex and poorly understood. Experimental studies have revealed differences in performance that could potentially be associated with injury risk. However, underlying causes for performance differences are often difficult to identify. For example, many patterns of muscle activity can potentially result in similar behavioral output. Muscle activity is one factor contributing to forces in tissues that could contribute to injury. However, experimental measurements of muscle activity and force for humans are extremely challenging. Models of the musculoskeletal system can be used to make specific estimates of neuromuscular coordination and musculoskeletal forces. However, existing models cannot easily be used to describe complex, multi-finger gestures such as those used for multi-touch human computer interaction (HCI) tasks. We therefore seek to develop a dynamic musculoskeletal simulation capable of estimating internal musculoskeletal loading during multi-touch tasks involving multi digits of the hand, and use the simulation to better understand complex multi-touch and gestural movements, and potentially guide the design of technologies the reduce injury risk. To accomplish these, we focused on three specific tasks. First, we aimed at determining the optimal index finger muscle attachment points within the context of the established, validated OpenSim arm model using measured moment arm data taken from the literature. Second, we aimed at deriving moment arm values from experimentally-measured muscle attachments and using these values to determine muscle-tendon paths for both extrinsic and intrinsic muscles of middle, ring and little fingers. Finally, we aimed at exploring differences in hand muscle activation patterns during zooming and rotating tasks on the tablet computer in twelve subjects. Towards this end, our musculoskeletal hand model will help better address the neuromuscular coordination, safe gesture performance and internal loadings for multi-touch applications.Dissertation/ThesisDoctoral Dissertation Mechanical Engineering 201