A musculoskeletal model of the human hand to improve human-device interaction

abstract: Multi-touch tablets and smart phones are now widely used in both workplace and consumer settings. Interacting with these devices requires hand and arm movements that are potentially complex and poorly understood. Experimental studies have revealed differences in performance that could potentially be associated with injury risk. However, underlying causes for performance differences are often difficult to identify. For example, many patterns of muscle activity can potentially result in similar behavioral output. Muscle activity is one factor contributing to forces in tissues that could contribute to injury. However, experimental measurements of muscle activity and force for humans are extremely challenging. Models of the musculoskeletal system can be used to make specific estimates of neuromuscular coordination and musculoskeletal forces. However, existing models cannot easily be used to describe complex, multi-finger gestures such as those used for multi-touch human computer interaction (HCI) tasks. We therefore seek to develop a dynamic musculoskeletal simulation capable of estimating internal musculoskeletal loading during multi-touch tasks involving multi digits of the hand, and use the simulation to better understand complex multi-touch and gestural movements, and potentially guide the design of technologies the reduce injury risk. To accomplish these, we focused on three specific tasks. First, we aimed at determining the optimal index finger muscle attachment points within the context of the established, validated OpenSim arm model using measured moment arm data taken from the literature. Second, we aimed at deriving moment arm values from experimentally-measured muscle attachments and using these values to determine muscle-tendon paths for both extrinsic and intrinsic muscles of middle, ring and little fingers. Finally, we aimed at exploring differences in hand muscle activation patterns during zooming and rotating tasks on the tablet computer in twelve subjects. Towards this end, our musculoskeletal hand model will help better address the neuromuscular coordination, safe gesture performance and internal loadings for multi-touch applications.Dissertation/ThesisDoctoral Dissertation Mechanical Engineering 201

Etude comparative de l'analyse des mouvements lors d'une interaction tactile : adultes versus ainés

International audienceThis paper reports a comparative study on the analysis of the users’ movements and their consequences in users’ performances (time, error rates) during interaction of 15 older adults and 15 adults with touchscreen. The analysis of the movement of the users’ wrist during the execution of interaction gestures allow to understand the differences in performances between older and younger users.Cet article présente une étude comparative de l'analyse des mouvements de l'utilisateur et leurs conséquences sur la performance (temps, taux d'erreurs) durant l'interaction avec des écrans tactiles sur une population de 15 participants âgés versus 15 adultes. L'analyse du mouvement du poignet durant l'exécution des gestes d'interaction permet de comprendre les différences de performances entre utilisateurs âgés et plus jeunes

Text Entry Performance and Situation Awareness of a Joint Optical See-Through Head-Mounted Display and Smartphone System

Optical see-through head-mounted displays (OST HMDs) are a popular output medium for mobile Augmented Reality (AR) applications. To date, they lack efficient text entry techniques. Smartphones are a major text entry medium in mobile contexts but attentional demands can contribute to accidents while typing on the go. Mobile multi-display ecologies, such as combined OST HMD-smartphone systems, promise performance and situation awareness benefits over single-device use. We study the joint performance of text entry on mobile phones with text output on optical see-through head-mounted displays. A series of five experiments with a total of 86 participants indicate that, as of today, the challenges in such a joint interactive system outweigh the potential benefits.Comment: To appear in IEEE Transactions on Visualization and Computer Graphics On page(s): 1-17 Print ISSN: 1077-2626 Online ISSN: 1077-262

Neuromotor Control of the Hand During Smartphone Manipulation

The primary focus of this dissertation was to understand the motor control strategy used by our neuromuscular system for the multi-layered motor tasks involved during smartphone manipulation. To understand this control strategy, we recorded the kinematics and multi-muscle activation pattern of the right limb during smartphone manipulation, including grasping with/out tapping, movement conditions (MCOND), and arm heights. In the first study (chapter 2), we examined the neuromuscular control strategy of the upper limb during grasping with/out tapping executed with a smartphone by evaluating muscle-activation patterns of the upper limb during different movement conditions (MCOND). There was a change in muscle activity for MCOND and segments. We concluded that our neuromuscular system generates the motor strategy that would allow smartphone manipulation involving grasping and tapping while maintaining MCOND by generating continuous and distinct multi-muscle activation patterns in the upper limb muscles. In the second study (chapter 3), we examined the muscle activity of the upper limb when the smartphone was manipulated at two arm heights: shoulder and abdomen to understand the influence of the arm height on the neuromuscular control strategy of the upper limb. Some muscles showed a significant effect for ABD, while some muscle showed a significant effect for SHD. We concluded that the motor control strategy was influenced by the arm height as there were changes in the shoulder and elbow joint angles along with the muscular activity of the upper limb. Further, shoulder position helped in holding the head upright while abdomen reduced the moment arm and moment and ultimately, muscle loading compared to the shoulder. Overall, our neuromuscular system generates motor command by activating a multi-muscle activation pattern in the upper limb, which would be dependent upon the task demands such as grasping with/out tapping, MCOND, and arm heights. Similarly, our neuromuscular system does not appear to increase muscle activation when there is a combined effect of MCOND and arm heights. Instead, it utilizes a simple control strategy that would select an appropriate muscle and activate them based on the levels of MCOND and arm heights

Study of the interaction of older adults with touchscreen

Utiliser une tablette ou un smartphone est désormais courant. Cependant, les effets de l'âge sur les capacités motrices nécessaires pour l'exécution des gestes d'interaction tactile n'ont pas été suffisamment pris en compte lors de la conception et de l'évaluation des systèmes interactifs, une des raisons qui a empêché l'inclusion numérique de ce groupe d'utilisateurs. L'objectif de cette thèse est d'étudier l'interaction des personnes âgées avec les écrans tactiles afin d'identifier des problèmes d'utilisabilité sur des supports variés (smartphone et tablette, doigt et stylet). Pour cette étude, nous avons conçu un système interactif constitué de jeux de type puzzle numérique tactiles, où le geste d'interaction drag-and-drop (glisser-déposer) est employé pour positionner les cibles. Dans ce contexte, une attention particulière a été portée à l'analyse des mouvements de l'utilisateur. L'analyse des postures du poignet durant l'interaction a permis d'élucider la relation entre les caractéristiques des mouvements des personnes âgées avec leurs performances, à savoir, des temps plus longs et une augmentation du nombre d'erreurs par rapport aux utilisateurs adultes plus jeunes. Prendre en compte la variabilité des capacités motrices des utilisateurs lors des phases de conception et évaluation des systèmes interactifs est nécessaire pour comprendre leurs difficultés et améliorer l'ergonomie et utilisabilité de l'interaction tactile.Tablets and smartphones have become mainstream technologies. However, the aging effects on the motor skills implied on tactile interaction haven't been enough considered during the design and evaluation of tactile interactive systems, what prevent this group of older adult users to be digitally included successfully. This thesis aims to study the interaction of older adults with touchscreens in order to identify usability issues on different devices and input modalities (smartphone and tablet, finger and stylus). To this study, we designed an interactive system consisted of tactile puzzle games and using drag-and-drop interaction for positioning the puzzle pieces into their corresponding targets. In this framework, a special attention was given to the analysis of the movements of the user. The analysis of the postures of the users' wrists during interaction allowed to elucidate the relationship between the characteristics of the movements of older adults and their performances, particularly concerning the longer times needed for executing the gestures of interaction as well as the increased error rates of this group of users when compared to younger adults. Taking into account the variability of users' motor skills during the design and evaluation of interactive systems is necessary to better understand their difficulties as well as to improve the ergonomics and the usability levels of tactile interaction

Moving Away From the Traditional Desktop Computer Workstations: Identifying Opportunities to Improve Upper Extremity Biomechanics

Statement of Problem: Office computer workers have elevated risks of adverse health outcome such as musculoskeletal disorders (MSDs) associated with computer work. Although they now have many alternatives, these modern computer workstations and associated technologies require new guidelines and recommendations for proper practice. We see this as an opportunity to improve current and future computer workstation designs through an ergonomics approach by improving users’ upper extremity biomechanics while interacting with these modern technologies. Method: The dissertation first utilized a psychophysical protocol to compare users’ self-selected set ups for sitting and standing computer workstations. Users’ biomechanics and perceived comfort across different computer tasks for the two workstations are then compared. Subsequently, a hand mapping technique was developed to evaluate effects of computer pointing devices on users’ hand posture and associated forearm muscle effort using 3-D motion analysis and surface electromyography. To improve mobile device ergonomics, we investigated tablet users’ biomechanical load, comfort level and performance while performing swipe actions at different tablet locations. Results: Different selected computer workstation set ups were found for sitting and standing. Compared to sitting, users while standing kept workstation components closer to their sternum and adopted a more neutral shoulder posture while working. However, users had greater wrist extension and started reporting more low back discomfort after 45 minutes. While investigating different computer pointing devices, we found device affordance associated with significantly different hand posture and forearm muscle load. Devices that required less holding and were centrally placed associated with more neutral shoulder and hand postures, with significantly less forearm muscle load. For tablet interface, swipe locations closer to the palm had significantly smaller forearm muscle load and more neutral posture across wrist and thumb joints. Conclusion: Through empirical results described in the dissertation, we demonstrated how users’ upper extremity biomechanics can provide insights into the complex interactions between users and modern computer workstations, both as a whole and with specific components. For technology innovation, ergonomics concepts and methodologies can be used to design future generation technologies that fit users’ physical capabilities to reduce MSDs risk while promoting performance

WearPut : Designing Dexterous Wearable Input based on the Characteristics of Human Finger Motions

Department of Biomedical Engineering (Human Factors Engineering)Powerful microchips for computing and networking allow a wide range of wearable devices to be miniaturized with high fidelity and availability. In particular, the commercially successful smartwatches placed on the wrist drive market growth by sharing the role of smartphones and health management. The emerging Head Mounted Displays (HMDs) for Augmented Reality (AR) and Virtual Reality (VR) also impact various application areas in video games, education, simulation, and productivity tools. However, these powerful wearables have challenges in interaction with the inevitably limited space for input and output due to the specialized form factors for fitting the body parts. To complement the constrained interaction experience, many wearable devices still rely on other large form factor devices (e.g., smartphones or hand-held controllers). Despite their usefulness, the additional devices for interaction can constrain the viability of wearable devices in many usage scenarios by tethering users' hands to the physical devices. This thesis argues that developing novel Human-Computer interaction techniques for the specialized wearable form factors is vital for wearables to be reliable standalone products. This thesis seeks to address the issue of constrained interaction experience with novel interaction techniques by exploring finger motions during input for the specialized form factors of wearable devices. The several characteristics of the finger input motions are promising to enable increases in the expressiveness of input on the physically limited input space of wearable devices. First, the input techniques with fingers are prevalent on many large form factor devices (e.g., touchscreen or physical keyboard) due to fast and accurate performance and high familiarity. Second, many commercial wearable products provide built-in sensors (e.g., touchscreen or hand tracking system) to detect finger motions. This enables the implementation of novel interaction systems without any additional sensors or devices. Third, the specialized form factors of wearable devices can create unique input contexts while the fingers approach their locations, shapes, and components. Finally, the dexterity of fingers with a distinctive appearance, high degrees of freedom, and high sensitivity of joint angle perception have the potential to widen the range of input available with various movement features on the surface and in the air. Accordingly, the general claim of this thesis is that understanding how users move their fingers during input will enable increases in the expressiveness of the interaction techniques we can create for resource-limited wearable devices. This thesis demonstrates the general claim by providing evidence in various wearable scenarios with smartwatches and HMDs. First, this thesis explored the comfort range of static and dynamic touch input with angles on the touchscreen of smartwatches. The results showed the specific comfort ranges on variations in fingers, finger regions, and poses due to the unique input context that the touching hand approaches a small and fixed touchscreen with a limited range of angles. Then, finger region-aware systems that recognize the flat and side of the finger were constructed based on the contact areas on the touchscreen to enhance the expressiveness of angle-based touch input. In the second scenario, this thesis revealed distinctive touch profiles of different fingers caused by the unique input context for the touchscreen of smartwatches. The results led to the implementation of finger identification systems for distinguishing two or three fingers. Two virtual keyboards with 12 and 16 keys showed the feasibility of touch-based finger identification that enables increases in the expressiveness of touch input techniques. In addition, this thesis supports the general claim with a range of wearable scenarios by exploring the finger input motions in the air. In the third scenario, this thesis investigated the motions of in-air finger stroking during unconstrained in-air typing for HMDs. The results of the observation study revealed details of in-air finger motions during fast sequential input, such as strategies, kinematics, correlated movements, inter-fingerstroke relationship, and individual in-air keys. The in-depth analysis led to a practical guideline for developing robust in-air typing systems with finger stroking. Lastly, this thesis examined the viable locations of in-air thumb touch input to the virtual targets above the palm. It was confirmed that fast and accurate sequential thumb touch can be achieved at a total of 8 key locations with the built-in hand tracking system in a commercial HMD. Final typing studies with a novel in-air thumb typing system verified increases in the expressiveness of virtual target selection on HMDs. This thesis argues that the objective and subjective results and novel interaction techniques in various wearable scenarios support the general claim that understanding how users move their fingers during input will enable increases in the expressiveness of the interaction techniques we can create for resource-limited wearable devices. Finally, this thesis concludes with thesis contributions, design considerations, and the scope of future research works, for future researchers and developers to implement robust finger-based interaction systems on various types of wearable devices.ope

Effects of a Multitouch Keyboard on Wrist Posture, Typing Performance and Comfort

Alan Hedge Geri GayThe design of computer keyboards is rapidly evolving as portable computing becomes increasingly ubiquitous due to wireless networking and the increased popularity of personal digital assistants and notebook computers. However, there is a balance between mobility and productivity, in terms of text-entry accuracy and speed, which needs to be maintained as computer keyboards become smaller and slimmer through the introduction of ultra low-profile designs. In addition, the ergonomic benefits, in terms of the reduction of awkward wrist postures and user comfort, of ultra-low profile designs are unclear. This study tests a new prototype ultra-low profile MultiTouch keyless keyboard (MTK) that uses a MultiTouch surface to create an extremely thin typing environment that requires no force to register a keystroke and allows mousing and gestural input on the same surface. In this study, the MTK was tested against a conventional keyboard (CK) for typing speed, accuracy, wrist postures and user comfort. It was hypothesized that the lack of key travel would increase speed and accuracy, while the ultra-thin design would reduce the amount of wrist extension, which could decrease the risk of a wrist injury or other hand and wrist musculoskeletal disorder. Finally, it was hypothesized that there would be a significant short-term learning effect on typing speed and accuracy for the MTK. A laboratory experiment was conducted with 6 males and 6 females typing using two QWERTY keyboard designs: a CK and a MTK. Subjects visited the lab for 1.5 hours for 2 non-consecutive days in the same week, for a total of 3 hours. Each visit consisted of eight randomly assigned 7.5-minute typing tasks of text passages of similar difficulty and identical length. Quantitative measures of typing speed and accuracy were collected using Typing Quick and Easy 13.0 and qualitative measures of user preference and comfort were gathered by self-report questionnaires. A wrist glove electrogoniometer system was used to record right-hand wrist positioning data, which was analyzed to assess the risk of injury. The two keyboards were evaluated in a repeated measures within-subjects factorial design. Subjects, typed slower (F1,11 = 41.86, p=0.000) and less accurately (F1,11 = 23.55, p=0.001) on the MTK during the typing tasks. Subjects preferred the CK and reported a higher level of ease (F1,11 = 49.732, p=0.00) and enjoyment (F1,11 = 51.129, p=0.00) during its use. Mean wrist extension was lower for the MTK (F1,11= 10.205, p=0.000) while radial and ulnar deviation did not differ significantly between the two keyboards. The MTK had a lower percentage of highest-risk wrist extension (F1,11= 6.437, p=0.028), and conversely, a higher percentage of neutral wrist posture (F1,11= 12.947, p=0.004). A significant positive linear trend was observed across the within-subjects scores for speed (F1,11= 9.308, p=0.011) and accuracy (F1,11= 11.903, p=0.005) across tasks in the MTK condition. Limitations to this study include practice effects, due to the naive subjects' lack of training on the MTK and the limited duration of exposure to this novel keyboard. Fatigue effects may have also been a factor, even though the experimental conditions were spread out over two non-consecutive days in the same week. Future research directions include additional testing of the unique mousing and gestural capabilities of the MTK. Other research suggests that practice and extended exposure to the MTK may raise performance to comparable levels associated with CK devices.College of Human Ecology, Cornell Universit