Neuromotor Control of the Hand During Smartphone Manipulation

The primary focus of this dissertation was to understand the motor control strategy used by our neuromuscular system for the multi-layered motor tasks involved during smartphone manipulation. To understand this control strategy, we recorded the kinematics and multi-muscle activation pattern of the right limb during smartphone manipulation, including grasping with/out tapping, movement conditions (MCOND), and arm heights. In the first study (chapter 2), we examined the neuromuscular control strategy of the upper limb during grasping with/out tapping executed with a smartphone by evaluating muscle-activation patterns of the upper limb during different movement conditions (MCOND). There was a change in muscle activity for MCOND and segments. We concluded that our neuromuscular system generates the motor strategy that would allow smartphone manipulation involving grasping and tapping while maintaining MCOND by generating continuous and distinct multi-muscle activation patterns in the upper limb muscles. In the second study (chapter 3), we examined the muscle activity of the upper limb when the smartphone was manipulated at two arm heights: shoulder and abdomen to understand the influence of the arm height on the neuromuscular control strategy of the upper limb. Some muscles showed a significant effect for ABD, while some muscle showed a significant effect for SHD. We concluded that the motor control strategy was influenced by the arm height as there were changes in the shoulder and elbow joint angles along with the muscular activity of the upper limb. Further, shoulder position helped in holding the head upright while abdomen reduced the moment arm and moment and ultimately, muscle loading compared to the shoulder. Overall, our neuromuscular system generates motor command by activating a multi-muscle activation pattern in the upper limb, which would be dependent upon the task demands such as grasping with/out tapping, MCOND, and arm heights. Similarly, our neuromuscular system does not appear to increase muscle activation when there is a combined effect of MCOND and arm heights. Instead, it utilizes a simple control strategy that would select an appropriate muscle and activate them based on the levels of MCOND and arm heights

Different strokes for different folks? Revealing the physical characteristics of smartphone users from their swipe gestures

AbstractAnthropometrics show that the lengths of many human body segments follow a common proportional relationship. To know the length of one body segment – such as a thumb – potentially provides a predictive route to other physical characteristics, such as overall standing height. In this study, we examined whether it is feasible that the length of a person׳s thumb could be revealed from the way in which they complete swipe gestures on a touchscreen-based smartphone.From a corpus of approx. 19,000 swipe gestures captured from 178 volunteers, we found that people with longer thumbs complete swipe gestures with shorter completion times, higher speeds and with higher accelerations than people with shorter thumbs. These differences were also observed to exist between our male and female volunteers, along with additional differences in the amount of touch pressure applied to the screen.Results are discussed in terms of linking behavioural and physical biometrics

Effects of Grip Curvature and Hand Anthropometry for the Manual Operation of Handheld Touchscreen Device

학위논문 (석사)-- 서울대학교 대학원 : 산업공학과, 2014. 2. 윤명환.To design handheld devices, physical comfort is one of the most crucial requirements. Recently, curved handheld touchscreens were released for enhancing comfort, but the effect has not been proved yet. There are two ergonomic factors, anthropometric and physiological factors, respecting comfort. This means the design should consider variation of hands in size and shape as well as muscle utilization. Through statistical analysis, it has been showed that the Korean population has large variability in both size and shape. Also, it has been observed that 1/3 of user population of smartphone operate the device unimanully by a previous research. This study aimed to verify the effect of anthropometric factors of hands and curvature on comfort when using handheld touchscreen devices. Comfort level was measured employing both the subjective rating and EMG methods. Three mock-ups of handheld touchscreen device with different curvatures were utilized. One was flat device and the others had curvatures of 400R and 100R. An experiment was conducted on tapping, typing and dragging tasks. The results indicated that curvature of the handheld touchscreen devices did not affect muscle activities, but subjective comfort level. Moreover, size and shape of hand were found to affect muscle activities and comfort level when using the handheld touchscreen devices. Target location and moving direction of thumb were also factors that significantly affected muscle activities. Overall, this study suggests that user interface design may be more important than curvature of handheld touchscreen determining comfort of touch screen use.TABLE OF CONTENTS I LIST OF FIGURES IV LIST OF TABLES VI CHAPTER 1. INTRODUCTION 1 1.1 Research Background 1 1.2 Objective and Scope of the Study 3 1.3 Definition and Terminology 4 CHAPTER 2. LITERATURE REVIEW 5 2.1 Comfort 5 2.1.1 The role of comfort in usability 5 2.1.2 Comfort and discomfort - Definition and dimensions 7 2.1.3 Measurement of comfort and discomfort 10 2.1.4 Comfort and ergonimics 11 2.2 Hand 14 2.2.1 Anthropometry of the hand 14 2.2.2 Hand and comfort 16 2.3 Researches Done on Smartphone 18 2.4 Limits of Previous Studies 19 CHAPTER 3. RESEARCH METHODOLOGY 21 3.1 Hypothesis 21 3.2 Statistical Analysis on Hand 22 3.3 Apparatus 23 3.4 Subjects 25 3.5 Experimental Design 25 3.5.1 Tasks 25 3.5.2 Measurements 29 3.5.3 Procedure 33 3.6 Data Analysis 35 CHAPTER 4. RESULTS 36 4.1 Group classification by the size and the shape of hand 36 4.1.1 Group classification by actual size of hand 36 4.1.2 Group classification by shape of hand 38 4.2 General statistical results of muscle activity and comfort 40 4.3 The Effects of Curvature on Muscle Activity and Comfort 42 4.3.1 The effect of curvature on muscle activity 43 4.3.2 The effect of curvature on comfort level 44 4.4 The Effects of Size of The Hand on Muscle Activity and Comfort 46 4.4.1 The effect of the hand size on muscle activity 46 4.4.2 The effect of the hand size on comfort 48 4.4.3 The compound effect of curvature and the hand size on comfort 48 4.5 The effect of the hand shape on muscle activity and comfort 49 4.5.1 The effect of the hand shape on muscle activity 49 4.5.2 The effect of the hand shape on comfort 51 4.5.3 The compound effect of curvature and the hand shape on comfort 52 4.6 Muscle activities of levels for each task 53 4.6.1 Tapping task 53 4.6.2 Dragging task 53 CHAPTER 5. CONCLUSION AND DISCUSSION 57 REFERENCE 63 국문 초록 77 APPENDICES 78 Appendix A. Hand Dimension Description 78 Appendix B. EMG System Specification 84 Appendix C. Experimental Sheet 85 Appendix D. Descriptive Statistics for Hand Dimensions of Participants 86 Appendix E. The Result of ANOVA Test 87 Appendix F. Curve Fitting Model Summary and Parameter Estimates 97Maste

Get a grip: Analysis of muscle activity and perceived comfort in using stylus grips

The design of handwriting instruments has been based primarily on touch, feel, aesthetics, and muscle exertion. Previous studies make it clear that different pen characteristics have to be considered along with hand-instrument interaction in the design of writing instruments. This should include pens designed for touch screens and computer based writing surfaces. Hence, this study focuses primarily on evaluating grip style’s impact on user comfort and muscle activity associated with handgrip while using a stylus-pen. Surface EMG measures were taken approximate to the adductor pollicis, flexor digitorum, and extensor indicis of eight participants while they performed writing, drawing, and point-and-click tasks on a tablet using a standard stylus and two grip options. Participants were also timed and surveyed on comfort level for each trial. Results of this study indicate that participants overall felt using a grip was more comfortable than using a stylus alone. The claw grip was the preferred choice for writing and drawing, and the crossover grip was preferred for pointing and clicking. There was reduction in muscle activity of the extensor indicis using the claw or crossover grip for the drawing and point and click tasks. The reduced muscle activity and the perceived comfort shows the claw grip to be a viable option for improving comfort for writing or drawing on a touchscreen device

Computational Modeling and Experimental Research on Touchscreen Gestures, Audio/Speech Interaction, and Driving

As humans are exposed to rapidly evolving complex systems, there are growing needs for humans and systems to use multiple communication modalities such as auditory, vocal (or speech), gesture, or visual channels; thus, it is important to evaluate multimodal human-machine interactions in multitasking conditions so as to improve human performance and safety. However, traditional methods of evaluating human performance and safety rely on experimental settings using human subjects which require costly and time-consuming efforts to conduct. To minimize the limitations from the use of traditional usability tests, digital human models are often developed and used, and they also help us better understand underlying human mental processes to effectively improve safety and avoid mental overload. In this regard, I have combined computational cognitive modeling and experimental methods to study mental processes and identify differences in human performance/workload in various conditions, through this dissertation research. The computational cognitive models were implemented by extending the Queuing Network-Model Human Processor (QN-MHP) Architecture that enables simulation of human multi-task behaviors and multimodal interactions in human-machine systems. Three experiments were conducted to investigate human behaviors in multimodal and multitasking scenarios, combining the following three specific research aims that are to understand: (1) how humans use their finger movements to input information on touchscreen devices (i.e., touchscreen gestures), (2) how humans use auditory/vocal signals to interact with the machines (i.e., audio/speech interaction), and (3) how humans drive vehicles (i.e., driving controls). Future research applications of computational modeling and experimental research are also discussed. Scientifically, the results of this dissertation research make significant contributions to our better understanding of the nature of touchscreen gestures, audio/speech interaction, and driving controls in human-machine systems and whether they benefit or jeopardize human performance and safety in the multimodal and concurrent task environments. Moreover, in contrast to the previous models for multitasking scenarios mainly focusing on the visual processes, this study develops quantitative models of the combined effects of auditory, tactile, and visual factors on multitasking performance. From the practical impact perspective, the modeling work conducted in this research may help multimodal interface designers minimize the limitations of traditional usability tests and make quick design comparisons, less constrained by other time-consuming factors, such as developing prototypes and running human subjects. Furthermore, the research conducted in this dissertation may help identify which elements in the multimodal and multitasking scenarios increase workload and completion time, which can be used to reduce the number of accidents and injuries caused by distraction.PHDIndustrial & Operations EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/143903/1/heejinj_1.pd

Target size guidelines for interactive displays on the flight deck

The avionics industry is seeking to understand the challenges and benefits of touchscreens on flight decks. This paper presents an investigation of interactive displays on the flight deck focusing on the impact of target size, placement and vibration on performance. A study was undertaken with search and rescue (SAR) crew members in an operational setting in helicopters. Results are essential to understand how to design effective touchscreen interfaces for the flight deck. Results show that device placement, vibration and target size have significant effects on targeting accuracy. However, increasing target size eliminates the negative effects of placement and vibration in most cases. The findings suggest that 15 mm targets are sufficiently large for non-safety critical Electronic Flight Bag (EFB) applications. For interaction with fixed displays where pilots have to extend their arms, and for safety critical tasks it is recommended to use interactive elements of about 20 mm size

Target size guidelines for interactive displays on the flight deck

The avionics industry is seeking to understand the challenges and benefits of touchscreens on flight decks. This paper presents an investigation of interactive displays on the flight deck focusing on the impact of target size, placement and vibration on performance. A study was undertaken with search and rescue (SAR) crew members in an operational setting in helicopters. Results are essential to understand how to design effective touchscreen interfaces for the flight deck. Results show that device placement, vibration and target size have significant effects on targeting accuracy. However, increasing target size eliminates the negative effects of placement and vibration in most cases. The findings suggest that 15 mm targets are sufficiently large for non-safety critical Electronic Flight Bag (EFB) applications. For interaction with fixed displays where pilots have to extend their arms, and for safety critical tasks it is recommended to use interactive elements of about 20 mm size

Ergonomic Design Guidelines for Non-flexible, Foldable, and Rollable Mobile Devices

Department of Human Factors EngineeringSmartphones are mobile devices used daily by people of almost all ages. Therefore, improving these devices from an ergonomic perspective can benefit many people. Similarly, future mobile devices with new displays must be designed from an ergonomic perspective. The purpose of this thesis was to develop ergonomic design guidelines for current non-flexible smartphones as well as future flexible display devices, considering perceived grip comfort, user preference, attractive design, and/or muscle activity. This thesis consists of six studies. The first two studies are on current smartphones with non-flexible displays, and the remaining four studies are on future mobile devices with flexible (foldable and rollable) displays. Study 1 examined the effects of task (neutral, comfortable, maximum, vertical, and horizontal strokes), phone width (60 and 90 mm), and hand length (small, medium, and large) on grasp, index finger reach zone, discomfort, and muscle activation for smartphone rear interaction. Ninety individuals participated in this study. The grasp was classified into two groups for rear interaction usage. The recommended zone for rear interaction was 8.8???10.1 cm from the bottom and 0.3???2.0 cm to the right of the vertical center line. Horizontal (vertical) strokes deviated from the horizontal axis in the range ???10.8?? to ???13.5?? (81.6 to 88.4??). Maximum strokes appeared to be excessive as these caused 43.8% greater discomfort than neutral strokes did. A 90-mm width also appeared to be excessive as it resulted in a 12.3% increase in discomfort relative to the 60-mm width. The small-hand group reported 11.9???18.2% higher discomfort ratings, and the percentage of maximum voluntary exertion of the flexor digitorum superficialis was 6.4% higher. Study 2 aimed to identify ergonomic forms of non-flexible smartphone by investigating the effects of hand length, four major smartphone dimensions (height, width, thickness, and edge roundness), and mass on one-handed grip comfort and design attractiveness. Seventy-two individuals participated. Study 2 was conducted in three stages. Stage 1 determined the ranges of the four smartphone dimensions suitable for grip comfort. Stage 2 investigated the effects of width and thickness (determined to have the greatest influence) on grip comfort and design attractiveness. Stage 3 investigated the effect of mass on grip comfort and design attractiveness. Phone width was found to significantly influence grip comfort and design attractiveness, and the dimensions of 140??65(or 70)??8??2.5 mm (height??width??thickness??edge roundness) provided higher one-handed grip comfort and design attractiveness. The selected dimensions were fit with a mass of 122 g and compared within a range of 106???137 g. Study 3 examined ergonomic forms for mobile foldable display devices in terms of folding/unfolding comfort and preference. Sixty individuals participated. Study 3 was conducted in two stages. In stage 1, suitable screen sizes for five tasks (messaging, calling, texting, web searching, and gaming) were determined. In stage 2, the most preferred folding methods among 14 different bi-folding and tri-folding methods were determined. The device dimension of 140H??60W was preferred for calling, whereas 140H??130W was preferred for web searches and gaming. The most preferred tri-fold concept (140H??198W) utilized Z-shaped screen folding. A trade-off was observed between screen protection and easy screen access. Study 4 examined the effects of gripping condition, device thickness, and hand length on bimanual grip comfort when using mobile devices with a rollable display. Thirty individuals evaluated three rollable display device prototypes (2, 6, and 10 mm right-side thickness) using three distinct gripping conditions (unrestricted, restricted, and pulp pinch grips). Rollable display devices should have at least 20 mm side bezel width and 10 mm thickness to ensure high grip comfort for bilateral screen pulling. Grip comfort increased as the device thickness was increased. Relative to device thickness, gripping condition greatly influenced bimanual grip comfort. Study 5 examined the effects of device height (70, 140, and 210 mm), task (web searching, video watching, and E-mail composing), and hand length (small, medium, and large hand groups) on various UX elements associated with using rollable display devices. Thirty individuals participated. Six UX elements (preferred screen width, preferred screen aspect ratio, user satisfaction, grip comfort, portability, design attractiveness, and gripping method) were assessed. Among device height, task, and hand length, device height was the most influential on the UX elements. The 95th percentile preferred screen width of three prototypes (device heights of 210, 140, and 70 mm) was 311.1, 206.2, and 100.0 mm, respectively. The larger the hand length, the wider the preferred screen width. A device (screen) height of 140 (120) mm with a 206.2 mm wide screen improved the overall user experience. Study 6 examined the effects of gender (15 males and 15 females), device thickness (2T, 6T, and 10T), and pulling duration (0.5s, 1.0s, and 1.5s) on preferred and acceptable pulling forces, muscle activities, and perceived comfort of the upper limbs associated with unrolling rollable displays. Thirty individuals evaluated three rollable display prototypes by laterally pulling each prototype for three different durations. Preferred and acceptable pulling forces of the upper limbs were measured, and the corresponding muscle activation and perceived comfort were obtained. Pulling duration largely accounted for %MVC of posterior deltoid (PD), flexor carpi radialis (FCR), and extensor carpi radialis (ECR), whereas gender largely accounted for perceived comfort. In consideration of perceived comfort, the device thickness was recommended to be 2 to 6T for both genders. %MVC of PD, FCR, and ECR of the female group was 1.4-2.4 times as high as that of the male group. The perceived comfort of the male group was 1.1-1.3 times higher than that of the female group. Overall, 6T was the best thickness. Users preferred a shorter pulling duration with a higher level of muscle activation than a longer pulling duration with a lower level of muscle activation to unroll the rollable screen. This work suggested ergonomic design guidelines for non-flexible smartphones and flexible mobile devices. Through these guidelines, basic dimensions and concepts for current and future mobile devices can be specified. In future studies, it is necessary to consider the intangible UX for future mobile devices by investigating the GUI based on the PUI proposed in this study.clos

Mixed method approach in designing flight decks with touch screens: a framework

Touch screen technology’s first public appearance was in the early 2000s. Touch screens became a part of the daily life with the invention of smartphones and tablets. Now, this technology has the potential to be the next big change in flight deck design. To date, mobile devices are deployed by several air carriers to perform a host of non-safety critical pre-flight and in-flight tasks. Due to high safety requirements requested by authorities, new technologies cannot be adopted as fast as in other settings. Flight deck evolution, which is briefly presented in this paper, is reflecting this natural time delay. Avionics manufacturers are exploring and working on future concepts with touch screen displays. This paper investigates the potential benefits and challenges of touch screen technology on flight decks by means of a variety of qualitative and quantitative research methods (mixed method approach). On the basis of this, a framework was constructed showing the relation between various aspects that could impact the usability of touch screens on the flight deck. This paper concludes with a preliminary questionnaire that can help avionic designers to evaluate whether a touch screen is an appropriate user interface for their system

Prevalence and Risk Factors for Neck and Shoulder Musculoskeletal Symptoms in Users of Touch-Screen Tablet Computers

Background and Purpose: Neck and upper extremity symptoms are common health complaints in the general population and cause a substantial financial burden to the health care system. Neck pain is known to be a multi factorial disorder, with head and spinal posture being one of the most important factors. Currently there is little evidence showing a relationship between touch-screen tablet computer usage and neck/upper extremity pathologies. However, it is fair to postulate that the incidence of neck/upper extremity pathologies is higher within touch-screen tablet computer users, considering how users maintain static postures while using the device. The primary purposes of the current study were to 1) investigate the prevalence of neck and shoulder symptoms and potential risk factors, 2) identify risk factors associated with symptoms during device use, especially in sitting postures. The secondary purpose was to investigate gender differences in device usage behavior, symptoms, and postural factors. Subjects: A cross-sectional survey study was conducted in a population of university students, staff, faculty, and alumni (N = 412). Methods and Results: Prevalence of symptoms during device use was 67.9 %. Most symptoms were reported in the neck (84.6%) and upper back/shoulder areas (65.4%). With sitting postures, significant symptoms during tablet use included sitting without back support (p= 0.016) and sitting with device in the lap (p= 0.002) based on Chi-square analyses. A multiple logistic regression analysis further demonstrated that female gender (p=0.004) and sitting in a chair without back support (p=0.006) are the predictors of experiencing symptoms during the use of tablets. For other general postures, the significant factors are lying on the side (p= 0.002) and lying on the back (p= 0.016) during tablet use. Furthermore, our results showed that 70.1% of the female respondents reported to have musculoskeletal symptoms during device usage, in comparison to 29.9% of the male respondents (p= 0.019). Additionally, women (75.7%) demonstrated significantly more symptoms at the upper back and shoulder regions than men (24.3%) (p = 0.014)