인간 기계 상호작용을 위한 강건하고 정확한 손동작 추적 기술 연구

학위논문(박사) -- 서울대학교대학원 : 공과대학 기계항공공학부, 2021.8. 이동준.Hand-based interface is promising for realizing intuitive, natural and accurate human machine interaction (HMI), as the human hand is main source of dexterity in our daily activities. For this, the thesis begins with the human perception study on the detection threshold of visuo-proprioceptive conflict (i.e., allowable tracking error) with or without cutantoues haptic feedback, and suggests tracking error specification for realistic and fluidic hand-based HMI. The thesis then proceeds to propose a novel wearable hand tracking module, which, to be compatible with the cutaneous haptic devices spewing magnetic noise, opportunistically employ heterogeneous sensors (IMU/compass module and soft sensor) reflecting the anatomical properties of human hand, which is suitable for specific application (i.e., finger-based interaction with finger-tip haptic devices). This hand tracking module however loses its tracking when interacting with, or being nearby, electrical machines or ferromagnetic materials. For this, the thesis presents its main contribution, a novel visual-inertial skeleton tracking (VIST) framework, that can provide accurate and robust hand (and finger) motion tracking even for many challenging real-world scenarios and environments, for which the state-of-the-art technologies are known to fail due to their respective fundamental limitations (e.g., severe occlusions for tracking purely with vision sensors; electromagnetic interference for tracking purely with IMUs (inertial measurement units) and compasses; and mechanical contacts for tracking purely with soft sensors). The proposed VIST framework comprises a sensor glove with multiple IMUs and passive visual markers as well as a head-mounted stereo camera; and a tightly-coupled filtering-based visual-inertial fusion algorithm to estimate the hand/finger motion and auto-calibrate hand/glove-related kinematic parameters simultaneously while taking into account the hand anatomical constraints. The VIST framework exhibits good tracking accuracy and robustness, affordable material cost, light hardware and software weights, and ruggedness/durability even to permit washing. Quantitative and qualitative experiments are also performed to validate the advantages and properties of our VIST framework, thereby, clearly demonstrating its potential for real-world applications.손 동작을 기반으로 한 인터페이스는 인간-기계 상호작용 분야에서 직관성, 몰입감, 정교함을 제공해줄 수 있어 많은 주목을 받고 있고, 이를 위해 가장 필수적인 기술 중 하나가 손 동작의 강건하고 정확한 추적 기술 이다. 이를 위해 본 학위논문에서는 먼저 사람 인지의 관점에서 손 동작 추적 오차의 인지 범위를 규명한다. 이 오차 인지 범위는 새로운 손 동작 추적 기술 개발 시 중요한 설계 기준이 될 수 있어 이를 피험자 실험을 통해 정량적으로 밝히고, 특히 손끝 촉각 장비가 있을때 이 인지 범위의 변화도 밝힌다. 이를 토대로, 촉각 피드백을 주는 것이 다양한 인간-기계 상호작용 분야에서 널리 연구되어 왔으므로, 먼저 손끝 촉각 장비와 함께 사용할 수 있는 손 동작 추적 모듈을 개발한다. 이 손끝 촉각 장비는 자기장 외란을 일으켜 착용형 기술에서 흔히 사용되는 지자기 센서를 교란하는데, 이를 적절한 사람 손의 해부학적 특성과 관성 센서/지자기 센서/소프트 센서의 적절한 활용을 통해 해결한다. 이를 확장하여 본 논문에서는, 촉각 장비 착용 시 뿐 아니라 모든 장비 착용 / 환경 / 물체와의 상호작용 시에도 사용 가능한 새로운 손 동작 추적 기술을 제안한다. 기존의 손 동작 추적 기술들은 가림 현상 (영상 기반 기술), 지자기 외란 (관성/지자기 센서 기반 기술), 물체와의 접촉 (소프트 센서 기반 기술) 등으로 인해 제한된 환경에서 밖에 사용하지 못한다. 이를 위해 많은 문제를 일으키는 지자기 센서 없이 상보적인 특성을 지니는 관성 센서와 영상 센서를 융합하고, 이때 작은 공간에 다 자유도의 움직임을 갖는 손 동작을 추적하기 위해 다수의 구분되지 않는 마커들을 사용한다. 이 마커의 구분 과정 (correspondence search)를 위해 기존의 약결합 (loosely-coupled) 기반이 아닌 강결합 (tightly-coupled 기반 센서 융합 기술을 제안하고, 이를 통해 지자기 센서 없이 정확한 손 동작이 가능할 뿐 아니라 착용형 센서들의 정확성/편의성에 문제를 일으키던 센서 부착 오차 / 사용자의 손 모양 등을 자동으로 정확히 보정한다. 이 제안된 영상-관성 센서 융합 기술 (Visual-Inertial Skeleton Tracking (VIST)) 의 뛰어난 성능과 강건성이 다양한 정량/정성 실험을 통해 검증되었고, 이는 VIST의 다양한 일상환경에서 기존 시스템이 구현하지 못하던 손 동작 추적을 가능케 함으로써, 많은 인간-기계 상호작용 분야에서의 가능성을 보여준다.1 Introduction 1 1.1. Motivation 1 1.2. Related Work 5 1.3. Contribution 12 2 Detection Threshold of Hand Tracking Error 16 2.1. Motivation 16 2.2. Experimental Environment 20 2.2.1. Hardware Setup 21 2.2.2. Virtual Environment Rendering 23 2.2.3. HMD Calibration 23 2.3. Identifying the Detection Threshold of Tracking Error 26 2.3.1. Experimental Setup 27 2.3.2. Procedure 27 2.3.3. Experimental Result 31 2.4. Enlarging the Detection Threshold of Tracking Error by Haptic Feedback 31 2.4.1. Experimental Setup 31 2.4.2. Procedure 32 2.4.3. Experimental Result 34 2.5. Discussion 34 3 Wearable Finger Tracking Module for Haptic Interaction 38 3.1. Motivation 38 3.2. Development of Finger Tracking Module 42 3.2.1. Hardware Setup 42 3.2.2. Tracking algorithm 45 3.2.3. Calibration method 48 3.3. Evaluation for VR Haptic Interaction Task 50 3.3.1. Quantitative evaluation of FTM 50 3.3.2. Implementation of Wearable Cutaneous Haptic Interface 51 3.3.3. Usability evaluation for VR peg-in-hole task 53 3.4. Discussion 57 4 Visual-Inertial Skeleton Tracking for Human Hand 59 4.1. Motivation 59 4.2. Hardware Setup and Hand Models 62 4.2.1. Human Hand Model 62 4.2.2. Wearable Sensor Glove 62 4.2.3. Stereo Camera 66 4.3. Visual Information Extraction 66 4.3.1. Marker Detection in Raw Images 68 4.3.2. Cost Function for Point Matching 68 4.3.3. Left-Right Stereo Matching 69 4.4. IMU-Aided Correspondence Search 72 4.5. Filtering-based Visual-Inertial Sensor Fusion 76 4.5.1. EKF States for Hand Tracking and Auto-Calibration 78 4.5.2. Prediction with IMU Information 79 4.5.3. Correction with Visual Information 82 4.5.4. Correction with Anatomical Constraints 84 4.6. Quantitative Evaluation for Free Hand Motion 87 4.6.1. Experimental Setup 87 4.6.2. Procedure 88 4.6.3. Experimental Result 90 4.7. Quantitative and Comparative Evaluation for Challenging Hand Motion 95 4.7.1. Experimental Setup 95 4.7.2. Procedure 96 4.7.3. Experimental Result 98 4.7.4. Performance Comparison with Existing Methods for Challenging Hand Motion 101 4.8. Qualitative Evaluation for Real-World Scenarios 105 4.8.1. Visually Complex Background 105 4.8.2. Object Interaction 106 4.8.3. Wearing Fingertip Cutaneous Haptic Devices 109 4.8.4. Outdoor Environment 111 4.9. Discussion 112 5 Conclusion 116 References 124 Abstract (in Korean) 139 Acknowledgment 141박

손끝 햅틱 장비를 위한 의사 햅틱의 활용

학위논문 (석사)-- 서울대학교 대학원 : 기계항공공학부, 2016. 2. 이동준.We propose a novel design of cutaneous fingertip haptic device and approach of integrating pseudo-haptics into our cutaneous haptic device. With 2-DoF cutaneous device, angle-force calibration result is presented for its operation. Then, 3-DoF cutaneous haptic device is designed for more realistic contact feedback in virtual reality (VR). Preliminary result of integrating cutaneous device and hand tracking device for complete wearable haptic interface is also demonstrated. Meanwhile, we explore possible utility of pseudo-haptics for cutaneous fingertip haptic device, whose performance is inherently limited due to the lack of kinesthetic feedback. We experimentally demonstrate that: 1) pseudo-haptics can render virtual stiffness to be more rigid or softer only by modulating visual cueand 2) pseudo-haptics can be used to expand the range of the perceived virtual stiffness to be doubled.Chapter 1 Introduction 1 1.1 Motivation and Objectives 1 1.2 Related Works 3 Chapter 2 Cutaneous Fingertip Haptic Device 6 2.1 2-DoF Cutaneous Haptic Device 6 2.1.1 Design and Specification 6 2.1.2 Angle-Force Calibration 8 2.1.3 Application of 2-DoF Cutaneous Haptic Device 10 2.2 3-DoF Cutaneous Haptic Device 11 2.2.1 Design and Specification 11 2.2.2 Control Design 14 2.2.3 IMU Distortion Offset Calibration 17 2.2.4 Device Validation 20 2.2.5 Integration with Wearable Hand Tracking Interface 21 Chapter 3 Pseudo-Haptics with Cutaneous Haptic Feedback 25 3.1 Limitation of Cutaneous Haptic Device 25 3.2 Application of Pseudo-Haptics Effect 26 Chapter 4 Experimental Study 28 4.1 Experimental Settings 28 4.2 Experiment #1 32 4.3 Experiment #2 34 4.4 Experiment #3 36 4.5 Discussion 38 Chapter 5 Conclusion and Future Work 40 5.1 Conclusion 40 5.2 Future Work 41 Bibliography 42 요약 50Maste

Design of a wearable fingertip haptic device for remote palpation: Characterisation and interface with a virtual environment

© 2018 Tzemanaki, Al, Melhuish and Dogramadzi. This paper presents the development of a wearable Fingertip Haptic Device (FHD) that can provide cutaneous feedback via a Variable Compliance Platform (VCP). The FHD includes an inertial measurement unit, which tracks the motion of the user's finger while its haptic functionality relies on two parameters: pressure in the VCP and its linear displacement towards the fingertip. The combination of these two features results in various conditions of the FHD, which emulate the remote object or surface stiffness properties. Such a device can be used in tele-operation, including virtual reality applications, where rendering the level of stiffness of different physical or virtual materials could provide a more realistic haptic perception to the user. The FHD stiffness representation is characterised in terms of resulting pressure and force applied to the fingertip created through the relationship of the two functional parameters - pressure and displacement of the VCP. The FHD was tested in a series of user studies to assess its potential to create a user perception of the object's variable stiffness. The viability of the FHD as a haptic device has been further confirmed by interfacing the users with a virtual environment. The developed virtual environment task required the users to follow a virtual path, identify objects of different hardness on the path and navigate away from "no-go" zones. The task was performed with and without the use of the variable compliance on the FHD. The results showed improved performance with the presence of the variable compliance provided by the FHD in all assessed categories and particularly in the ability to identify correctly between objects of different hardness

Wearable haptic systems for the fingertip and the hand: taxonomy, review and perspectives

In the last decade, we have witnessed a drastic change in the form factor of audio and vision technologies, from heavy and grounded machines to lightweight devices that naturally fit our bodies. However, only recently, haptic systems have started to be designed with wearability in mind. The wearability of haptic systems enables novel forms of communication, cooperation, and integration between humans and machines. Wearable haptic interfaces are capable of communicating with the human wearers during their interaction with the environment they share, in a natural and yet private way. This paper presents a taxonomy and review of wearable haptic systems for the fingertip and the hand, focusing on those systems directly addressing wearability challenges. The paper also discusses the main technological and design challenges for the development of wearable haptic interfaces, and it reports on the future perspectives of the field. Finally, the paper includes two tables summarizing the characteristics and features of the most representative wearable haptic systems for the fingertip and the hand

An Overview of Wearable Haptic Technologies and Their Performance in Virtual Object Exploration.

We often interact with our environment through manual handling of objects and exploration of their properties. Object properties (OP), such as texture, stiffness, size, shape, temperature, weight, and orientation provide necessary information to successfully perform interactions. The human haptic perception system plays a key role in this. As virtual reality (VR) has been a growing field of interest with many applications, adding haptic feedback to virtual experiences is another step towards more realistic virtual interactions. However, integrating haptics in a realistic manner, requires complex technological solutions and actual user-testing in virtual environments (VEs) for verification. This review provides a comprehensive overview of recent wearable haptic devices (HDs) categorized by the OP exploration for which they have been verified in a VE. We found 13 studies which specifically addressed user-testing of wearable HDs in healthy subjects. We map and discuss the different technological solutions for different OP exploration which are useful for the design of future haptic object interactions in VR, and provide future recommendations

Sensory Communication

Contains table of contents for Section 2, an introduction and reports on fifteen research projects.National Institutes of Health Grant RO1 DC00117National Institutes of Health Grant RO1 DC02032National Institutes of Health Contract P01-DC00361National Institutes of Health Contract N01-DC22402National Institutes of Health/National Institute on Deafness and Other Communication Disorders Grant 2 R01 DC00126National Institutes of Health Grant 2 R01 DC00270National Institutes of Health Contract N01 DC-5-2107National Institutes of Health Grant 2 R01 DC00100U.S. Navy - Office of Naval Research/Naval Air Warfare Center Contract N61339-94-C-0087U.S. Navy - Office of Naval Research/Naval Air Warfare Center Contract N61339-95-K-0014U.S. Navy - Office of Naval Research/Naval Air Warfare Center Grant N00014-93-1-1399U.S. Navy - Office of Naval Research/Naval Air Warfare Center Grant N00014-94-1-1079U.S. Navy - Office of Naval Research Subcontract 40167U.S. Navy - Office of Naval Research Grant N00014-92-J-1814National Institutes of Health Grant R01-NS33778U.S. Navy - Office of Naval Research Grant N00014-88-K-0604National Aeronautics and Space Administration Grant NCC 2-771U.S. Air Force - Office of Scientific Research Grant F49620-94-1-0236U.S. Air Force - Office of Scientific Research Agreement with Brandeis Universit

First validation of the Haptic Sandwich: a shape changing handheld haptic navigation aid

This paper presents the Haptic Sandwich, a handheld robotic device that designed to provide pedestrian navigation instructions through a novel shape changing modality. The device resembles a cube with an articulated upper half that is able to rotate and translate (extend) relative to the bottom half, which is grounded in the user’s hand when the device is held. The poses assumed by the device simultaneously correspond to heading and proximity to a navigational target. The Haptic Sandwich provides an alternative to screen and/or audio based pedestrian navigation technologies for both visually impaired and sighted users. Unlike other robotic or haptic navigational solutions, the haptic sandwich is discrete in terms of form and sensory stimulus. Due to the novel and unexplored nature of shape changing interfaces, two user studies were undertaken to validate the concept and device. In the first experiment, stationary participants attempted to identify poses assumed by the device, which was hidden from view. In the second experiment, participants attempted to locate a sequence of invisible navigational targets while walking with the device. Of 1080 pose presentations to 10 individuals in experiment one, 80% were correctly identified and 17.5% had the minimal possible error. Multi-DOF errors accounted for only 1.1% of all answers. The role of simultaneous or independent actuator motion on final shape perception was tested with no significant performance difference. The rotation and extension DOF had significantly different perception accuracy. In the second experiment, participants demonstrated good navigational ability with the device after minimal training and were able to locate all presented targets. Mean motion efficiency of the participants was between 32%-56%. Participants made use of both DOF

Sensory Augmentation for Balance Rehabilitation Using Skin Stretch Feedback

This dissertation focuses on the development and evaluation of portable sensory augmentation systems that render skin-stretch feedback of posture for standing balance training and for postural control improvement. Falling is one of the main causes of fatal injuries among all members of the population. The high incidence of fall-related injuries also leads to high medical expenses, which cost approximately $34 billion annually in the United States. People with neurological diseases, e.g., stroke, multiple sclerosis, spinal cord injuries, and the elderly are more prone to falling when compared to healthy individuals. Falls among these populations can also lead to hip fracture, or even death. Thus, several balance and gait rehabilitation approaches have been developed to reduce the risk of falling. Traditionally, a balance-retraining program includes a series of exercises for trainees to strengthen their sensorimotor and musculoskeletal systems. Recent advances in technology have incorporated biofeedback such as visual, auditory, or haptic feedback to provide the users with extra cues about their postural sway. Studies have also demonstrated the positive effects of biofeedback on balance control. However, current applications of biofeedback for interventions in people with impaired balance are still lacking some important characteristics such as portability (in-home care), small-size, and long-term viability. Inspired by the concept of light touch, a light, small, and wearable sensory augmentation system that detects body sway and supplements skin stretch on one’s fingertip pad was first developed. The addition of a shear tactile display could significantly enhance the sensation to body movement. Preliminary results have shown that the application of passive skin stretch feedback at the fingertip enhanced standing balance of healthy young adults. Based on these findings, two research directions were initiated to investigate i) which dynamical information of postural sway could be more effectively conveyed by skin stretch feedback, and ii) how can such feedback device be easily used in the clinical setting or on a daily basis. The major sections of this research are focused on understanding how the skin stretch feedback affects the standing balance and on quantifying the ability of humans to interpret the cutaneous feedback as the cues of their physiological states. Experimental results from both static and dynamic balancing tasks revealed that healthy subjects were able to respond to the cues and subsequently correct their posture. However, it was observed that the postural sway did not generally improve in healthy subjects due to skin stretch feedback. A possible reason was that healthy subjects already had good enough quality sensory information such that the additional artificial biofeedback may have interfered with other sensory cues. Experiments incorporating simulated sensory deficits were further conducted and it was found that subjects with perturbed sensory systems (e.g., unstable surface) showed improved balance due to skin stretch feedback when compared to the neutral standing conditions. Positive impacts on balance performance have also been demonstrated among multiple sclerosis patients when they receive skin stretch feedback from a sensory augmentation walker. The findings in this research indicated that the skin stretch feedback rendered by the developed devices affected the human balance and can potentially compensate underlying neurological or musculoskeletal disorders, therefore enhancing quiet standing postural control

Sensory Augmentation for Balance Rehabilitation Using Skin Stretch Feedback

This dissertation focuses on the development and evaluation of portable sensory augmentation systems that render skin-stretch feedback of posture for standing balance training and for postural control improvement. Falling is one of the main causes of fatal injuries among all members of the population. The high incidence of fall-related injuries also leads to high medical expenses, which cost approximately $34 billion annually in the United States. People with neurological diseases, e.g., stroke, multiple sclerosis, spinal cord injuries, and the elderly are more prone to falling when compared to healthy individuals. Falls among these populations can also lead to hip fracture, or even death. Thus, several balance and gait rehabilitation approaches have been developed to reduce the risk of falling. Traditionally, a balance-retraining program includes a series of exercises for trainees to strengthen their sensorimotor and musculoskeletal systems. Recent advances in technology have incorporated biofeedback such as visual, auditory, or haptic feedback to provide the users with extra cues about their postural sway. Studies have also demonstrated the positive effects of biofeedback on balance control. However, current applications of biofeedback for interventions in people with impaired balance are still lacking some important characteristics such as portability (in-home care), small-size, and long-term viability. Inspired by the concept of light touch, a light, small, and wearable sensory augmentation system that detects body sway and supplements skin stretch on one’s fingertip pad was first developed. The addition of a shear tactile display could significantly enhance the sensation to body movement. Preliminary results have shown that the application of passive skin stretch feedback at the fingertip enhanced standing balance of healthy young adults. Based on these findings, two research directions were initiated to investigate i) which dynamical information of postural sway could be more effectively conveyed by skin stretch feedback, and ii) how can such feedback device be easily used in the clinical setting or on a daily basis. The major sections of this research are focused on understanding how the skin stretch feedback affects the standing balance and on quantifying the ability of humans to interpret the cutaneous feedback as the cues of their physiological states. Experimental results from both static and dynamic balancing tasks revealed that healthy subjects were able to respond to the cues and subsequently correct their posture. However, it was observed that the postural sway did not generally improve in healthy subjects due to skin stretch feedback. A possible reason was that healthy subjects already had good enough quality sensory information such that the additional artificial biofeedback may have interfered with other sensory cues. Experiments incorporating simulated sensory deficits were further conducted and it was found that subjects with perturbed sensory systems (e.g., unstable surface) showed improved balance due to skin stretch feedback when compared to the neutral standing conditions. Positive impacts on balance performance have also been demonstrated among multiple sclerosis patients when they receive skin stretch feedback from a sensory augmentation walker. The findings in this research indicated that the skin stretch feedback rendered by the developed devices affected the human balance and can potentially compensate underlying neurological or musculoskeletal disorders, therefore enhancing quiet standing postural control

Review of Wearable Devices and Data Collection Considerations for Connected Health

Wearable sensor technology has gradually extended its usability into a wide range of well-known applications. Wearable sensors can typically assess and quantify the wearer’s physiology and are commonly employed for human activity detection and quantified self-assessment. Wearable sensors are increasingly utilised to monitor patient health, rapidly assist with disease diagnosis, and help predict and often improve patient outcomes. Clinicians use various self-report questionnaires and well-known tests to report patient symptoms and assess their functional ability. These assessments are time consuming and costly and depend on subjective patient recall. Moreover, measurements may not accurately demonstrate the patient’s functional ability whilst at home. Wearable sensors can be used to detect and quantify specific movements in different applications. The volume of data collected by wearable sensors during long-term assessment of ambulatory movement can become immense in tuple size. This paper discusses current techniques used to track and record various human body movements, as well as techniques used to measure activity and sleep from long-term data collected by wearable technology devices