A Synergy-Based Optimally Designed Sensing Glove for Functional Grasp Recognition

Achieving accurate and reliable kinematic hand pose reconstructions represents a challenging task. The main reason for this is the complexity of hand biomechanics, where several degrees of freedom are distributed along a continuous deformable structure. Wearable sensing can represent a viable solution to tackle this issue, since it enables a more natural kinematic monitoring. However, the intrinsic accuracy (as well as the number of sensing elements) of wearable hand pose reconstruction (HPR) systems can be severely limited by ergonomics and cost considerations. In this paper, we combined the theoretical foundations of the optimal design of HPR devices based on hand synergy information, i.e., the inter-joint covariation patterns, with textile goniometers based on knitted piezoresistive fabrics (KPF) technology, to develop, for the first time, an optimally-designed under-sensed glove for measuring hand kinematics. We used only five sensors optimally placed on the hand and completed hand pose reconstruction (described according to a kinematic model with 19 degrees of freedom) leveraging upon synergistic information. The reconstructions we obtained from five different subjects were used to implement an unsupervised method for the recognition of eight functional grasps, showing a high degree of accuracy and robustness

Hand motion analysis during the execution of the action research arm test using multiple sensors

The Action Research Arm Test (ARAT) is a standardized outcome measure that can be improved by integrating sensors for hand motion analysis. The purpose of this study is to measure the flexion angle of the finger joints and fingertip forces during the performance of three subscales (Grasp, Grip, and Pinch) of the ARAT, using a data glove (CyberGlove II®) and five force-sensing resistors (FSRs) simultaneously. An experimental study was carried out with 25 healthy subjects (right-handed). The results showed that the mean flexion angles of the finger joints required to perform the 16 activities were Thumb (Carpometacarpal Joint (CMC) 28.56°, Metacarpophalangeal Joint (MCP) 26.84°, and Interphalangeal Joint (IP) 13.23°), Index (MCP 46.18°, Index Proximal Interphalangeal Joint (PIP) 38.89°), Middle (MCP 47.5°, PIP 42.62°), Ring (MCP 44.09°, PIP 39.22°), and Little (MCP 31.50°, PIP 22.10°). The averaged fingertip force exerted in the Grasp Subscale was 8.2 N, in Grip subscale 6.61 N and Pinch subscale 3.89 N. These results suggest that the integration of multiple sensors during the performance of the ARAT has clinical relevance, allowing therapists and other health professionals to perform a more sensitive, objective, and quantitative assessment of the hand function.Postprint (published version

Glove-based systems for medical applications: review of recent advancements

Human hand motion analysis is attracting researchers in the areas of neuroscience, biomedical engineering, robotics, human-machines interfaces (HMI), human-computer interaction (HCI), and artificial intelligence (AI). Among the others, the fields of medical rehabilitation and physiological assessments are suggesting high impact applications for wearable sensing systems. Glove-based systems are one of the most significant devices in assessing quantities related to hand movements. This paper provides updated survey among the main glove solutions proposed in literature for hand rehabilitation. Then, the process for designing glove-based systems is defined, by including all relevant design issues for researchers and makers. The main goal of the paper is to describe the basics of glove-based systems and to outline their potentialities and limitations. At the same time, roadmap to design and prototype the next generation of these devices is defined, according to the results of previous experiences in the scientific community

A Two-Axis Goniometric Sensor for Tracking Finger Motion

The study of finger kinematics has developed into an important research area. Various hand tracking systems are currently available; however, they all have limited functionality. Generally, the most commonly adopted sensors are limited to measurements with one degree of freedom, i.e., flexion/extension of fingers. More advanced measurements including finger abduction, adduction, and circumduction are much more difficult to achieve. To overcome these limitations, we propose a two-axis 3D printed optical sensor with a compact configuration for tracking finger motion. Based on Malus’ law, this sensor detects the angular changes by analyzing the attenuation of light transmitted through polarizing film. The sensor consists of two orthogonal axes each containing two pathways. The two readings from each axis are fused using a weighted average approach, enabling a measurement range up to 180 ∘ and an improvement in sensitivity. The sensor demonstrates high accuracy (±0.3 ∘ ), high repeatability, and low hysteresis error. Attaching the sensor to the index finger’s metacarpophalangeal joint, real-time movements consisting of flexion/extension, abduction/adduction and circumduction have been successfully recorded. The proposed two-axis sensor has demonstrated its capability for measuring finger movements with two degrees of freedom and can be potentially used to monitor other types of body motion

인간 기계 상호작용을 위한 강건하고 정확한 손동작 추적 기술 연구

학위논문(박사) -- 서울대학교대학원 : 공과대학 기계항공공학부, 2021.8. 이동준.Hand-based interface is promising for realizing intuitive, natural and accurate human machine interaction (HMI), as the human hand is main source of dexterity in our daily activities. For this, the thesis begins with the human perception study on the detection threshold of visuo-proprioceptive conflict (i.e., allowable tracking error) with or without cutantoues haptic feedback, and suggests tracking error specification for realistic and fluidic hand-based HMI. The thesis then proceeds to propose a novel wearable hand tracking module, which, to be compatible with the cutaneous haptic devices spewing magnetic noise, opportunistically employ heterogeneous sensors (IMU/compass module and soft sensor) reflecting the anatomical properties of human hand, which is suitable for specific application (i.e., finger-based interaction with finger-tip haptic devices). This hand tracking module however loses its tracking when interacting with, or being nearby, electrical machines or ferromagnetic materials. For this, the thesis presents its main contribution, a novel visual-inertial skeleton tracking (VIST) framework, that can provide accurate and robust hand (and finger) motion tracking even for many challenging real-world scenarios and environments, for which the state-of-the-art technologies are known to fail due to their respective fundamental limitations (e.g., severe occlusions for tracking purely with vision sensors; electromagnetic interference for tracking purely with IMUs (inertial measurement units) and compasses; and mechanical contacts for tracking purely with soft sensors). The proposed VIST framework comprises a sensor glove with multiple IMUs and passive visual markers as well as a head-mounted stereo camera; and a tightly-coupled filtering-based visual-inertial fusion algorithm to estimate the hand/finger motion and auto-calibrate hand/glove-related kinematic parameters simultaneously while taking into account the hand anatomical constraints. The VIST framework exhibits good tracking accuracy and robustness, affordable material cost, light hardware and software weights, and ruggedness/durability even to permit washing. Quantitative and qualitative experiments are also performed to validate the advantages and properties of our VIST framework, thereby, clearly demonstrating its potential for real-world applications.손 동작을 기반으로 한 인터페이스는 인간-기계 상호작용 분야에서 직관성, 몰입감, 정교함을 제공해줄 수 있어 많은 주목을 받고 있고, 이를 위해 가장 필수적인 기술 중 하나가 손 동작의 강건하고 정확한 추적 기술 이다. 이를 위해 본 학위논문에서는 먼저 사람 인지의 관점에서 손 동작 추적 오차의 인지 범위를 규명한다. 이 오차 인지 범위는 새로운 손 동작 추적 기술 개발 시 중요한 설계 기준이 될 수 있어 이를 피험자 실험을 통해 정량적으로 밝히고, 특히 손끝 촉각 장비가 있을때 이 인지 범위의 변화도 밝힌다. 이를 토대로, 촉각 피드백을 주는 것이 다양한 인간-기계 상호작용 분야에서 널리 연구되어 왔으므로, 먼저 손끝 촉각 장비와 함께 사용할 수 있는 손 동작 추적 모듈을 개발한다. 이 손끝 촉각 장비는 자기장 외란을 일으켜 착용형 기술에서 흔히 사용되는 지자기 센서를 교란하는데, 이를 적절한 사람 손의 해부학적 특성과 관성 센서/지자기 센서/소프트 센서의 적절한 활용을 통해 해결한다. 이를 확장하여 본 논문에서는, 촉각 장비 착용 시 뿐 아니라 모든 장비 착용 / 환경 / 물체와의 상호작용 시에도 사용 가능한 새로운 손 동작 추적 기술을 제안한다. 기존의 손 동작 추적 기술들은 가림 현상 (영상 기반 기술), 지자기 외란 (관성/지자기 센서 기반 기술), 물체와의 접촉 (소프트 센서 기반 기술) 등으로 인해 제한된 환경에서 밖에 사용하지 못한다. 이를 위해 많은 문제를 일으키는 지자기 센서 없이 상보적인 특성을 지니는 관성 센서와 영상 센서를 융합하고, 이때 작은 공간에 다 자유도의 움직임을 갖는 손 동작을 추적하기 위해 다수의 구분되지 않는 마커들을 사용한다. 이 마커의 구분 과정 (correspondence search)를 위해 기존의 약결합 (loosely-coupled) 기반이 아닌 강결합 (tightly-coupled 기반 센서 융합 기술을 제안하고, 이를 통해 지자기 센서 없이 정확한 손 동작이 가능할 뿐 아니라 착용형 센서들의 정확성/편의성에 문제를 일으키던 센서 부착 오차 / 사용자의 손 모양 등을 자동으로 정확히 보정한다. 이 제안된 영상-관성 센서 융합 기술 (Visual-Inertial Skeleton Tracking (VIST)) 의 뛰어난 성능과 강건성이 다양한 정량/정성 실험을 통해 검증되었고, 이는 VIST의 다양한 일상환경에서 기존 시스템이 구현하지 못하던 손 동작 추적을 가능케 함으로써, 많은 인간-기계 상호작용 분야에서의 가능성을 보여준다.1 Introduction 1 1.1. Motivation 1 1.2. Related Work 5 1.3. Contribution 12 2 Detection Threshold of Hand Tracking Error 16 2.1. Motivation 16 2.2. Experimental Environment 20 2.2.1. Hardware Setup 21 2.2.2. Virtual Environment Rendering 23 2.2.3. HMD Calibration 23 2.3. Identifying the Detection Threshold of Tracking Error 26 2.3.1. Experimental Setup 27 2.3.2. Procedure 27 2.3.3. Experimental Result 31 2.4. Enlarging the Detection Threshold of Tracking Error by Haptic Feedback 31 2.4.1. Experimental Setup 31 2.4.2. Procedure 32 2.4.3. Experimental Result 34 2.5. Discussion 34 3 Wearable Finger Tracking Module for Haptic Interaction 38 3.1. Motivation 38 3.2. Development of Finger Tracking Module 42 3.2.1. Hardware Setup 42 3.2.2. Tracking algorithm 45 3.2.3. Calibration method 48 3.3. Evaluation for VR Haptic Interaction Task 50 3.3.1. Quantitative evaluation of FTM 50 3.3.2. Implementation of Wearable Cutaneous Haptic Interface 51 3.3.3. Usability evaluation for VR peg-in-hole task 53 3.4. Discussion 57 4 Visual-Inertial Skeleton Tracking for Human Hand 59 4.1. Motivation 59 4.2. Hardware Setup and Hand Models 62 4.2.1. Human Hand Model 62 4.2.2. Wearable Sensor Glove 62 4.2.3. Stereo Camera 66 4.3. Visual Information Extraction 66 4.3.1. Marker Detection in Raw Images 68 4.3.2. Cost Function for Point Matching 68 4.3.3. Left-Right Stereo Matching 69 4.4. IMU-Aided Correspondence Search 72 4.5. Filtering-based Visual-Inertial Sensor Fusion 76 4.5.1. EKF States for Hand Tracking and Auto-Calibration 78 4.5.2. Prediction with IMU Information 79 4.5.3. Correction with Visual Information 82 4.5.4. Correction with Anatomical Constraints 84 4.6. Quantitative Evaluation for Free Hand Motion 87 4.6.1. Experimental Setup 87 4.6.2. Procedure 88 4.6.3. Experimental Result 90 4.7. Quantitative and Comparative Evaluation for Challenging Hand Motion 95 4.7.1. Experimental Setup 95 4.7.2. Procedure 96 4.7.3. Experimental Result 98 4.7.4. Performance Comparison with Existing Methods for Challenging Hand Motion 101 4.8. Qualitative Evaluation for Real-World Scenarios 105 4.8.1. Visually Complex Background 105 4.8.2. Object Interaction 106 4.8.3. Wearing Fingertip Cutaneous Haptic Devices 109 4.8.4. Outdoor Environment 111 4.9. Discussion 112 5 Conclusion 116 References 124 Abstract (in Korean) 139 Acknowledgment 141박

Robotic arm inertial control for recreational child physiotherapy application

According to reports from physical therapists, physiotherapy presents a painful and uninteresting process because it distances itself from the ludic, which is a natural essence of the human being, and easily attracts children when applied. In 2021, the “Robotic System for Children's Physiotherapy with Recreational Activities” project was developed, which consisted of a robotic arm to assist in the physiotherapy process of children with Cerebral Palsy. According to the conclusions of the initial project, the developed control was effective in what was proposed but not sufficient for project application in rehabilitation environments In this way, this work present a new control that is safer and that allows its application helping in the rehabilitation of these children. This new control allows the robotic arm to reproduce the movements of the human arm, so that the movement of the forearm, arm and hand are reproduced independently, simultaneously and in real time. Therefore, the prototype control operates reliably and robustly, achieving what was aimed at, a safer and more efficient physiotherapeutic process for users

Physical human-robot collaboration: Robotic systems, learning methods, collaborative strategies, sensors, and actuators

This article presents a state-of-the-art survey on the robotic systems, sensors, actuators, and collaborative strategies for physical human-robot collaboration (pHRC). This article starts with an overview of some robotic systems with cutting-edge technologies (sensors and actuators) suitable for pHRC operations and the intelligent assist devices employed in pHRC. Sensors being among the essential components to establish communication between a human and a robotic system are surveyed. The sensor supplies the signal needed to drive the robotic actuators. The survey reveals that the design of new generation collaborative robots and other intelligent robotic systems has paved the way for sophisticated learning techniques and control algorithms to be deployed in pHRC. Furthermore, it revealed the relevant components needed to be considered for effective pHRC to be accomplished. Finally, a discussion of the major advances is made, some research directions, and future challenges are presented

A Multi-Modal Sensing Glove for Human Manual-Interaction Studies

We present an integrated sensing glove that combines two of the most visionary wearable sensing technologies to provide both hand posture sensing and tactile pressure sensing in a unique, lightweight, and stretchable device. Namely, hand posture reconstruction employs Knitted Piezoresistive Fabrics that allows us to measure bending. From only five of these sensors (one for each finger) the full hand pose of a 19 degrees of freedom (DOF) hand model is reconstructed leveraging optimal sensor placement and estimation techniques. To this end, we exploit a-priori information of synergistic coordination patterns in grasping tasks. Tactile sensing employs a piezoresistive fabric allowing us to measure normal forces in more than 50 taxels spread over the palmar surface of the glove. We describe both sensing technologies, report on the software integration of both modalities, and describe a preliminary evaluation experiment analyzing hand postures and force patterns during grasping. Results of the reconstruction are promising and encourage us to push further our approach with potential applications in neuroscience, virtual reality, robotics and tele-operation

Accuracy and repeatability of wrist joint angles in boxing using an electromagnetic tracking system

© 2019, The Author(s). The hand-wrist region is reported as the most common injury site in boxing. Boxers are at risk due to the amount of wrist motions when impacting training equipment or their opponents, yet we know relatively little about these motions. This paper describes a new method for quantifying wrist motion in boxing using an electromagnetic tracking system. Surrogate testing procedure utilising a polyamide hand and forearm shape, and in vivo testing procedure utilising 29 elite boxers, were used to assess the accuracy and repeatability of the system. 2D kinematic analysis was used to calculate wrist angles using photogrammetry, whilst the data from the electromagnetic tracking system was processed with visual 3D software. The electromagnetic tracking system agreed with the video-based system (paired t tests) in both the surrogate ( 0.9). In the punch testing, for both repeated jab and hook shots, the electromagnetic tracking system showed good reliability (ICCs > 0.8) and substantial reliability (ICCs > 0.6) for flexion–extension and radial-ulnar deviation angles, respectively. The results indicate that wrist kinematics during punching activities can be measured using an electromagnetic tracking system