Doctor of Philosophy

dissertationThe optimization of novel stretchable fingernail sensors for detecting fingertip touch force direction is introduced. The fingernail sensor uses optical reflectance photoplethysmography to measure the change in blood perfusion in the fingernail bed when the finger pad touches a surface with various forces. This "fingernail sensing" technique involves mounting an array of LEDs (Light Emitting Diodes) and photodetectors on the fingernail surface to detect changes in the reflection intensity as a function of applied force. The intensity changes correspond to changes in blood volume underneath the fingernail and allow for fingertip force detection without haptic obstruction, which has several applications in the area of human-machine interaction. This dissertation experimentally determines the optimal optical parameters for the transmittance of light through the human fingernail bed. Specifically, the effect of varying the wavelength and optical path length on light transmittance through the nail bed are thoroughly investigated. Light transmittance through the human fingernail is optimized when using green light (525nm) and when placing optoelectronic pairs as close together as possible. The optimal locations of the optoelectronic devices are predicted by introducing an optical model that describes light transmittance between an LED and a photodiode in the fingernail area based on optical experimentation. A reduced configuration is derived from the optimal optoelectronic locations in order to facilitate iv the fabrication of the optimized fingernail sensor without significantly compromising the recognition accuracy. This results in an overall force direction recognition accuracy of 95%. Using novel fabrication techniques, we successfully build a stretchable fingernail sensor prototype, which fully conforms to the two-dimensional fingernail surface and is independent of its geometry. Namely, we overcome the challenges of patterning conductive lines on a stretchable substrate, and embedding rigid optical components in a stretchable platform while maintaining electrical conductivity. A finite element analysis is conducted to optimize the electrical contact resistance between the optoelectronic components and underlying stretchable conductors, as a function of the bending curvature and substrate thickness. The functionality of the stretchable sensor is tested in relation to the design parameters. Finally, applications and potential impacts of this work are discussed

FingerTac -- An Interchangeable and Wearable Tactile Sensor for the Fingertips of Human and Robot Hands

Skill transfer from humans to robots is challenging. Presently, many researchers focus on capturing only position or joint angle data from humans to teach the robots. Even though this approach has yielded impressive results for grasping applications, reconstructing motion for object handling or fine manipulation from a human hand to a robot hand has been sparsely explored. Humans use tactile feedback to adjust their motion to various objects, but capturing and reproducing the applied forces is an open research question. In this paper we introduce a wearable fingertip tactile sensor, which captures the distributed 3-axis force vectors on the fingertip. The fingertip tactile sensor is interchangeable between the human hand and the robot hand, meaning that it can also be assembled to fit on a robot hand such as the Allegro hand. This paper presents the structural aspects of the sensor as well as the methodology and approach used to design, manufacture, and calibrate the sensor. The sensor is able to measure forces accurately with a mean absolute error of 0.21, 0.16, and 0.44 Newtons in X, Y, and Z directions, respectively

Doctor of Philosophy

dissertationFingernail imaging is a method of sensing finger force using the color patterns on the nail and surrounding skin. These patterns form as the underlying tissue is compressed and blood pools in the surrounding vessels. Photos of the finger and surrounding skin may be correlated to the magnitude and direction of force on the fingerpad. An automated calibration routine is developed to improve the data-collection process. This includes a novel hybrid force/position controller that manages the interaction between the fingerpad and a flat surface, implemented on a Magnetic Levitation Haptic Device. The kinematic and dynamics parameters of the system are characterized in order to appropriately design a nonlinear compensator. The controller settles within 0.13 s with less than 30% overshoot. A new registration A new registration technique, based on Active Appearance Models, is presented. Since this method accounts for the variation inherent in the finger, it reduces registration and force prediction errors while removing the need to tune registration parameters or reject unregistered images. Modifications to the standard model are also investigated. The number of landmark points is reduced to 25 points with no loss of accuracy, while the use of the green channel is found to have no significant effect on either registration or force prediction accuracy. Several force prediction models are characterized, and the EigenNail Magnitude Model, a Principal Component Regression model on the gray-level intensity, is shown to fit the data most accurately. The mean force prediction error using this prediction and modeling method is 0.55 N. White LEDs and green LEDs are shown to have no statistically significant effect on registration or force prediction. Finally, two different calibration grid designs are compared and found to have no significant effect. Together, these improvements prepare the way for fingernail imaging to be used in less controlled situations. With a wider range of calibration data and a more robust registration method, a larger range of force data may be predicted. Potential applications for this technology include human-computer interaction and measuring finger interaction forces during grasping experiments

Lateral Touch and Frictional Vibrations of Human Fingerprints

In this report, we experimentally show that human fingerprints play a significant role in the lateral touch vibratory mechanisms and that direction of movement results in differ-ent dynamic strains on the pulp. Sources that generate sound at the interface are first empiri-cally identified and a spherical model of the index finger is proposed to forecast the radiated fields. Then, haptic sounds are recorded using a miniature microphone placed in the near field. Results are compared with vibrometric measurements carried out in quite similar conditions to assess of the results relevance. Results show that fingerprints can?t be neglected in the tri-bologic interaction. Finally, this research provide a useful information for tactile displays de-signers and offers future investigations perspectives in many research fields linked with the haptic science

Force-Aware Interface via Electromyography for Natural VR/AR Interaction

While tremendous advances in visual and auditory realism have been made for virtual and augmented reality (VR/AR), introducing a plausible sense of physicality into the virtual world remains challenging. Closing the gap between real-world physicality and immersive virtual experience requires a closed interaction loop: applying user-exerted physical forces to the virtual environment and generating haptic sensations back to the users. However, existing VR/AR solutions either completely ignore the force inputs from the users or rely on obtrusive sensing devices that compromise user experience. By identifying users' muscle activation patterns while engaging in VR/AR, we design a learning-based neural interface for natural and intuitive force inputs. Specifically, we show that lightweight electromyography sensors, resting non-invasively on users' forearm skin, inform and establish a robust understanding of their complex hand activities. Fuelled by a neural-network-based model, our interface can decode finger-wise forces in real-time with 3.3% mean error, and generalize to new users with little calibration. Through an interactive psychophysical study, we show that human perception of virtual objects' physical properties, such as stiffness, can be significantly enhanced by our interface. We further demonstrate that our interface enables ubiquitous control via finger tapping. Ultimately, we envision our findings to push forward research towards more realistic physicality in future VR/AR.Comment: ACM Transactions on Graphics (SIGGRAPH Asia 2022

Visual Estimation of Fingertip Pressure on Diverse Surfaces using Easily Captured Data

People often use their hands to make contact with the world and apply pressure. Machine perception of this important human activity could be widely applied. Prior research has shown that deep models can estimate hand pressure based on a single RGB image. Yet, evaluations have been limited to controlled settings, since performance relies on training data with high-resolution pressure measurements that are difficult to obtain. We present a novel approach that enables diverse data to be captured with only an RGB camera and a cooperative participant. Our key insight is that people can be prompted to perform actions that correspond with categorical labels describing contact pressure (contact labels), and that the resulting weakly labeled data can be used to train models that perform well under varied conditions. We demonstrate the effectiveness of our approach by training on a novel dataset with 51 participants making fingertip contact with instrumented and uninstrumented objects. Our network, ContactLabelNet, dramatically outperforms prior work, performs well under diverse conditions, and matched or exceeded the performance of human annotators

Design and analysis of fingernail sensors for measurement of fingertip touch fouce and finger posture

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering, 2002.Includes bibliographical references (leaves 142-148).A new type of wearable sensor for detecting fingertip touch force and finger posture is presented. Unlike traditional electronic gloves, in which sensors are embedded along the finger and on the fingerpads, this new device does not constrict finger motion and allows the fingers to directly contact the environment without obstructing the human's natural haptic senses. The fingertip touch force and finger posture are detected by measuring changes in the coloration of the fingernail; hence, the sensor is mounted on the fingernail and does not interfere with bending or touching actions. Specifically, the fingernail is instrumented with miniature light emitting diodes (LEDs) and photodetectors in order to measure changes in the reflection intensity when the fingertip is pressed against a surface or when the finger is bent. The changes in intensity are then used to determine changes in the blood volume under the fingernail, a technique termed "reflectance photoplethysmography." By arranging the LEDs and photodetectors in a spatial array, the two-dimensional pattern of blood volume can be measured and used to predict the touch force and posture. This thesis first underscores the role of the fingernail sensor as a means of indirectly detecting fingertip touch force and finger posture by measuring the internal state of the finger. Desired functionality and principles of photoplethysmography are used to create a set of design goals and guidelines for such a sensor.(cont.) A working miniaturized prototype nail sensor is designed, built, tested, and analyzed. Based on fingertip anatomy and photographic evidence, mechanical and hemodynamic models are created in order to understand the mechanism of the blood volume change at multiple locations within the fingernail bed. These models are verified through experiment and simulation. Next, data-driven, mathematical models or filters are designed to comprehensively predict normal touching forces, shear touching forces, and finger bending based on readings from the sensor. A method to experimentally calibrate the filters is designed, implemented, and validated. Using these filters, the sensors are capable of predicting forces to within 0.5 N RMS error and posture angle to within 10 degrees RMS error. Performances of the filters are analyzed, compared, and used to suggest design guidelines for the next generation of sensors. Finally, applications to human-machine interface are discussed and tested, and potential impacts of this work on the fields of virtual reality and robotics are proposed.by Stephen A. Mascaro.Ph.D