Enhancing Upper Limb Prostheses Through Neuromorphic Sensory Feedback

Upper limb prostheses are rapidly improving in terms of both control and sensory feedback, giving rise to lifelike robotic devices that aim to restore function to amputees. Recent progress in forward control has enabled prosthesis users to make complicated grip patterns with a prosthetic hand and nerve stimulation has enabled sensations of touch in the missing hand of an amputee. A brief overview of the motivation behind the work in this thesis is given in Chapter 1, which is followed by a general overview of the field and state of the art research (Chapter 2). Chapters 3 and 4 look at the use of closed loop tactile feedback for improving prosthesis grasping functionality. This entails development of two algorithms for improving object manipulation (Chapter 3) and the first real-time implementation of neuromorphic tactile signals being used as feedback to a prosthesis controller for improved grasping (Chapter 4). The second half of the thesis (Chatpers 5 - 7) details how sensory information can be conveyed back to an amputee and how the tactile sensations can be utilized for creating a more lifelike prosthesis. Noninvasive electrical nerve stimulation was shown to provide sensations in multiple regions of the phantom hand of amputees both with and without targeted sensory reinnervation surgery (Chapter 5). A multilayered electronic dermis (e-dermis) was developed to mimic the behavior of receptors in the skin to provide, for the first time, sensations of both touch and pain back to an amputee and the prosthesis (Chapter 6). Finally, the first demonstration of sensory feedback as a key component of phantom hand movement for myoelectric pattern recognition shows that enhanced perceptions of the phantom hand can lead to improved prosthesis control (Chapter 7). This work provides the first demonstration of how amputees can perceive multiple tactile sensations through a neuromorphic stimulation paradigm. Furthermore, it describes the unique role that nerve stimulation and phantom hand activation play in the sensorimotor loop of upper limb amputees

E-skin: from humanoids to humans

With robots starting to enter our lives in a number of ways (e.g., social, assistive, and surgery), the electronic skin (e-skin) is becoming increasingly important. The capability of detecting subtle pressure or temperature changes makes the e-skin an essential component of a robot's body or an artificial limb [1], [2]. This is because the tactile feedback enabled by e-skin plays a fundamental role in providing action-related information such as slip during manipulation/control tasks such as grasping, and estimation of contact parameters (e.g., force, soft contact, hardness, texture, and temperature during exploration [3]). It is critical for the safe robotic interaction - albeit as a coworker in the futuristic industry 4.0 setting or to assist the elderly at home

Flexible Carbon-Based Electronics and Sensorized Neuroprosthesis

In the United States alone, there are more than 2 million people living with limb loss and prosthetic devices have long been the solution to recover their activities of daily living. However, many of the prosthetic users reported their dissatisfaction with current prostheses and some even abandoned theirs due to poor comfort and limited performance. To improve prosthetic control, advancements in surgical interfaces and sensorized neuroprosthesis are two major focus and have seen great potential. Both perspectives are presented in this work. Several reinnervated muscle surgeries have been invented to enable a better communication with muscle and nerves and a stable interface is essential to record robust muscle signals which are utilized to control a neuroprosthesis. Each muscle target may have slightly different anatomy and the current state-of-the-art implantable electrodes are complex and not easily reproducible and customizable. To address this problem, I present a simple, rapid electrode fabrication method to record muscle signals and easy-to-use electrode materials using carbon black/polydimethylsiloxane (PDMS) composite. Acute in vivo testing shows that the electrodes are highly functional and have the potential to enable large-scale muscle signal recordings with extensive data to improve the neuroprosthetic control. In addition to novel neural interfaces, sensory perception is also critical to improve the manipulation of objects with a prosthesis and enhances prosthetic performance and embodiment with feedback to the user. With recent advances in tactile sensing technology and neuromorphic stimulation interface, efficient real-time communication and functioning between them are still missing. In this work, I build and test a closed-loop system that integrates tactile sensing and neuromorphic electrical stimulation. The system functions in real time and the parameters of the sensory stimulation through transcutaneous electrical nerve stimulation (TENS) convey temporal information and dynamically change responding to real-time tactile data

Dermal Sensory Regenerative Peripheral Nerve Interface for Reestablishing Sensory Nerve Feedback in Peripheral Afferents in the Rat

Background: Without meaningful, intuitive sensory feedback, even the most advanced myoelectric devices require significant cognitive demand to control. The dermal sensory regenerative peripheral nerve interface (DS-RPNI) is a biological interface designed to establish high-fidelity sensory feedback from prosthetic limbs. Methods: DS-RPNIs were constructed in rats by securing fascicles of residual sensory peripheral nerves into autologous dermal grafts, with the objectives of confirming regeneration of sensory afferents within DS-RPNIs and establishing the reliability of afferent neural response generation with either mechanical or electrical stimulation. Results: Two months after implantation, DS-RPNIs were healthy and displayed well-vascularized dermis with organized axonal collaterals throughout and no evidence of neuroma. Electrophysiologic signals were recorded proximal from DS-RPNI's sural nerve in response to both mechanical and electrical stimuli and compared with (1) full-thickness skin, (2) deepithelialized skin, and (3) transected sural nerves without DS-RPNI. Mechanical indentation of DS-RPNIs evoked compound sensory nerve action potentials (CSNAPs) that were like those evoked during indentation of full-thickness skin. CSNAP firing rates and waveform amplitudes increased in a graded fashion with increased mechanical indentation. Electrical stimuli delivered to DS-RPNIs reliably elicited CSNAPs at low current thresholds, and CSNAPs gradually increased in amplitude with increasing stimulation current. Conclusions: These findings suggest that afferent nerve fibers successfully reinnervate DS-RPNIs, and that graded stimuli applied to DS-RPNIs produce proximal sensory afferent responses similar to those evoked from normal skin. This confirmation of graded afferent signal transduction through DS-RPNI neural interfaces validate DS-RPNI's potential role of facilitating sensation in human-machine interfacing. Clinical Relevance Statement: The DS-RPNI is a novel biotic-abiotic neural interface that allows for transduction of sensory stimuli into neural signals. It is expected to advance the restoration of natural sensation and development of sensorimotor control in prosthetics.</p

A hierarchical sensorimotor control framework for human-in-the-loop robotic hands.

Human manual dexterity relies critically on touch. Robotic and prosthetic hands are much less dexterous and make little use of the many tactile sensors available. We propose a framework modeled on the hierarchical sensorimotor controllers of the nervous system to link sensing to action in human-in-the-loop, haptically enabled, artificial hands

Human Sensation of Transcranial Electric Stimulation.

Noninvasive transcranial electric stimulation is increasingly being used as an advantageous therapy alternative that may activate deep tissues while avoiding drug side-effects. However, not only is there limited evidence for activation of deep tissues by transcranial electric stimulation, its evoked human sensation is understudied and often dismissed as a placebo or secondary effect. By systematically characterizing the human sensation evoked by transcranial alternating-current stimulation, we observed not only stimulus frequency and electrode position dependencies specific for auditory and visual sensation but also a broader presence of somatic sensation ranging from touch and vibration to pain and pressure. We found generally monotonic input-output functions at suprathreshold levels, and often multiple types of sensation occurring simultaneously in response to the same electric stimulation. We further used a recording circuit embedded in a cochlear implant to directly and objectively measure the amount of transcranial electric stimulation reaching the auditory nerve, a deep intercranial target located in the densest bone of the skull. We found an optimal configuration using an ear canal electrode and low-frequency (&lt;300 Hz) sinusoids that delivered maximally ~1% of the transcranial current to the auditory nerve, which was sufficient to produce sound sensation even in deafened ears. Our results suggest that frequency resonance due to neuronal intrinsic electric properties need to be explored for targeted deep brain stimulation and novel brain-computer interfaces

Non-rectangular neurostimulation waveforms elicit varied sensation quality and perceptive fields on the hand

Electrical stimulation of the nerves is known to elicit distinct sensations perceived in distal parts of the body. The stimulation is typically modulated in current with charge-balanced rectangular shapes that, although easily generated by stimulators available on the market, are not able to cover the entire range of somatosensory experiences from daily life. In this regard, we have investigated the effect of electrical neurostimulation with four non-rectangular waveforms in an experiment involving 11 healthy able-bodied subjects. Weiss curves were estimated and rheobase and chronaxie values were obtained showing increases in stimulation time required to elicit sensations for some waveforms. The localization of the sensations reported in the hand also appeared to differ between waveforms, although the total area did not vary significantly. Finally, the possibility of distinguishing different charge- and amplitude-matched stimuli was demonstrated through a two-alternative-forced-choice (2AFC) match-to-sample task, showing the ability of participants to successfully distinguish between waveforms with similar electrical characteristics but different shapes and charge transfer rates. This study provides evidence that, by using different waveforms to stimulate nerves, it is possible to affect not only the required charge to elicit sensations but also the sensation\ua0quality and its localization

Soft Biomimetic Finger with Tactile Sensing and Sensory Feedback Capabilities

The compliant nature of soft fingers allows for safe and dexterous manipulation of objects by humans in an unstructured environment. A soft prosthetic finger design with tactile sensing capabilities for texture discrimination and subsequent sensory stimulation has the potential to create a more natural experience for an amputee. In this work, a pneumatically actuated soft biomimetic finger is integrated with a textile neuromorphic tactile sensor array for a texture discrimination task. The tactile sensor outputs were converted into neuromorphic spike trains, which emulate the firing pattern of biological mechanoreceptors. Spike-based features from each taxel compressed the information and were then used as inputs for the support vector machine (SVM) classifier to differentiate the textures. Our soft biomimetic finger with neuromorphic encoding was able to achieve an average overall classification accuracy of 99.57% over sixteen independent parameters when tested on thirteen standardized textured surfaces. The sixteen parameters were the combination of four angles of flexion of the soft finger and four speeds of palpation. To aid in the perception of more natural objects and their manipulation, subjects were provided with transcutaneous electrical nerve stimulation (TENS) to convey a subset of four textures with varied textural information. Three able-bodied subjects successfully distinguished two or three textures with the applied stimuli. This work paves the way for a more human-like prosthesis through a soft biomimetic finger with texture discrimination capabilities using neuromorphic techniques that provides sensory feedback; furthermore, texture feedback has the potential to enhance the user experience when interacting with their surroundings. Additionally, this work showed that an inexpensive, soft biomimetic finger combined with a flexible tactile sensor array can potentially help users perceive their environment better