Prosthetic Control and Sensory Feedback for Upper Limb Amputees

Hand amputation could dramatically degrade the life quality of amputees. Many amputees use prostheses to restore part of the hand functions. Myoelectric prosthesis provides the most dexterous control. However, they are facing high rejection rate. One of the reasons is the lack of sensory feedback. There is a need for providing sensory feedback for myoelectric prosthesis users. It can improve object manipulation abilities, enhance the perceptual embodiment of myoelectric prostheses and help reduce phantom limb pain. This PhD work focuses on building bi-directional prostheses for upper limb amputees. In the introduction chapter, first, an overview of upper limb amputee demographics and upper limb prosthesis is given. Then the human somatosensory system is briefly introduced. The next part reviews invasive and non-invasive sensory feedback methods reported in the literature. The rest of the chapter describes the motivation of the project and the thesis organization. The first step to build a bi-directional prostheses is to investigate natural and robust multifunctional prosthetic control. Most of the commerical prostheses apply non-pattern recognition based myoelectric control methods, which offers only limited functionalities. In this thesis work, pattern recognition based prosthetic control employing three commonly used and representative machine learning algorithms is investigated. Three datasets involving different levels of upper arm movements are used for testing the algorithm effectiveness. The influence of time-domain features, window and increment sizes, algorithms, and post-processing techniques are analyzed and discussed. The next three chapters address different aspects of providing sensory feedback. The first focus of sensory feedback process is the automatic phantom map detection. Many amputees have referred sensation from their missing hand on their residual limbs (phantom maps). This skin area can serve as a target for providing amputees with non-invasive tactile sensory feedback. One of the challenges of providing sensory feedback on the phantom map is to define the accurate boundary of each phantom digit because the phantom map distribution varies from person to person. Automatic phantom map detection methods based on four decomposition support vector machine algorithms and three sampling methods are proposed. The accuracy and training/ classification time of each algorithm using a dense stimulation array and two coarse stimulation arrays are presented and compared. The next focus of the thesis is to develop non-invasive tactile display. The design and psychophysical testing results of three types of non-invasive tactile feedback arrays are presented: two with vibrotactile modality and one with multi modality. For vibrotactile, two types of miniaturized vibrators: eccentric rotating masses (ERMs) and linear resonant actuators (LRAs) were first tested on healthy subjects and their effectiveness was compared. Then the ERMs are integrated into a vibrotactile glove to assess the feasibility of providing sensory feedback for unilateral upper limb amputees on the contralateral hand. For multimodal stimulation, miniature multimodal actuators integrating servomotors and vibrators were designed. The actuator can be used to deliver both high-frequency vibration and low-frequency pressures simultaneously. By utilizing two modalities at the same time, the actuator stimulates different types of mechanoreceptors and thus h

Ultra Low Power FM-UWB Transceiver for High-Density Wireless Sensor Networks

The WiseSkin project aims to provide a non-invasive solution for restoration of a natural sense of touch to persons using prosthetic limbs. By embedding sensor nodes into the silicone coating of the prosthesis, which acts as a sensory skin, WiseSkin targets to provide improved gripping, manipulation and mobility for amputees. Flexibility, freedom of movement and comfort demand unobtrusive, highly miniaturized, low-power sensing capabilities built into the artificial skin, which is then integrated with a sensory feedback system. Wireless communication between the sensor nodes provides more flexibility, better scalability and robustness compared to wired solution, and is therefore a preferred approach for WiseSkin. Design of an RF transceiver tailored for the specific needs of WiseSkin is the topic of this work. The properties of FM ultra-wide band (FM-UWB) modulation make it a good candidate for High-Density Wireless Sensor Networks (HD-WSN). The proposed FM-UWB receivers take advantage of short range to reduce power consumption, and exploit robustness of this wideband modulation scheme. The LNA, identified as the biggest consumer, is removed and signal is directly converted to dc, where amplification and demodulation are performed. Owing to 500 MHz bandwidth, frequency offset and phase noise can be tolerated, and a low-power, free-running ring oscillator can be used to generate the LO signal. The receiver is referred to as an approximate zero-IF receiver. Two receiver architectures are studied. The first one performs quadrature downconversion, and owing to the demodulator linearity, provides the multi-user capability. In the second receiver, quadrature demodulation is replaced by the single-ended one. Due to the nature of the demodulator, sensitivity degrades, and multiple FM-UWB signals cannot be resolved, but the consumption is almost halved compared to the first receiver. The proposed approach is verified through two integrations, both in a standard 65 nm bulk CMOS process. In the first run, a standalone quadrature receiver was integrated. Power consumption of 423 uW was measured, while achieving -70 dBm sensitivity. Good narrow-band interference rejection and multiuser capability with up to 4 FM-UWB channels could be achieved. In the second run, a full transceiver is integrated, with both quadrature and single-ended receivers and a transmitter, all sharing a single IO pad, without the need for any external passive components or switches. The quadrature receiver, with on-chip baseband processing and multi-user support, in this case consumes 550 uW, with a sesensitivity of -68 dBm. The low power receiver consumes 267 uW, and provides -57 dBm sensitivity, at a single FM-UWB channel. The implemented trantransmitter transmits a 100 kb/s FM-UWB signal at -11.4 dBm, while drawing 583 uW from the 1 V supply. The on-chip clock recovery allows reference frequency offset up to 8000 ppm. Since state of the art on-chip RC oscillators can provide below 2100 ppm across the temperature range of interest, the implemented transceiver demonstrates the feasibility of a fully integrated FM-UWB radio with no need for a quartz reference or any external components. In addition, the transceiver can tolerate up to 3 dBm narrow-band interferer at 2.4 GHz. Such a strong signal can be used to remotely power the sensor nodes inside the artificial skin and enable a truly wirelessWiseSkin solution

Stretchable metallization technologies for skin-like transducers

The skin is not only the largest human organ, capable of accomplishing distributed and multimodal sensing functions. Replicating the versatility of skin artificially is a significant challenge, not only in terms of signal processing but also in mechanics. Stretchable electronics are an approach designed to cover human and artificial limbs and provide wearable sensing capabilities: motion sensors distributed on the hand of neurologically impaired patients could help therapists quantify their abilities; prostheses equipped with multiple tactile sensors could enable amputees to naturally adjust their grasp force. Skin-like electronic systems have specific requirements: they must mechanically adapt to the deformations imposed by the body they equip with minimal impediment to its natural movements, while also providing sufficient electrical performance for sensor transduction and passing electrical signals and power. A metallization ensuring stable conductivity under large strains is a prerequisite to designing and assembling wearable circuits that are integrated with several types of sensors. In this work, two innovative metallization processes have been developed to enable scalable integration of multiple sensing modalities in stretchable circuits. First, stretchable micro-cracked gold (Au) thin films were interfaced with gallium indium eutectic (EGaIn) liquid metal wires. The Au films, thermally evaporated on silicone elastomer substrates, combined high sheet resistance (9 to 30 Ohm/sq) and high sensitivity to strain up to 50%. The EGaIn wires drawn using a micro-plotting setup had a low gauge factor (2) and a low sheet resistance (5 mOhm/sq). Second, a novel physical vapor deposition method to deposit of thin gallium-based biphasic (solid-liquid) films over large areas was achieved. The obtained conductors combined a low sheet resistance (0.5 Ohm/sq), a low gauge factor (~1 up to 80% strain), and a failure strain of more than 400%. They could be patterned down to 10 µm critical dimensions. Skin-like sensors for the hand were assembled using the two processes and their capabilities were demonstrated. Thin (0.5 mm) silicone strips integrating EGaIN wires and micro-cracked Au strain gauges were mounted on gloves to encode the position of a biomimetic robotic finger and a human finger. In combination with soft pressure sensors, they enabled precise grasp analysis over a limited range of motion. Then, biphasic films were micro-patterned on silicone to assemble 50 µm thin epidermal strain gauges. The strain gauges were attached on a user's finger and accurately encoded fine grasping tasks covering most of the human hand range of motion. The biphasic films were also used to power wireless MEMS pressure sensors integrated in a rubber scaffold. The device was mounted on a prosthetic hand to encode normal forces in the 0 N to 20 N range with excellent linearity. The epidermal strain sensors are currently being used to quantify the tremors of patients with Parkinson's disease. In the future, the unique properties of the biphasic films could enable advanced artificial skins integrating a high density of soft transducers and traditional high-performance circuits

A system for electrotactile feedback using electronic skin and flexible matrix electrodes: Experimental evaluation

Myoelectric prostheses are successfully controlled using muscle electrical activity, thereby restoring lost motor functions. However, the somatosensory feedback from the prosthesis to the user is still missing. The sensory substitution methods described in the literature comprise mostly simple position and force sensors combined with discrete stimulation units. The present study describes a novel system for sophisticated electrotactile feedback integrating advanced distributed sensing (electronic skin) and stimulation (matrix electrodes). The system was tested in eight healthy subjects who were asked to recognize the shape, trajectory, and direction of a set of dynamic movement patterns (single lines, geometrical objects, letters) presented on the electronic skin. The experiments demonstrated that the system successfully translated the mechanical interaction into the moving electrotactile profiles, which the subjects could recognize with a good performance (shape recognition: 86±8% lines, 73±13% geometries, 72±12% letters). In particular, the subjects could identify the movement direction with a high confidence. These results are in accordance with previous studies investigating the recognition of moving stimuli in human subjects. This is an important development towards closed-loop prostheses providing comprehensive and sophisticated tactile feedback to the user, facilitating the control and the embodiment of the artificial device into the user body scheme

nano-tera.ch: Electronic Technology for Health Management

The Swiss Nano-Tera.ch program addresses – among others – issues at the crossing of engineering and medical domains. Specifically, electronic-health (or E-Health) is a broad area of engineering that leverages transducer, circuit and systems technologies for applications to health management and lifestyle. Scientific challenges relate to the acquisition of accurate medical information from various forms of sensing inside/outside the body and to the processing of this information to support or actuate medical decisions. E-health is motivated by the social and economic goals of achieving better health care at lower costs and will revolutionize medical practice in the years to come

Full-hand electrotactile feedback using electronic skin and matrix electrodes for high-bandwidth human–machine interfacing

Tactile feedback is relevant in a broad range of human–machine interaction systems (e.g. teleoperation, virtual reality and prosthetics). The available tactile feedback interfaces comprise few sensing and stimulation units, which limits the amount of information conveyed to the user. The present study describes a novel technology that relies on distributed sensing and stimulation to convey comprehensive tactile feedback to the user of a robotic end effector. The system comprises six flexible sensing arrays (57 sensors) integrated on the fingers and palm of a robotic hand, embedded electronics (64 recording channels), a multichannel stimulator and seven flexible electrodes (64 stimulation pads) placed on the volar side of the subject’s hand. The system was tested in seven subjects asked to recognize contact positions and identify contact sliding on the electronic skin, using distributed anode configuration (DAC) and single dedicated anode configuration. The experiments demonstrated that DAC resulted in substantially better performance. Using DAC, the system successfully translated the contact patterns into electrotactile profiles that the subjects could recognize with satisfactory accuracy (i.e. median{IQR} of 88.6{11}% for static and 93.3{5}% for dynamic patterns). The proposed system is an important step towards the development of a high-density human–machine interfacing between the user and a robotic han

Automatic hand phantom map generation and detection using decomposition support vector machines

Background: There is a need for providing sensory feedback for myoelectric prosthesis users. Providing tactile feedback can improve object manipulation abilities, enhance the perceptual embodiment of myoelectric prostheses and help reduce phantom limb pain. Many amputees have referred sensation from their missing hand on their residual limbs (phantom maps). This skin area can serve as a target for providing amputees with non-invasive tactile sensory feedback. One of the challenges of providing sensory feedback on the phantom map is to define the accurate boundary of each phantom digit because the phantom map distribution varies from person to person. Methods: In this paper, automatic phantom map detection methods based on four decomposition support vector machine algorithms and three sampling methods are proposed, complemented by fuzzy logic and active learning strategies. The algorithms and methods are tested on two databases: the first one includes 400 generated phantom maps, whereby the phantom map generation algorithm was based on our observation of the phantom maps to ensure smooth phantom digit edges, variety, and representativeness. The second database includes five reported phantom map images and transformations thereof. The accuracy and training/ classification time of each algorithm using a dense stimulation array (with 100 $$\times$$ × 100 actuators) and two coarse stimulation arrays (with 3 $$\times$$ × 5 and 4 $$\times$$ × 6 actuators) are presented and compared. Results: Both generated and reported phantom map images share the same trends. Majority-pooling sampling effectively increases the training size, albeit introducing some noise, and thus produces the smallest error rates among the three proposed sampling methods. For different decomposition architectures, one-vs-one reduces unclassified regions and in general has higher classification accuracy than the other architectures. By introducing fuzzy logic to bias the penalty parameter, the influence of pooling-induced noise is reduced. Moreover, active learning with different strategies was also tested and shown to improve the accuracy by introducing more representative training samples. Overall, dense arrays employing one-vs-one fuzzy support vector machines with majority-pooling sampling have the smallest average absolute error rate (8.78% for generated phantom maps and 11.5% for reported and transformed phantom map images). The detection accuracy of coarse arrays was found to be significantly lower than for dense array. Conclusions: The results demonstrate the effectiveness of support vector machines using a dense array in detecting refined phantom map shapes, whereas coarse arrays are unsuitable for this task. We therefore propose a two-step approach, using first a non-wearable dense array to detect an accurate phantom map shape, then to apply a wearable coarse stimulation array customized according to the detection results. The proposed methodology can be used as a tool for helping haptic feedback designers and for tracking the evolvement of phantom maps

Routing for Wireless Sensor Networks: From Collection to Event-Triggered Applications

Wireless Sensor Networks (WSNs) are collections of sensing devices using wireless communication to exchange data. In the past decades, steep advancements in the areas of microelectronics and communication systems have driven an explosive growth in the deployment of WSNs. Novel WSN applications have penetrated multiple areas, from monitoring the structural stability of historic buildings, to tracking animals in order to understand their behavior, or monitoring humans' health. The need to convey data from increasingly complex applications in a reliable and cost-effective manner translates into stringent performance requirements for the underlying WSNs. In the frame of this thesis, we have focused on developing routing protocols for multi-hop WSNs, that significantly improve their reliability, energy consumption and latency. Acknowledging the need for application-specific trade-offs, we have split our contribution into two parts. Part 1 focuses on collection protocols, catering to applications with high reliability and energy efficiency constraints, while the protocols developed in part 2 are subject to an additional bounded latency constraint. The two mechanisms introduced in the first part, WiseNE and Rep, enable the use of composite metrics, and thus significantly improve the link estimation accuracy and transmission reliability, at an energy expense far lower than the one achieved in previous proposals. The novel beaconing scheme WiseNE enables the energy-efficient addition of the RSSI (Received Signal Strength Indication) and LQI (Link Quality Indication) metrics to the link quality estimate by decoupling the sampling and exploration periods of each mote. This decoupling allows the use of the Trickle Algorithm, a key driver of protocols' energy efficiency, in conjunction with composite metrics. WiseNE has been applied to the Triangle Metric and validated in an online deployment. The section continues by introducing Rep, a novel sampling mechanism that leverages the packet repetitions already present in low-power preamble-sampling MAC protocols in order to improve the WSN energy consumption by one order of magnitude. WiseNE, Rep and the novel PRSSI (Penalized RSSI, a combination of PRR and RSSI) composite metric have been validated in a real smart city deployment. Part 2 introduces two mechanisms that were developed in the frame of the WiseSkin project (an initiative aimed at designing highly sensitive artificial skin for human limb prostheses), and are generally applicable to the domain of cyber-physical systems. It starts with Glossy-W, a protocol that leverages the superior energy-latency trade-off of flooding schemes based on concurrent transmissions. Glossy-W ensures the stringent synchronization requirements necessary for robust flooding, irrespective of the number of motes simultaneously reporting an event. Part 2 also introduces SCS (Synchronized Channel Sampling), a novel mechanism capable of reducing the power required for periodic polling, while maintaining the event detection reliability, and enhancing the network coexistence. The testbed experiments performed show that SCS manages to reduce the energy consumption of the state-of-the-art protocol Back-to-Back Robust Flooding by over one third, while maintaining an equivalent reliability, and remaining compatible with simultaneous event detection. SCS' benefits can be extended to the entire family of state-of-the-art protocols relying on concurrent transmissions

Distributed Sensing and Stimulation Systems Towards Sense of Touch Restoration in Prosthetics

Modern prostheses aim at restoring the functional and aesthetic characteristics of the lost limb. To foster prosthesis embodiment and functionality, it is necessary to restitute both volitional control and sensory feedback. Contemporary feedback interfaces presented in research use few sensors and stimulation units to feedback at most two discrete feedback variables (e.g. grasping force and aperture), whereas the human sense of touch relies on a distributed network of mechanoreceptors providing high-fidelity spatial information. To provide this type of feedback in prosthetics, it is necessary to sense tactile information from artificial skin placed on the prosthesis and transmit tactile feedback above the amputation in order to map the interaction between the prosthesis and the environment. This thesis proposes the integration of distributed sensing systems (e-skin) to acquire tactile sensation, and non-invasive multichannel electrotactile feedback and virtual reality to deliver high-bandwidth information to the user. Its core focus addresses the development and testing of close-loop sensory feedback human-machine interface, based on the latest distributed sensing and stimulation techniques for restoring the sense of touch in prosthetics. To this end, the thesis is comprised of two introductory chapters that describe the state of art in the field, the objectives and the used methodology and contributions; as well as three studies distributed over stimulation system level and sensing system level. The first study presents the development of close-loop compensatory tracking system to evaluate the usability and effectiveness of electrotactile sensory feedback in enabling real-time close-loop control in prosthetics. It examines and compares the subject\u2019s adaptive performance and tolerance to random latencies while performing the dynamic control task (i.e. position control) and simultaneously receiving either visual feedback or electrotactile feedback for communicating the momentary tracking error. Moreover, it reported the minimum time delay needed for an abrupt impairment of users\u2019 performance. The experimental results have shown that electrotactile feedback performance is less prone to changes with longer delays. However, visual feedback drops faster than electrotactile with increased time delays. This is a good indication for the effectiveness of electrotactile feedback in enabling close- loop control in prosthetics, since some delays are inevitable. The second study describes the development of a novel non-invasive compact multichannel interface for electrotactile feedback, containing 24 pads electrode matrix, with fully programmable stimulation unit, that investigates the ability of able-bodied human subjects to localize the electrotactile stimulus delivered through the electrode matrix. Furthermore, it designed a novel dual parameter -modulation (interleaved frequency and intensity) and compared it to conventional stimulation (same frequency for all pads). In addition and for the first time, it compared the electrotactile stimulation to mechanical stimulation. More, it exposes the integration of virtual prosthesis with the developed system in order to achieve better user experience and object manipulation through mapping the acquired real-time collected tactile data and feedback it simultaneously to the user. The experimental results demonstrated that the proposed interleaved coding substantially improved the spatial localization compared to same-frequency stimulation. Furthermore, it showed that same-frequency stimulation was equivalent to mechanical stimulation, whereas the performance with dual-parameter modulation was significantly better. The third study presents the realization of a novel, flexible, screen- printed e-skin based on P(VDF-TrFE) piezoelectric polymers, that would cover the fingertips and the palm of the prosthetic hand (particularly the Michelangelo hand by Ottobock) and an assistive sensorized glove for stroke patients. Moreover, it developed a new validation methodology to examine the sensors behavior while being solicited. The characterization results showed compatibility between the expected (modeled) behavior of the electrical response of each sensor to measured mechanical (normal) force at the skin surface, which in turn proved the combination of both fabrication and assembly processes was successful. This paves the way to define a practical, simplified and reproducible characterization protocol for e-skin patches In conclusion, by adopting innovative methodologies in sensing and stimulation systems, this thesis advances the overall development of close-loop sensory feedback human-machine interface used for restoration of sense of touch in prosthetics. Moreover, this research could lead to high-bandwidth high-fidelity transmission of tactile information for modern dexterous prostheses that could ameliorate the end user experience and facilitate it acceptance in the daily life

Electronic systems for the restoration of the sense of touch in upper limb prosthetics

In the last few years, research on active prosthetics for upper limbs focused on improving the human functionalities and the control. New methods have been proposed for measuring the user muscle activity and translating it into the prosthesis control commands. Developing the feed-forward interface so that the prosthesis better follows the intention of the user is an important step towards improving the quality of life of people with limb amputation. However, prosthesis users can neither feel if something or someone is touching them over the prosthesis and nor perceive the temperature or roughness of objects. Prosthesis users are helped by looking at an object, but they cannot detect anything otherwise. Their sight gives them most information. Therefore, to foster the prosthesis embodiment and utility, it is necessary to have a prosthetic system that not only responds to the control signals provided by the user, but also transmits back to the user the information about the current state of the prosthesis. This thesis presents an electronic skin system to close the loop in prostheses towards the restoration of the sense of touch in prosthesis users. The proposed electronic skin system inlcudes an advanced distributed sensing (electronic skin), a system for (i) signal conditioning, (ii) data acquisition, and (iii) data processing, and a stimulation system. The idea is to integrate all these components into a myoelectric prosthesis. Embedding the electronic system and the sensing materials is a critical issue on the way of development of new prostheses. In particular, processing the data, originated from the electronic skin, into low- or high-level information is the key issue to be addressed by the embedded electronic system. Recently, it has been proved that the Machine Learning is a promising approach in processing tactile sensors information. Many studies have been shown the Machine Learning eectiveness in the classication of input touch modalities.More specically, this thesis is focused on the stimulation system, allowing the communication of a mechanical interaction from the electronic skin to prosthesis users, and the dedicated implementation of algorithms for processing tactile data originating from the electronic skin. On system level, the thesis provides design of the experimental setup, experimental protocol, and of algorithms to process tactile data. On architectural level, the thesis proposes a design ow for the implementation of digital circuits for both FPGA and integrated circuits, and techniques for the power management of embedded systems for Machine Learning algorithms