Viia-hand: a Reach-and-grasp Restoration System Integrating Voice interaction, Computer vision and Auditory feedback for Blind Amputees

Visual feedback plays a crucial role in the process of amputation patients completing grasping in the field of prosthesis control. However, for blind and visually impaired (BVI) amputees, the loss of both visual and grasping abilities makes the "easy" reach-and-grasp task a feasible challenge. In this paper, we propose a novel multi-sensory prosthesis system helping BVI amputees with sensing, navigation and grasp operations. It combines modules of voice interaction, environmental perception, grasp guidance, collaborative control, and auditory/tactile feedback. In particular, the voice interaction module receives user instructions and invokes other functional modules according to the instructions. The environmental perception and grasp guidance module obtains environmental information through computer vision, and feedbacks the information to the user through auditory feedback modules (voice prompts and spatial sound sources) and tactile feedback modules (vibration stimulation). The prosthesis collaborative control module obtains the context information of the grasp guidance process and completes the collaborative control of grasp gestures and wrist angles of prosthesis in conjunction with the user's control intention in order to achieve stable grasp of various objects. This paper details a prototyping design (named viia-hand) and presents its preliminary experimental verification on healthy subjects completing specific reach-and-grasp tasks. Our results showed that, with the help of our new design, the subjects were able to achieve a precise reach and reliable grasp of the target objects in a relatively cluttered environment. Additionally, the system is extremely user-friendly, as users can quickly adapt to it with minimal training

New developments in prosthetic arm systems

Absence of an upper limb leads to severe impairments in everyday life, which can further influence the social and mental state. For these reasons, early developments in cosmetic and body-driven prostheses date some centuries ago, and they have been evolving ever since. Following the end of the Second World War, rapid developments in technology resulted in powered myoelectric hand prosthetics. In the years to come, these devices were common on the market, though they still suffered high user abandonment rates. The reasons for rejection were trifold - insufficient functionality of the hardware, fragile design, and cumbersome control. In the last decade, both academia and industry have reached major improvements concerning technical features of upper limb prosthetics and methods for their interfacing and control. Advanced robotic hands are offered by several vendors and research groups, with a variety of active and passive wrist options that can be articulated across several degrees of freedom. Nowadays, elbow joint designs include active solutions with different weight and power options. Control features are getting progressively more sophisticated, offering options for multiple sensor integration and multi-joint articulation. Latest developments in socket designs are capable of facilitating implantable and multiple surface electromyography sensors in both traditional and osseointegration-based systems. Novel surgical techniques in combination with modern, sophisticated hardware are enabling restoration of dexterous upper limb functionality. This article is aimed at reviewing the latest state of the upper limb prosthetic market, offering insights on the accompanying technologies and techniques. We also examine the capabilities and features of some of academia’s flagship solutions and methods

Simulation And Control At the Boundaries Between Humans And Assistive Robots

Human-machine interaction has become an important area of research as progress is made in the fields of rehabilitation robotics, powered prostheses, and advanced exercise machines. Adding to the advances in this area, a novel controller for a powered transfemoral prosthesis is introduced that requires limited tuning and explicitly considers energy regeneration. Results from a trial conducted with an individual with an amputation show self-powering operation for the prosthesis while concurrently attaining basic gait fidelity across varied walking speeds. Experience in prosthesis development revealed that, though every effort is made to ensure the safety of the human subject, limited testing of such devices prior to human trials can be completed in the current research environment. Two complementary alternatives are developed to fill that gap. First, the feasibility of implementing impulse-momentum sliding mode control on a robot that can physically replace a human with a transfemoral amputation to emulate weight-bearing for initial prototype walking tests is established. Second, a more general human simulation approach is proposed that can be used in any of the aforementioned human-machine interaction fields. Seeking this general human simulation method, a unique pair of solutions for simulating a Hill muscle-actuated linkage system is formulated. These include using the Lyapunov-based backstepping control method to generate a closed-loop tracking simulation and, motivated by limitations observed in backstepping, an optimal control solver based on differential flatness and sum of squares polynomials in support of receding horizon controlled (e.g. model predictive control) or open-loop simulations. v The backstepping framework provides insight into muscle redundancy resolution. The optimal control framework uses this insight to produce a computationally efficient approach to musculoskeletal system modeling. A simulation of a human arm is evaluated in both structures. Strong tracking performance is achieved in the backstepping case. An exercise optimization application using the optimal control solver showcases the computational benefits of the solver and reveals the feasibility of finding trajectories for human-exercise machine interaction that can isolate a muscle of interest for strengthening

Neuro-Musculoskeletal Mapping for Man-Machine Interfacing.

We propose a myoelectric control method based on neural data regression and musculoskeletal modeling. This paradigm uses the timings of motor neuron discharges decoded by high-density surface electromyogram (HD-EMG) decomposition to estimate muscle excitations. The muscle excitations are then mapped into the kinematics of the wrist joint using forward dynamics. The offline tracking performance of the proposed method was superior to that of state-of-the-art myoelectric regression methods based on artificial neural networks in two amputees and in four out of six intact-bodied subjects. In addition to joint kinematics, the proposed data-driven model-based approach also estimated several biomechanical variables in a full feed-forward manner that could potentially be useful in supporting the rehabilitation and training process. These results indicate that using a full forward dynamics musculoskeletal model directly driven by motor neuron activity is a promising approach in rehabilitation and prosthetics to model the series of transformations from muscle excitation to resulting joint function

Development of an Embedded Myokinetic Prosthetic Hand Controller

The quest for an intuitive and physiologically appropriate human machine interface for the control of dexterous prostheses is far from being completed. In the last decade, much effort has been dedicated to explore innovative control strategies based on the electrical signals generated by the muscles during contraction. In contrast, a novel approach, dubbed myokinetic interface, derives the control signals from the localization of multiple magnetic markers (MMs) directly implanted into the residual muscles of the amputee. Building on this idea, here we present an embedded system based on 32 magnetic field sensors and a real time computation platform. We demonstrate that the platform can simultaneously localize in real-time up to five MMs in an anatomically relevant workspace. The system proved highly linear (R2 = 0.99) and precise (1% repeatability), yet exhibiting short computation times (4 ms) and limited cross talk errors (10% the mean stroke of the magnets). Compared to a previous PC implementation, the system exhibited similar precision and accuracy, while being ~75% faster. These results proved for the first time the viability of using an embedded system for magnet localization. They also suggest that, by using an adequate number of sensors, it is possible to increase the number of simultaneously tracked MMs while introducing delays that are not perceivable by the human operator. This could allow to control more degrees of freedom than those controllable with current technologies

Robust Electromyography Based Control of Multifunctional Prostheses of The Upper Extremity

Multifunctional, highly dexterous and complex mechanic hand prostheses are emerging and currently entering the market. However, the bottleneck to fully exploiting all capabilities of these mechatronic devices, and to making all available functions controllable reliably and intuitively by the users, remains a considerable challenge. The robustness of scientific methods proposed to overcome this barrier is a crucial factor for their future commercial success. Therefore, in this thesis the matter of robust, multifunctional and dexterous control of prostheses of the upper limb was addressed and some significant advancements in the scientific field were aspired. To this end, several investigations grouped in four studies were conducted, all with the same focus on understanding mechanisms that influence the robustness of myoelectric control and resolving their deteriorating effects. For the first study, a thorough literature review of the field was conducted and it was revealed that many non-stationarities, which could be expected to affect the reliability of surface EMG pattern recognition myoprosthesis control, had been identified and studied previously. However, one significant factor had not been addressed to a sufficient extent: the effect of long-term usage and day-to-day testing. Therefore, a dedicated study was designed and carried out, in order to address the previously unanswered question of how reliable surface electromyography pattern recognition was across days. Eleven subjects, involving both able-bodied and amputees, participated in this study over the course of 5 days, and a pattern recognition system was tested without daily retraining. As the main result of this study, it was revealed that the time between training and testing a classifier was indeed a very relevant factor influencing the classification accuracy. More estimation errors were observed as more time lay between the classifier training and testing. With the insights obtained from the first study, the need for compensating signal non-stationarities was identified. Hence, in a second study, building up on the data obtained from the first investigation, a self-correction mechanism was elaborated. The goal of this approach was to increase the systems robustness towards non-stationarities such as those identified in the first study. The system was capable of detecting and correcting its own mistakes, yielding a better estimation of movements than the uncorrected classification or other, previously proposed strategies for error removal. In the third part of this thesis, the previously investigated ideas for error suppression for increased robustness of a classification based system were extended to regression based movement estimation. While the same method as tested in the second study was not directly applicable to regression, the same underlying idea was used for developing a novel proportional estimator. It was validated in online tests, with the control of physical prostheses by able-bodied and transradial amputee subjects. The proposed method, based on common spatial patterns, outperformed two state-of-the art control methods, demonstrating the benefit of increased robustness in movement estimation during applied tasks. The results showed the superior performance of robust movement estimation in real life investigations, which would have hardly been observable in offline or abstract cursor control tests, underlining the importance of tests with physical prostheses. In the last part of this work, the limitation of sequential movements of the previously explored system was addressed and a methodology for enhancing the system with simultaneous and proportional control was developed. As a result of these efforts, a system robust, natural and fluent in its movements was conceived. Again, online control tests of physical prostheses were performed by able-bodied and amputee subjects, and the novel system proved to outperform the sequential controller of the third study of this thesis, yielding the best control technique tested. An extensive set of tests was conducted with both able-bodied and amputee subjects, in scenarios close to clinical routine. Custom prosthetic sockets were manufactured for all subjects, allowing for experimental control of multifunction prostheses with advanced machine learning based algorithms in real-life scenarios. The tests involved grasping and manipulating objects, in ways as they are often encountered in everyday living. Similar investigations had not been conducted before. One of the main conclusions of this thesis was that the suppression of wrong prosthetic motions was a key factor for robust prosthesis control and that simultaneous wrist control was a beneficial asset especially for experienced users. As a result of all investigations performed, clinically relevant conclusions were drawn from these tests, maximizing the impact of the developed systems on potential future commercialization of the newly conceived control methods. This was emphasized by the close collaboration with Otto Bock as an industrial partner of the AMYO project and hence this work.2016-02-2