Addressing the challenges posed by human machine interfaces based on force sensitive resistors for powered prostheses

Despite the advancements in the mechatronics aspect of prosthetic devices, prostheses control still lacks an interface that satisfies the needs of the majority of users. The research community has put great effort into the advancements of prostheses control techniques to address users’ needs. However, most of these efforts are focused on the development and assessment of technologies in the controlled environments of laboratories. Such findings do not fully transfer to the daily application of prosthetic systems. The objectives of this thesis focus on factors that affect the use of Force Myography (FMG) controlled prostheses in practical scenarios. The first objective of this thesis assessed the use of FMG as an alternative or synergist Human Machine Interface (HMI) to the more traditional HMI, i.e. surface Electromyography (sEMG). The assessment for this study was conducted in conditions that are relatively close to the real use case of prosthetic applications. The HMI was embedded in the custom prosthetic prototype that was developed for the pilot participant of the study using an off-the-shelf prosthetic end effector. Moreover, prostheses control was assessed as the user moved their limb in a dynamic protocol.The results of the aforementioned study motivated the second objective of this thesis: to investigate the possibility of reducing the complexity of high density FMG systems without sacrificing classification accuracies. This was achieved through a design method that uses a high density FMG apparatus and feature selection to determine the number and location of sensors that can be eliminated without significantly sacrificing the system’s performance. The third objective of this thesis investigated two of the factors that contribute to increased errors in force sensitive resistor (FSR) signals used in FMG controlled prostheses: bending of force sensors and variations in the volume of the residual limb. Two studies were conducted that proposed solutions to mitigate the negative impact of these factors. The incorporation of these solutions into prosthetic devices is discussed in these studies.It was demonstrated that FMG is a promising HMI for prostheses control. The facilitation of pattern recognition with FMG showed potential for intuitive prosthetic control. Moreover, a method for the design of a system that can determine the required number of sensors and their locations on each individual to achieve a simpler system with comparable performance to high density FMG systems was proposed and tested. The effects of the two factors considered in the third objective were determined. It was also demonstrated that the proposed solutions in the studies conducted for this objective can be used to increase the accuracy of signals that are commonly used in FMG controlled prostheses

Optimization of Forcemyography Sensor Placement for Arm Movement Recognition

How to design an optimal wearable device for human movement recognition is vital to reliable and accurate human-machine collaboration. Previous works mainly fabricate wearable devices heuristically. Instead, this paper raises an academic question: can we design an optimization algorithm to optimize the fabrication of wearable devices such as figuring out the best sensor arrangement automatically? Specifically, this work focuses on optimizing the placement of Forcemyography (FMG) sensors for FMG armbands in the application of arm movement recognition. Firstly, based on graph theory, the armband is modeled considering sensors' signals and connectivity. Then, a Graph-based Armband Modeling Network (GAM-Net) is introduced for arm movement recognition. Afterward, the sensor placement optimization for FMG armbands is formulated and an optimization algorithm with greedy local search is proposed. To study the effectiveness of our optimization algorithm, a dataset for mechanical maintenance tasks using FMG armbands with 16 sensors is collected. Our experiments show that using only 4 sensors optimized with our algorithm can help maintain a comparable recognition accuracy to using all sensors. Finally, the optimized sensor placement result is verified from a physiological view. This work would like to shed light on the automatic fabrication of wearable devices considering downstream tasks, such as human biological signal collection and movement recognition. Our code and dataset are available at https://github.com/JerryX1110/IROS22-FMG-Sensor-OptimizationComment: 6 pages, 10 figures, Accepted by IROS22 (The 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS

Towards electrodeless EMG linear envelope signal recording for myo-activated prostheses control

After amputation, the residual muscles of the limb may function in a normal way, enabling the electromyogram (EMG) signals recorded from them to be used to drive a replacement limb. These replacement limbs are called myoelectric prosthesis. The prostheses that use EMG have always been the first choice for both clinicians and engineers. Unfortunately, due to the many drawbacks of EMG (e.g. skin preparation, electromagnetic interferences, high sample rate, etc.); researchers have aspired to find suitable alternatives. One proposes the dry-contact, low-cost sensor based on a force-sensitive resistor (FSR) as a valid alternative which instead of detecting electrical events, detects mechanical events of muscle. FSR sensor is placed on the skin through a hard, circular base to sense the muscle contraction and to acquire the signal. Similarly, to reduce the output drift (resistance) caused by FSR edges (creep) and to maintain the FSR sensitivity over a wide input force range, signal conditioning (Voltage output proportional to force) is implemented. This FSR signal acquired using FSR sensor can be used directly to replace the EMG linear envelope (an important control signal in prosthetics applications). To find the best FSR position(s) to replace a single EMG lead, the simultaneous recording of EMG and FSR output is performed. Three FSRs are placed directly over the EMG electrodes, in the middle of the targeted muscle and then the individual (FSR1, FSR2 and FSR3) and combination of FSR (e.g. FSR1+FSR2, FSR2-FSR3) is evaluated. The experiment is performed on a small sample of five volunteer subjects. The result shows a high correlation (up to 0.94) between FSR output and EMG linear envelope. Consequently, the usage of the best FSR sensor position shows the ability of electrode less FSR-LE to proportionally control the prosthesis (3-D claw). Furthermore, FSR can be used to develop a universal programmable muscle signal sensor that can be suitable to control the myo-activated prosthesis

Force Myography-Based Human Robot Interactions via Deep Domain Adaptation and Generalization

Estimating applied force using force myography (FMG) technique can be effective in human-robot interactions (HRI) using data-driven models. A model predicts well when adequate training and evaluation are observed in same session, which is sometimes time consuming and impractical. In real scenarios, a pretrained transfer learning model predicting forces quickly once fine-tuned to target distribution would be a favorable choice and hence needs to be examined. Therefore, in this study a unified supervised FMG-based deep transfer learner (SFMG-DTL) model using CNN architecture was pretrained with multiple sessions FMG source data (Ds, Ts) and evaluated in estimating forces in separate target domains (Dt, Tt) via supervised domain adaptation (SDA) and supervised domain generalization (SDG). For SDA, case (i) intra-subject evaluation (Ds ≠ Dt-SDA, Ts ≈ Tt-SDA) was examined, while for SDG, case (ii) cross-subject evaluation (Ds ≠ Dt-SDG, Ts ≠ Tt-SDG) was examined. Fine tuning with few “target training data” calibrated the model effectively towards target adaptation. The proposed SFMG-DTL model performed better with higher estimation accuracies and lower errors (R2 ≥ 88%, NRMSE ≤ 0.6) in both cases. These results reveal that interactive force estimations via transfer learning will improve daily HRI experiences where “target training data” is limited, or faster adaptation is require

Toward Long-Term FMG Model-Based Estimation of Applied Hand Force in Dynamic Motion During Human–Robot Interactions

Physical human-robot interaction (pHRI) is reliant on human actions and can be addressed by studying human upper-limb motions during interactions. Use of force myography (FMG) signals, which detect muscle contractions, can be useful in developing machine learning algorithms as controls. In this paper, a novel long-term calibrated FMG-based trained model is presented to estimate applied force in dynamic motion during real-time interactions between a human and a linear robot. The proposed FMG-based pHRI framework was investigated in new, unseen, real-time scenarios for the first time. Initially, a long-term reference dataset (multiple source distributions) of upper-limb FMG data was generated as five participants interacted with the robot applying force in five different dynamic motions. Ten other participants interacted with the robot in two intended motions to evaluate the out-of-distribution (OOD) target data (new, unlearned), which was different than the population data. Two practical scenarios were considered for assessment: i) a participant applied force in a new, unlearned motion (scenario 1), and ii) a new, unlearned participant applied force in an intended motion (scenario 2). In each scenario, few long-term FMG-based models were trained using a baseline dataset [reference dataset (scenario 1, 2) and/or a learnt participant dataset (scenario 1)] and a calibration dataset (collected during evaluation). Real-time evaluation showed that the proposed long-term calibrated FMG-based models (LCFMG) could achieve estimation accuracies of 80%-94% in all scenarios. These results are useful towards integrating and generalizing human activity data in a robot control scheme by avoiding extensive HRI training phase in regular applications

Continuous Myoelectric Prediction of Future Ankle Angle and Moment Across Ambulation Conditions and Their Transitions

A hallmark of human locomotion is that it continuously adapts to changes in the environment and predictively adjusts to changes in the terrain, both of which are major challenges to lower limb amputees due to the limitations in prostheses and control algorithms. Here, the ability of a single-network nonlinear autoregressive model to continuously predict future ankle kinematics and kinetics simultaneously across ambulation conditions using lower limb surface electromyography (EMG) signals was examined. Ankle plantarflexor and dorsiflexor EMG from ten healthy young adults were mapped to normal ranges of ankle angle and ankle moment during level overground walking, stair ascent, and stair descent, including transitions between terrains (i.e., transitions to/from staircase). Prediction performance was characterized as a function of the time between current EMG/angle/moment inputs and future angle/moment model predictions (prediction interval), the number of past EMG/angle/moment input values over time (sampling window), and the number of units in the network hidden layer that minimized error between experimentally measured values (targets) and model predictions of ankle angle and moment. Ankle angle and moment predictions were robust across ambulation conditions with root mean squared errors less than 1° and 0.04 Nm/kg, respectively, and cross-correlations (R2) greater than 0.99 for prediction intervals of 58 ms. Model predictions at critical points of trip-related fall risk fell within the variability of the ankle angle and moment targets (Benjamini-Hochberg adjusted p \u3e 0.065). EMG contribution to ankle angle and moment predictions occurred consistently across ambulation conditions and model outputs. EMG signals had the greatest impact on noncyclic regions of gait such as double limb support, transitions between terrains, and around plantarflexion and moment peaks. The use of natural muscle activation patterns to continuously predict variations in normal gait and the model’s predictive capabilities to counteract electromechanical inherent delays suggest that this approach could provide robust and intuitive user-driven real-time control of a wide variety of lower limb robotic devices, including active powered ankle-foot prostheses