Fused mechanomyography and inertial measurement for human-robot interface

Human-Machine Interfaces (HMI) are the technology through which we interact with the ever-increasing quantity of smart devices surrounding us. The fundamental goal of an HMI is to facilitate robot control through uniting a human operator as the supervisor with a machine as the task executor. Sensors, actuators, and onboard intelligence have not reached the point where robotic manipulators may function with complete autonomy and therefore some form of HMI is still necessary in unstructured environments. These may include environments where direct human action is undesirable or infeasible, and situations where a robot must assist and/or interface with people. Contemporary literature has introduced concepts such as body-worn mechanical devices, instrumented gloves, inertial or electromagnetic motion tracking sensors on the arms, head, or legs, electroencephalographic (EEG) brain activity sensors, electromyographic (EMG) muscular activity sensors and camera-based (vision) interfaces to recognize hand gestures and/or track arm motions for assessment of operator intent and generation of robotic control signals. While these developments offer a wealth of future potential their utility has been largely restricted to laboratory demonstrations in controlled environments due to issues such as lack of portability and robustness and an inability to extract operator intent for both arm and hand motion. Wearable physiological sensors hold particular promise for capture of human intent/command. EMG-based gesture recognition systems in particular have received significant attention in recent literature. As wearable pervasive devices, they offer benefits over camera or physical input systems in that they neither inhibit the user physically nor constrain the user to a location where the sensors are deployed. Despite these benefits, EMG alone has yet to demonstrate the capacity to recognize both gross movement (e.g. arm motion) and finer grasping (e.g. hand movement). As such, many researchers have proposed fusing muscle activity (EMG) and motion tracking e.g. (inertial measurement) to combine arm motion and grasp intent as HMI input for manipulator control. However, such work has arguably reached a plateau since EMG suffers from interference from environmental factors which cause signal degradation over time, demands an electrical connection with the skin, and has not demonstrated the capacity to function out of controlled environments for long periods of time. This thesis proposes a new form of gesture-based interface utilising a novel combination of inertial measurement units (IMUs) and mechanomyography sensors (MMGs). The modular system permits numerous configurations of IMU to derive body kinematics in real-time and uses this to convert arm movements into control signals. Additionally, bands containing six mechanomyography sensors were used to observe muscular contractions in the forearm which are generated using specific hand motions. This combination of continuous and discrete control signals allows a large variety of smart devices to be controlled. Several methods of pattern recognition were implemented to provide accurate decoding of the mechanomyographic information, including Linear Discriminant Analysis and Support Vector Machines. Based on these techniques, accuracies of 94.5% and 94.6% respectively were achieved for 12 gesture classification. In real-time tests, accuracies of 95.6% were achieved in 5 gesture classification. It has previously been noted that MMG sensors are susceptible to motion induced interference. The thesis also established that arm pose also changes the measured signal. This thesis introduces a new method of fusing of IMU and MMG to provide a classification that is robust to both of these sources of interference. Additionally, an improvement in orientation estimation, and a new orientation estimation algorithm are proposed. These improvements to the robustness of the system provide the first solution that is able to reliably track both motion and muscle activity for extended periods of time for HMI outside a clinical environment. Application in robot teleoperation in both real-world and virtual environments were explored. With multiple degrees of freedom, robot teleoperation provides an ideal test platform for HMI devices, since it requires a combination of continuous and discrete control signals. The field of prosthetics also represents a unique challenge for HMI applications. In an ideal situation, the sensor suite should be capable of detecting the muscular activity in the residual limb which is naturally indicative of intent to perform a specific hand pose and trigger this post in the prosthetic device. Dynamic environmental conditions within a socket such as skin impedance have delayed the translation of gesture control systems into prosthetic devices, however mechanomyography sensors are unaffected by such issues. There is huge potential for a system like this to be utilised as a controller as ubiquitous computing systems become more prevalent, and as the desire for a simple, universal interface increases. Such systems have the potential to impact significantly on the quality of life of prosthetic users and others.Open Acces

Sensor-Based Adaptive Control and Optimization of Lower-Limb Prosthesis.

Recent developments in prosthetics have enabled the development of powered prosthetic ankles (PPA). The advent of such technologies drastically improved impaired gait by increasing balance and reducing metabolic energy consumption by providing net positive power. However, control challenges limit performance and feasibility of today’s devices. With addition of sensors and motors, PPA systems should continuously make control decisions and adapt the system by manipulating control parameters of the prostheses. There are multiple challenges in optimization and control of PPAs. A prominent challenge is the objective setup of the system and calibration parameters to fit each subject. Another is whether it is possible to detect changes in intention and terrain before prosthetic use and how the system should react and adapt to it. In the first part of this study, a model for energy expenditure was proposed using electromyogram (EMG) signals from the residual lower-limbs PPA users. The proposed model was optimized to minimize energy expenditure. Optimization was performed using a modified Nelder-Mead approach with a Latin Hypercube sampling. Results of the proposed method were compared to expert values and it was shown to be a feasible alternative for tuning in a shorter time. In the second part of the study, the control challenges regarding lack of adaptivity for PPAs was investigated. The current PPA system used is enhanced with impedance-controlled parameters that allow the system to provide different assistance. However, current systems are set to a fixed value and fail to acknowledge various terrain and intentions throughout the day. In this study, a pseudo-real-time adaptive control system was proposed to predict the changes in the gait and provide a smoother gait. The proposed control system used physiological, kinetic, and kinematic data and fused them to predict the change. The prediction was done using machine learning-based methods. Results of the study showed an accuracy of up to 89.7 percent for prediction of change for four different cases

Entry Guidance for the 2011 Mars Science Laboratory Mission

The 2011 Mars Science Laboratory will be the first Mars mission to attempt a guided entry to safely deliver the rover to a touchdown ellipse of 25 km x 20 km. The Entry Terminal Point Controller guidance algorithm is derived from the final phase Apollo Command Module guidance and, like Apollo, modulates the bank angle to control the range flown. For application to Mars landers which must make use of the tenuous Martian atmosphere, it is critical to balance the lift of the vehicle to minimize the range error while still ensuring a safe deploy altitude. An overview of the process to generate optimized guidance settings is presented, discussing improvements made over the last nine years. Key dispersions driving deploy ellipse and altitude performance are identified. Performance sensitivities including attitude initialization error and the velocity of transition from range control to heading alignment are presented

SANE: The phases of gradient descent through Sharpness Adjusted Number of Effective parameters

Modern neural networks are undeniably successful. Numerous studies have investigated how the curvature of loss landscapes can affect the quality of solutions. In this work we consider the Hessian matrix during network training. We reiterate the connection between the number of "well-determined" or "effective" parameters and the generalisation performance of neural nets, and we demonstrate its use as a tool for model comparison. By considering the local curvature, we propose Sharpness Adjusted Number of Effective parameters (SANE), a measure of effective dimensionality for the quality of solutions. We show that SANE is robust to large learning rates, which represent learning regimes that are attractive but (in)famously unstable. We provide evidence and characterise the Hessian shifts across "loss basins" at large learning rates. Finally, extending our analysis to deeper neural networks, we provide an approximation to the full-network Hessian, exploiting the natural ordering of neural weights, and use this approximation to provide extensive empirical evidence for our claims

On the Utility of Representation Learning Algorithms for Myoelectric Interfacing

Electrical activity produced by muscles during voluntary movement is a reflection of the firing patterns of relevant motor neurons and, by extension, the latent motor intent driving the movement. Once transduced via electromyography (EMG) and converted into digital form, this activity can be processed to provide an estimate of the original motor intent and is as such a feasible basis for non-invasive efferent neural interfacing. EMG-based motor intent decoding has so far received the most attention in the field of upper-limb prosthetics, where alternative means of interfacing are scarce and the utility of better control apparent. Whereas myoelectric prostheses have been available since the 1960s, available EMG control interfaces still lag behind the mechanical capabilities of the artificial limbs they are intended to steer—a gap at least partially due to limitations in current methods for translating EMG into appropriate motion commands. As the relationship between EMG signals and concurrent effector kinematics is highly non-linear and apparently stochastic, finding ways to accurately extract and combine relevant information from across electrode sites is still an active area of inquiry.This dissertation comprises an introduction and eight papers that explore issues afflicting the status quo of myoelectric decoding and possible solutions, all related through their use of learning algorithms and deep Artificial Neural Network (ANN) models. Paper I presents a Convolutional Neural Network (CNN) for multi-label movement decoding of high-density surface EMG (HD-sEMG) signals. Inspired by the successful use of CNNs in Paper I and the work of others, Paper II presents a method for automatic design of CNN architectures for use in myocontrol. Paper III introduces an ANN architecture with an appertaining training framework from which simultaneous and proportional control emerges. Paper Iv introduce a dataset of HD-sEMG signals for use with learning algorithms. Paper v applies a Recurrent Neural Network (RNN) model to decode finger forces from intramuscular EMG. Paper vI introduces a Transformer model for myoelectric interfacing that do not need additional training data to function with previously unseen users. Paper vII compares the performance of a Long Short-Term Memory (LSTM) network to that of classical pattern recognition algorithms. Lastly, paper vIII describes a framework for synthesizing EMG from multi-articulate gestures intended to reduce training burden

Design and Development of Atraumatic Vacuum Assisted Delivery Devices

Vacuum-Assisted Delivery (VAD) is an obstetric practice used to assist child delivery during the second stage of labour. During the procedure, the obstetric professional attaches the VAD device to the scalp of the foetus through suction and tractive force is then applied alongside maternal contractions to assist the baby’s passage through the delivery channel. VAD is more prevalent than obstetric forceps due to its ease of use, lower maternal morbidity and improved cosmetic outcome for the mother and her baby. However, safety concerns such as unintentional cup detachment or high vacuum, can lead to induced trauma to the foetus. Since its original inception, there have been limited efforts to evaluate the safety of VAD devices or optimise their design and operation. Here, an engineering approach to assess the devices’ failure modes is proposed to inform training, best obstetric practice and improved VAD design. An instrumented experimental recreation of VAD has been developed to achieve a comprehensive understanding of the mechanics of VAD devices and the associated trauma. It features an instrumented adaptation of a commercially available VAD device (the Kiwi® Omnicup™) connected to a tensile testing machine to simulate obstetric traction onto a head scalp model (fabricated using textile reinforced silicone). A pneumatic control system provides an actively controlled vacuum to the instrumented device. Optical markers, placed onto the scalp model, combined with a high-speed camera system provide tracking of scalp deformation during the mechanical simulation of an obstetric traction. Experimental factors such as traction speed, magnitude of vacuum imposed & changes to the design geometry of the VAD cup and pneumatic architecture including the consideration of frictional attributes of the maternal environment, were investigated. The results from the experimental studies show that a simulated obstetric VAD traction produces a characteristic response from which a number of key clinically relevant metrics can be determined and highlight the association of clinical factors and mechanical factors to device performance. The research informed on the conception of an atraumatic concept to prevent cup detachment. Upon evaluation of the technical and commercial feasibility of the concept, commercial and research opportunities were identified, which could help improve the performance of VAD devices, in the future

Assist-as-needed EMG-based control strategy for wearable powered assistive devices

Dissertação de mestrado integrado em Engenharia Biomédica (área de especialização em Eletrónica Médica)Robotic-based gait rehabilitation and assistance using Wearable Powered Assistive Devices (WPADs), such as orthosis and exoskeletons, has been growing in the rehabilitation area to recover and augment the motor function of neurologically impaired subjects. These WPADs should provide a personalized assistance, since physical condition and muscular fatigue modify from patient to patient. In this field, electromyography (EMG) signals have been used to control WPADs given their ability to infer the user’s motion intention. However, in cases of motor disability conditions, EMG signals present lower magnitudes when compared to EMG signals under healthy conditions. Thus, the use of WPADs managed by EMG signals may not have potential to provide the assistance that the patient requires. The main goal of this dissertation aims the development of an Assisted-As-Needed (AAN) EMG-based control strategy for a future insertion in a Smart Active Orthotic System (SmartOs). To achieve this goal, the following elements were developed and validated: (i) an EMG system to acquire muscle activity signals from the most relevant muscles during the motion of the ankle joint; (ii) machine learning-based tool for ankle joint torque estimation to serve as reference in the AAN EMG-based control strategy; and (iii) a tool for real EMG-based torque estimation using Tibialis Anterior (TA) and Gastrocnemius Lateralis (GASL) muscles and real ankle joint angles. EMG system showed satisfactory pattern correlations with a commercial system. The reference ankle joint torque was generated based on predicted reference ankle joint kinematics, walking speed information (from 1 to 4 km/h) and anthropometric data (body height from 1.51 m to 1.83 m and body mass from 52.0 kg to 83.7 kg), using five machine learning algorithms: Support Vector Regression (SVR), Random Forest (RF), Multilayer Perceptron (MLP), Long-Short Term Memory (LSTM) and Convolutional Neural Network (CNN). CNN provided the best performance, predicting the reference ankle joint torque with fitting curves ranging from 74.7 to 89.8 % and Normalized Root Mean Square Errors (NRMSEs) between 3.16 and 8.02 %. EMG-based torque estimation beneficiates of a higher number of muscles, since EMG data from TA and GASL are not enough to estimate the real ankle joint torque.A assistência e reabilitação robótica usando dispositivos de assistência ativos vestíveis (WPADs), como ortóteses e exosqueletos, tem crescido na área da reabilitação com o fim de recuperar e aumentar a função motora de sujeitos com alterações neurológicas. Estes dispositivos devem fornecer uma assistência personalizada, uma vez que a condição física e a fadiga muscular variam de paciente para paciente. Nesta área, sinais de eletromiografia (EMG) têm sido usados para controlar WPADs, dada a sua capacidade de inferir a intenção de movimento do utilizador. Contudo, em casos de deficiência motora, os sinais de EMG apresentam menor amplitude quando comparados com sinais de EMG em condições saudáveis e, portanto, o uso de WPADs geridos por sinais de EMG pode não oferecer a assistência que o paciente necessita. O principal objetivo desta dissertação visa o desenvolvimento de uma estratégia de controlo baseada em EMG capaz de fornecer assistência quando necessário, para futura integração num sistema ortótico ativo e inteligente (SmartOs). Para atingir este objetivo foram desenvolvidos e validados os seguintes elementos: (i) sistema de EMG para adquirir sinais de atividade muscular dos músculos mais relevantes no movimento da articulação do tornozelo; (ii) ferramenta de machine learning para estimação do binário da articulação do tornozelo para servir como referência na estratégia de controlo; e (iii) ferramenta de estimação do binário real do tornozelo considerando sinais de EMG dos músculos Tibialis Anterior (TA) e Gastrocnemius Lateralis (GASL) e ângulo real do tornozelo. O sistema de EMG apresentou correlações satisfatórias com um sistema comercial. O binário de referência para o tornozelo foi gerado com base no ângulo de referência da mesma articulação, velocidade de marcha (de 1 até 4 km/h) e dados antropométricos (alturas de 1.51 m até 1.83 e massas de 52.0 kg até 83.7 kg), usando cinco algoritmos de machine learning: Support Vector Machine, Random Forest, Multilayer Perceptron, Long-Short Term Memory e Convolutional Neural Network. CNN apresentou a melhor performance, prevendo binários de referência do tornozelo com um fit entre 74.7 e 89.8 % e Normalized Root Mean Square Errors (NRMSE) entre 3.16 e 8.02 %. A estimativa do torque com base em sinais de EMG requer a inclusão de um maior número de músculos, uma vez que sinais de EMG dos músculos TA e GASL não foram suficientes