Tactile signatures and hand motion intent recognition for wearable assistive devices

Within the field of robotics and autonomous systems where there is a human in the loop, intent recognition plays an important role. This is especially true for wearable assistive devices used for rehabilitation, particularly post-stroke recovery. This paper reports results on the use of tactile patterns to detect weak muscle contractions in the forearm while at the same time associating these patterns with the muscle synergies during different grips. To investigate this concept, a series of experiments with healthy participants were carried out using a tactile arm brace (TAB) on the forearm while performing four different types of grip. The expected force patterns were established by analysing the muscle synergies of the four grip types and the forearm physiology. The results showed that the tactile signatures of the forearm recorded on the TAB align with the anticipated force patterns. Furthermore, a linear separability of the data across all four grip types was identified. Using the TAB data, machine learning algorithms achieved a 99% classification accuracy. The TAB results were highly comparable to a similar commercial intent recognition system based on a surface electromyography (sEMG) sensing

Towards electrodeless EMG linear envelope signal recording for myo-activated prostheses control

After amputation, the residual muscles of the limb may function in a normal way, enabling the electromyogram (EMG) signals recorded from them to be used to drive a replacement limb. These replacement limbs are called myoelectric prosthesis. The prostheses that use EMG have always been the first choice for both clinicians and engineers. Unfortunately, due to the many drawbacks of EMG (e.g. skin preparation, electromagnetic interferences, high sample rate, etc.); researchers have aspired to find suitable alternatives. One proposes the dry-contact, low-cost sensor based on a force-sensitive resistor (FSR) as a valid alternative which instead of detecting electrical events, detects mechanical events of muscle. FSR sensor is placed on the skin through a hard, circular base to sense the muscle contraction and to acquire the signal. Similarly, to reduce the output drift (resistance) caused by FSR edges (creep) and to maintain the FSR sensitivity over a wide input force range, signal conditioning (Voltage output proportional to force) is implemented. This FSR signal acquired using FSR sensor can be used directly to replace the EMG linear envelope (an important control signal in prosthetics applications). To find the best FSR position(s) to replace a single EMG lead, the simultaneous recording of EMG and FSR output is performed. Three FSRs are placed directly over the EMG electrodes, in the middle of the targeted muscle and then the individual (FSR1, FSR2 and FSR3) and combination of FSR (e.g. FSR1+FSR2, FSR2-FSR3) is evaluated. The experiment is performed on a small sample of five volunteer subjects. The result shows a high correlation (up to 0.94) between FSR output and EMG linear envelope. Consequently, the usage of the best FSR sensor position shows the ability of electrode less FSR-LE to proportionally control the prosthesis (3-D claw). Furthermore, FSR can be used to develop a universal programmable muscle signal sensor that can be suitable to control the myo-activated prosthesis

Fused mechanomyography and inertial measurement for human-robot interface

Human-Machine Interfaces (HMI) are the technology through which we interact with the ever-increasing quantity of smart devices surrounding us. The fundamental goal of an HMI is to facilitate robot control through uniting a human operator as the supervisor with a machine as the task executor. Sensors, actuators, and onboard intelligence have not reached the point where robotic manipulators may function with complete autonomy and therefore some form of HMI is still necessary in unstructured environments. These may include environments where direct human action is undesirable or infeasible, and situations where a robot must assist and/or interface with people. Contemporary literature has introduced concepts such as body-worn mechanical devices, instrumented gloves, inertial or electromagnetic motion tracking sensors on the arms, head, or legs, electroencephalographic (EEG) brain activity sensors, electromyographic (EMG) muscular activity sensors and camera-based (vision) interfaces to recognize hand gestures and/or track arm motions for assessment of operator intent and generation of robotic control signals. While these developments offer a wealth of future potential their utility has been largely restricted to laboratory demonstrations in controlled environments due to issues such as lack of portability and robustness and an inability to extract operator intent for both arm and hand motion. Wearable physiological sensors hold particular promise for capture of human intent/command. EMG-based gesture recognition systems in particular have received significant attention in recent literature. As wearable pervasive devices, they offer benefits over camera or physical input systems in that they neither inhibit the user physically nor constrain the user to a location where the sensors are deployed. Despite these benefits, EMG alone has yet to demonstrate the capacity to recognize both gross movement (e.g. arm motion) and finer grasping (e.g. hand movement). As such, many researchers have proposed fusing muscle activity (EMG) and motion tracking e.g. (inertial measurement) to combine arm motion and grasp intent as HMI input for manipulator control. However, such work has arguably reached a plateau since EMG suffers from interference from environmental factors which cause signal degradation over time, demands an electrical connection with the skin, and has not demonstrated the capacity to function out of controlled environments for long periods of time. This thesis proposes a new form of gesture-based interface utilising a novel combination of inertial measurement units (IMUs) and mechanomyography sensors (MMGs). The modular system permits numerous configurations of IMU to derive body kinematics in real-time and uses this to convert arm movements into control signals. Additionally, bands containing six mechanomyography sensors were used to observe muscular contractions in the forearm which are generated using specific hand motions. This combination of continuous and discrete control signals allows a large variety of smart devices to be controlled. Several methods of pattern recognition were implemented to provide accurate decoding of the mechanomyographic information, including Linear Discriminant Analysis and Support Vector Machines. Based on these techniques, accuracies of 94.5% and 94.6% respectively were achieved for 12 gesture classification. In real-time tests, accuracies of 95.6% were achieved in 5 gesture classification. It has previously been noted that MMG sensors are susceptible to motion induced interference. The thesis also established that arm pose also changes the measured signal. This thesis introduces a new method of fusing of IMU and MMG to provide a classification that is robust to both of these sources of interference. Additionally, an improvement in orientation estimation, and a new orientation estimation algorithm are proposed. These improvements to the robustness of the system provide the first solution that is able to reliably track both motion and muscle activity for extended periods of time for HMI outside a clinical environment. Application in robot teleoperation in both real-world and virtual environments were explored. With multiple degrees of freedom, robot teleoperation provides an ideal test platform for HMI devices, since it requires a combination of continuous and discrete control signals. The field of prosthetics also represents a unique challenge for HMI applications. In an ideal situation, the sensor suite should be capable of detecting the muscular activity in the residual limb which is naturally indicative of intent to perform a specific hand pose and trigger this post in the prosthetic device. Dynamic environmental conditions within a socket such as skin impedance have delayed the translation of gesture control systems into prosthetic devices, however mechanomyography sensors are unaffected by such issues. There is huge potential for a system like this to be utilised as a controller as ubiquitous computing systems become more prevalent, and as the desire for a simple, universal interface increases. Such systems have the potential to impact significantly on the quality of life of prosthetic users and others.Open Acces

Understanding Gesture Expressivity through Muscle Sensing

Expressivity is a visceral capacity of the human body. To understand what makes a gesture expressive, we need to consider not only its spatial placement and orientation, but also its dynamics and the mechanisms enacting them. We start by defining gesture and gesture expressivity, and then present fundamental aspects of muscle activity and ways to capture information through electromyography (EMG) and mechanomyography (MMG). We present pilot studies that inspect the ability of users to control spatial and temporal variations of 2D shapes and that use muscle sensing to assess expressive information in gesture execution beyond space and time. This leads us to the design of a study that explores the notion of gesture power in terms of control and sensing. Results give insights to interaction designers to go beyond simplistic gestural interaction, towards the design of interactions that draw upon nuances of expressive gesture

Assistive Device For Elderly Rehabilitation: Signal Processing Techniques.

Ph.DDOCTOR OF PHILOSOPH

The selection and evaluation of a sensory technology for interaction in a warehouse environment

In recent years, Human-Computer Interaction (HCI) has become a significant part of modern life as it has improved human performance in the completion of daily tasks in using computerised systems. The increase in the variety of bio-sensing and wearable technologies on the market has propelled designers towards designing more efficient, effective and fully natural User-Interfaces (UI), such as the Brain-Computer Interface (BCI) and the Muscle-Computer Interface (MCI). BCI and MCI have been used for various purposes, such as controlling wheelchairs, piloting drones, providing alphanumeric inputs into a system and improving sports performance. Various challenges are experienced by workers in a warehouse environment. Because they often have to carry objects (referred to as hands-full) it is difficult to interact with traditional devices. Noise undeniably exists in some industrial environments and it is known as a major factor that causes communication problems. This has reduced the popularity of using verbal interfaces with computer applications, such as Warehouse Management Systems. Another factor that effects the performance of workers are action slips caused by a lack of concentration during, for example, routine picking activities. This can have a negative impact on job performance and allow a worker to incorrectly execute a task in a warehouse environment. This research project investigated the current challenges workers experience in a warehouse environment and the technologies utilised in this environment. The latest automation and identification systems and technologies are identified and discussed, specifically the technologies which have addressed known problems. Sensory technologies were identified that enable interaction between a human and a computerised warehouse environment. Biological and natural behaviours of humans which are applicable in the interaction with a computerised environment were described and discussed. The interactive behaviours included the visionary, auditory, speech production and physiological movement where other natural human behaviours such paying attention, action slips and the action of counting items were investigated. A number of modern sensory technologies, devices and techniques for HCI were identified with the aim of selecting and evaluating an appropriate sensory technology for MCI. iii MCI technologies enable a computer system to recognise hand and other gestures of a user, creating means of direct interaction between a user and a computer as they are able to detect specific features extracted from a specific biological or physiological activity. Thereafter, Machine Learning (ML) is applied in order to train a computer system to detect these features and convert them to a computer interface. An application of biomedical signals (bio-signals) in HCI using a MYO Armband for MCI is presented. An MCI prototype (MCIp) was developed and implemented to allow a user to provide input to an HCI, in a hands-free and hands-full situation. The MCIp was designed and developed to recognise the hand-finger gestures of a person when both hands are free or when holding an object, such a cardboard box. The MCIp applies an Artificial Neural Network (ANN) to classify features extracted from the surface Electromyography signals acquired by the MYO Armband around the forearm muscle. The MCIp provided the results of data classification for gesture recognition to an accuracy level of 34.87% with a hands-free situation. This was done by employing the ANN. The MCIp, furthermore, enabled users to provide numeric inputs to the MCIp system hands-full with an accuracy of 59.7% after a training session for each gesture of only 10 seconds. The results were obtained using eight participants. Similar experimentation with the MYO Armband has not been found to be reported in any literature at submission of this document. Based on this novel experimentation, the main contribution of this research study is a suggestion that the application of a MYO Armband, as a commercially available muscle-sensing device on the market, has the potential as an MCI to recognise the finger gestures hands-free and hands-full. An accurate MCI can increase the efficiency and effectiveness of an HCI tool when it is applied to different applications in a warehouse where noise and hands-full activities pose a challenge. Future work to improve its accuracy is proposed

Advanced Augmentative and Alternative Communication System Based in Physiological Control

Dyskinetic Cerebral Palsy (DCP) is mainly characterized by alterations in muscle tone and involuntary movements. Therefore, these people present with difficulties in coordination and movement control, which makes walking difficult and affects their posture when seated. Additionally, their cognitive performance varies between being completely normal and severe mental retardation. People with DCP were selected as the objective of this thesis due to their multiple and complex limitations (speech problems and motor control) and because their capabilities have a great margin for improvement thanks to physiological control systems. Given their communication difficulties, some people with DCP have good motor con-trol and can communicate with written language. However, most have difficulty using Augmentative and Alternative Communication (AAC) systems. People with DCP gen-erally use concept boards to indicate the idea they want to communicate. However, most communication solutions available today are based on proprietary software that makes it difficult to customize the concept board and this type of control system. This is the motivation behind this thesis, with the aim of creating an interface with characteristics, able to be adapted to the user needs and limitations. Thus, this thesis proposes an Augmentative and Alternative Communication System for people with DCP based on physiological control. In addition, an innovative system for direct con-trol of concept boards with EMG is proposed. This control system is based on a physi-cal model that reproduces the muscular mechanical response (stiffness, inertia and viscosity). It allows for a selection of elements thanks to small pulses of EMG signal with sensors on a muscle with motor control. Its main advantage is the possibility of correcting errors during selection associated with uncontrolled muscle impulses, avoid-ing sustained muscle effort and thus reduced fatigue.La Parálisis Cerebral de tipo Discinésica (DCP) se caracteriza principalmente por las alteraciones del tono muscular y los movimientos involuntarios. Por ello, estos pacientes presentan dificultades en la coordinación y en el control de movimientos, lo cual les dificulta el caminar y afecta su postura cuando están sentados. Cabe resaltar que la capacidad cognitiva de las personas con DCP puede variar desde completamente normal, hasta un retraso mental severo. Las personas con DCP han sido seleccionadas como objetivo de esta tesis ya el margen de mejora de sus capacidades es amplio gracias a sistemas de control fisiológico, debido a sus múltiples y complejas limitaciones (problemas de habla y control motor). Debido a sus dificultades de comunicación, algunas personas con DCP se pueden comunicar con lenguaje escrito, siempre y cuando tenga un buen control motor. Sin embargo, la mayoría tienen dificultades para usar sistemas de Comunicación Aumentativos y Alternativos (AAC). De hecho, las personas con DCP utilizan generalmente tableros de conceptos para indicar la idea que quieren transmitir. Sin embargo, la mayoría las soluciones de comunicación disponibles en la actualidad están basadas en software propietario que hacen difícil la personalización del tablero de conceptos y el tipo de sistema de control. Es aquí donde surge esta tesis, con el objetivo de crear una interfaz con esas características, capaz de adaptarse a las necesidades y limitaciones del usuario. De esta forma, esta tesis propone un sistema de comunicación aumentativo y alternativo para personas con DCP basado en control fisiológico. Además, se propone un Sistema innovador de control directo sobre tableros de conceptos basado en EMG. Este Sistema de control se basa en un modelo físico que reproduce la respuesta mecánica muscular (basado en parámetros como Rigidez, Inercia y Viscosidad), permitiendo la selección de elementos gracias a pequeños pulsos de señal EMG con sensores sobre un músculo con control motor. Sus principales ventajas son la posibilidad de corregir errores durante la selección asociado a los impulsos musculares no controlados, evitar el esfuerzo muscular mantenido para alcanzar un nivel y reducir la fatiga.La Paràlisi Cerebral de tipus Discinèsica (DCP) es caracteritza principalment per les alteracions del to muscular i els moviments involuntaris. Per açò, aquests pacients presenten dificultats en la coordinació i en el control de moviments, la qual cosa els dificulta el caminar i afecta la seua postura quan estan asseguts. Cal ressaltar que la capacitat cognitiva de les persones amb DCP pot variar des de completament normal, fins a un retard mental sever. Les persones amb DCP han sigut seleccionades com a objectiu d'aquesta tesi ja el marge de millora de les seues capacitats és ampli gràcies a sistemes de control fisiològic, a causa dels seus múltiples i complexes limitacions (problemes de parla i control motor). A causa de les seues dificultats de comunicació, algunes persones amb DCP es poden comunicar amb llenguatge escrit, sempre que tinga un bon control motor. No obstant açò, la majoria tenen dificultats per a usar sistemes de Comunicació Augmentatius i Alternatius (AAC). De fet, les persones amb DCP utilitzen generalment taulers de conceptes per a indicar la idea que volen transmetre. No obstant açò, la majoria les solucions de comunicació disponibles en l'actualitat estan basades en programari propietari que fan difícil la personalització del tauler de conceptes i el tipus de sistema de control. És ací on sorgeix aquesta tesi, amb l'objectiu de crear una interfície amb aqueixes característiques, capaç d'adaptar-se a les necessitats i limitacions de l'usuari. D'aquesta forma, aquesta tesi proposa un sistema de comunicació augmentatiu i alternatiu per a persones amb DCP basat en control fisiològic. A més, es proposa un sistema innovador de control directe sobre taulers de conceptes basat en EMG. Aquest sistema de control es basa en un model físic que reprodueix la resposta mecànica muscular (basat en paràmetres com a Rigidesa, Inèrcia i Viscositat), permetent la selecció d'elements gràcies a xicotets polsos de senyal EMG amb sensors sobre un múscul amb control motor. Els seus principals avantatges són la possibilitat de corregir errors durant la selecció associat als impulsos musculars no controlats, evitar l'esforç muscular mantingut per a aconseguir un nivell i reduir la fatiga.Díaz Pineda, JA. (2017). Advanced Augmentative and Alternative Communication System Based in Physiological Control [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/90418TESI

Design of a low-cost sensor matrix for use in human-machine interactions on the basis of myographic information

Myographic sensor matrices in the field of human-machine interfaces are often poorly developed and not pushing the limits in terms of a high spatial resolution. Many studies use sensor matrices as a tool to access myographic data for intention prediction algorithms regardless of the human anatomy and used sensor principles. The necessity for more sophisticated sensor matrices in the field of myographic human-machine interfaces is essential, and the community already called out for new sensor solutions. This work follows the neuromechanics of the human and designs customized sensor principles to acquire the occurring phenomena. Three low-cost sensor modalities Electromyography, Mechanomyography, and Force Myography) were developed in a miniaturized size and tested in a pre-evaluation study. All three sensors comprise the characteristic myographic information of its modality. Based on the pre-evaluated sensors, a sensor matrix with 32 exchangeable and high-density sensor modules was designed. The sensor matrix can be applied around the human limbs and takes the human anatomy into account. A data transmission protocol was customized for interfacing the sensor matrix to the periphery with reduced wiring. The designed sensor matrix offers high-density and multimodal myographic information for the field of human-machine interfaces. Especially the fields of prosthetics and telepresence can benefit from the higher spatial resolution of the sensor matrix