Intelligent upper-limb exoskeleton using deep learning to predict human intention for sensory-feedback augmentation

The age and stroke-associated decline in musculoskeletal strength degrades the ability to perform daily human tasks using the upper extremities. Although there are a few examples of exoskeletons, they need manual operations due to the absence of sensor feedback and no intention prediction of movements. Here, we introduce an intelligent upper-limb exoskeleton system that uses cloud-based deep learning to predict human intention for strength augmentation. The embedded soft wearable sensors provide sensory feedback by collecting real-time muscle signals, which are simultaneously computed to determine the user's intended movement. The cloud-based deep-learning predicts four upper-limb joint motions with an average accuracy of 96.2% at a 200-250 millisecond response rate, suggesting that the exoskeleton operates just by human intention. In addition, an array of soft pneumatics assists the intended movements by providing 897 newton of force and 78.7 millimeter of displacement at maximum. Collectively, the intent-driven exoskeleton can augment human strength by 5.15 times on average compared to the unassisted exoskeleton. This report demonstrates an exoskeleton robot that augments the upper-limb joint movements by human intention based on a machine-learning cloud computing and sensory feedback.Comment: 15 pages, 6 figures, 1 table, Submitted for possible publicatio

Haptic Feedback in Virtual Reality: An Investigation Into The Next Step of First Person Perspective Presence

Video games are becoming progressively sophisticated with new interesting mechanics and increasingly realistic graphics. Game technologies manufacturers are constantly striving to find innovative ways of providing additional layers of interactivity, and engagement with the player. In video games haptic feedback has traditionally been delivered by motors and pulleys through interfaces such as steering wheels and joysticks, or via a simple vibration mechanism in the controllers. However, while the growing popularity of commercial virtual reality technologies has provided video game developers with a new modality to introduce greater levels of immersion and presence into games, haptic technology in gaming has kept to its traditional roots. In this thesis we investigate the impact that haptic feedback has on player presence within virtual reality environments. We introduce a non-intrusive haptic interface that can be used alongside consumer grade virtual reality technology. This thesis will demonstrate the implementation and technical considerations made during the construction of this device. We then demonstrate the systems effectiveness through a user study evaluating users reactions towards the system when compared with traditional vibration-based haptics and with the absence of any feedback, in a virtual reality game environment. The results from this study show a positive impact on player presence when using the non-intrusive haptic device, with broken down presence scores suggesting the device was successful in delivering a satisfying haptic experience. Results also indicate an improvement in the way participants perceive their own performance when using the device, with presence scores suggesting this is due to participants being able to fully place themselves in the experience

A Sensorized Instrument for Minimally Invasive Surgery for the Measurement of Forces during Training and Surgery: Development and Applications

The reduced access conditions present in Minimally Invasive Surgery (MIS) affect the feel of interaction forces between the instruments and the tissue being treated. This loss of haptic information compromises the safety of the procedure and must be overcome through training. Haptics in MIS is the subject of extensive research, focused on establishing force feedback mechanisms and developing appropriate sensors. This latter task is complicated by the need to place the sensors as close as possible to the instrument tip, as the measurement of forces outside of the patient\u27s body does not represent the true tool--tissue interaction. Many force sensors have been proposed, but none are yet available for surgery. The objectives of this thesis were to develop a set of instruments capable of measuring tool--tissue force information in MIS, and to evaluate the usefulness of force information during surgery and for training and skills assessment. To address these objectives, a set of laparoscopic instruments was developed that can measure instrument position and tool--tissue interaction forces in multiple degrees of freedom. Different design iterations and the work performed towards the development of a sterilizable instrument are presented. Several experiments were performed using these instruments to establish the usefulness of force information in surgery and training. The results showed that the combination of force and position information can be used in the development of realistic tissue models or haptic interfaces specifically designed for MIS. This information is also valuable in order to create tactile maps to assist in the identification of areas of different stiffness. The real-time measurement of forces allows visual force feedback to be presented to the surgeon. When applied to training scenarios, the results show that experience level correlates better with force-based metrics than those currently used in training simulators. The proposed metrics can be automatically computed, are completely objective, and measure important aspects of performance. The primary contribution of this thesis is the design and development of highly versatile instruments capable of measuring force and position during surgery. A second contribution establishes the importance and usefulness of force data during skills assessment, training and surgery

Human skill capturing and modelling using wearable devices

Industrial robots are delivering more and more manipulation services in manufacturing. However, when the task is complex, it is difficult to programme a robot to fulfil all the requirements because even a relatively simple task such as a peg-in-hole insertion contains many uncertainties, e.g. clearance, initial grasping position and insertion path. Humans, on the other hand, can deal with these variations using their vision and haptic feedback. Although humans can adapt to uncertainties easily, most of the time, the skilled based performances that relate to their tacit knowledge cannot be easily articulated. Even though the automation solution may not fully imitate human motion since some of them are not necessary, it would be useful if the skill based performance from a human could be firstly interpreted and modelled, which will then allow it to be transferred to the robot. This thesis aims to reduce robot programming efforts significantly by developing a methodology to capture, model and transfer the manual manufacturing skills from a human demonstrator to the robot. Recently, Learning from Demonstration (LfD) is gaining interest as a framework to transfer skills from human teacher to robot using probability encoding approaches to model observations and state transition uncertainties. In close or actual contact manipulation tasks, it is difficult to reliabley record the state-action examples without interfering with the human senses and activities. Therefore, wearable sensors are investigated as a promising device to record the state-action examples without restricting the human experts during the skilled execution of their tasks. Firstly to track human motions accurately and reliably in a defined 3-dimensional workspace, a hybrid system of Vicon and IMUs is proposed to compensate for the known limitations of the individual system. The data fusion method was able to overcome occlusion and frame flipping problems in the two camera Vicon setup and the drifting problem associated with the IMUs. The results indicated that occlusion and frame flipping problems associated with Vicon can be mitigated by using the IMU measurements. Furthermore, the proposed method improves the Mean Square Error (MSE) tracking accuracy range from 0.8˚ to 6.4˚ compared with the IMU only method. Secondly, to record haptic feedback from a teacher without physically obstructing their interactions with the workpiece, wearable surface electromyography (sEMG) armbands were used as an indirect method to indicate contact feedback during manual manipulations. A muscle-force model using a Time Delayed Neural Network (TDNN) was built to map the sEMG signals to the known contact force. The results indicated that the model was capable of estimating the force from the sEMG armbands in the applications of interest, namely in peg-in-hole and beater winding tasks, with MSE of 2.75N and 0.18N respectively. Finally, given the force estimation and the motion trajectories, a Hidden Markov Model (HMM) based approach was utilised as a state recognition method to encode and generalise the spatial and temporal information of the skilled executions. This method would allow a more representative control policy to be derived. A modified Gaussian Mixture Regression (GMR) method was then applied to enable motions reproduction by using the learned state-action policy. To simplify the validation procedure, instead of using the robot, additional demonstrations from the teacher were used to verify the reproduction performance of the policy, by assuming human teacher and robot learner are physical identical systems. The results confirmed the generalisation capability of the HMM model across a number of demonstrations from different subjects; and the reproduced motions from GMR were acceptable in these additional tests. The proposed methodology provides a framework for producing a state-action model from skilled demonstrations that can be translated into robot kinematics and joint states for the robot to execute. The implication to industry is reduced efforts and time in programming the robots for applications where human skilled performances are required to cope robustly with various uncertainties during tasks execution

Haptics: Science, Technology, Applications

This open access book constitutes the proceedings of the 12th International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, EuroHaptics 2020, held in Leiden, The Netherlands, in September 2020. The 60 papers presented in this volume were carefully reviewed and selected from 111 submissions. The were organized in topical sections on haptic science, haptic technology, and haptic applications. This year's focus is on accessibility

Micro/Nano Manufacturing

Micro- and nano-scale manufacturing has been the subject of ever more research and industrial focus over the past 10 years. Traditional lithography-based technology forms the basis of micro-electro-mechanical systems (MEMS) manufacturing, but also precision manufacturing technologies have been developed to cover micro-scale dimensions and accuracies. Furthermore, these fundamentally different technology platforms are currently combined in order to exploit the strengths of both platforms. One example is the use of lithography-based technologies to establish nanostructures that are subsequently transferred to 3D geometries via injection molding. Manufacturing processes at the micro-scale are the key-enabling technologies to bridge the gap between the nano- and the macro-worlds to increase the accuracy of micro/nano-precision production technologies, and to integrate different dimensional scales in mass-manufacturing processes. Accordingly, this Special Issue seeks to showcase research papers, short communications, and review articles that focus on novel methodological developments in micro- and nano-scale manufacturing, i.e., on novel process chains including process optimization, quality assurance approaches and metrology

Mechanical Manipulation and Characterization of Biological Cells

Mechanical manipulation and characterization of an individual biological cell is currently one of the most exciting research areas in the field of medical robotics. Single cell manipulation is an important process in intracytoplasmic sperm injection (ICSI), pro-nuclei DNA injection, gene therapy, and other biomedical areas. However, conventional cell manipulation requires long training and the success rate depends on the experience of the operator. The goal of this research is to address the drawbacks of conventional cell manipulation by using force and vision feedback for cell manipulation tasks. We hypothesize that force feedback plays an important role in cell manipulation and possibly helps in cell characterization. This dissertation will summarize our research on: 1) the development of force and vision feedback interface for cell manipulation, 2) human subject studies to evaluate the addition of force feedback for cell injection tasks, 3) the development of haptics-enabled atomic force microscope system for cell indentation tasks, 4) appropriate analytical model for characterizing the mechanical property of mouse embryonic stem cells (mESC) and 5) several indentation studies on mESC to determine the mechanical property of undifferentiated and early differentiating (6 days under differentiation conditions) mESC. Our experimental results on zebrafish egg cells show that a system with force feedback capability when combined with vision feedback can lead to potentially higher success rates in cell injection tasks. Using this information, we performed experiments on mESC using the AFM to understand their characteristics in the undifferentiated pluripotent state as well as early differentiating state. These experiments were done on both live as well as fixed cells to understand the correlation between the two during cell indentation studies. Our results show that the mechanical property of undifferentiated mESC differs from early differentiating (6th day) mESC in both live and fixed cells. Thus, we hypothesize that mechanical characterization studies will potentially pave the way for developing a high throughput system with force feedback capability, to understand and predict the differentiation path a particular pluripotent cell will follow. This finding could also be used to develop improved methods of targeted cellular differentiation of stem cells for therapeutic and regenerative medicine

Haptics: Science, Technology, Applications

This open access book constitutes the proceedings of the 12th International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, EuroHaptics 2020, held in Leiden, The Netherlands, in September 2020. The 60 papers presented in this volume were carefully reviewed and selected from 111 submissions. The were organized in topical sections on haptic science, haptic technology, and haptic applications. This year's focus is on accessibility