Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review

It is generally accepted that augmented feedback, provided by a human expert or a technical display, effectively enhances motor learning. However, discussion of the way to most effectively provide augmented feedback has been controversial. Related studies have focused primarily on simple or artificial tasks enhanced by visual feedback. Recently, technical advances have made it possible also to investigate more complex, realistic motor tasks and to implement not only visual, but also auditory, haptic, or multimodal augmented feedback. The aim of this review is to address the potential of augmented unimodal and multimodal feedback in the framework of motor learning theories. The review addresses the reasons for the different impacts of feedback strategies within or between the visual, auditory, and haptic modalities and the challenges that need to be overcome to provide appropriate feedback in these modalities, either in isolation or in combination. Accordingly, the design criteria for successful visual, auditory, haptic, and multimodal feedback are elaborate

Investigating The Impact Of Visuohaptic Simulations For Conceptual Understanding In Electricity And Magnetism

The present study examined the efficacy of a haptic simulation used as a pedagogical tool to teach freshmen engineering students about electromagnetism. A quasi-experimental design-based research was executed in two iterations to compare the possible benefits the haptic device provided to the cognitive learning of students. In the first iteration of the experiment performance of learners who used visual-only simulations was compared to the performance of those who used visuohaptic. In the second iteration of the experiment modifications were made to learning materials and experiment procedures to enhance research design. Research hypothesis states that multimodal presentation of information may lead to better conceptual understanding of electromagnetism compared to visual presentation alone

Serious Games and Mixed Reality Applications for Healthcare

Virtual reality (VR) and augmented reality (AR) have long histories in the healthcare sector, offering the opportunity to develop a wide range of tools and applications aimed at improving the quality of care and efficiency of services for professionals and patients alike. The best-known examples of VR–AR applications in the healthcare domain include surgical planning and medical training by means of simulation technologies. Techniques used in surgical simulation have also been applied to cognitive and motor rehabilitation, pain management, and patient and professional education. Serious games are ones in which the main goal is not entertainment, but a crucial purpose, ranging from the acquisition of knowledge to interactive training.These games are attracting growing attention in healthcare because of their several benefits: motivation, interactivity, adaptation to user competence level, flexibility in time, repeatability, and continuous feedback. Recently, healthcare has also become one of the biggest adopters of mixed reality (MR), which merges real and virtual content to generate novel environments, where physical and digital objects not only coexist, but are also capable of interacting with each other in real time, encompassing both VR and AR applications.This Special Issue aims to gather and publish original scientific contributions exploring opportunities and addressing challenges in both the theoretical and applied aspects of VR–AR and MR applications in healthcare

The Shape of Damping: Optimizing Damping Coefficients to Improve Transparency on Bilateral Telemanipulation

This thesis presents a novel optimization-based passivity control algorithm for hapticenabled bilateral teleoperation systems involving multiple degrees of freedom. In particular, in the context of energy-bounding control, the contribution focuses on the implementation of a passivity layer for an existing time-domain scheme, ensuring optimal transparency of the interaction along subsets of the environment space which are preponderant for the given task, while preserving the energy bounds required for passivity. The involved optimization problem is convex and amenable to real-time implementation. The effectiveness of the proposed design is validated via an experiment performed on a virtual teleoperated environment. The interplay between transparency and stability is a critical aspect in haptic-enabled bilateral teleoperation control. While it is important to present the user with the true impedance of the environment, destabilizing factors such as time delays, stiff environments, and a relaxed grasp on the master device may compromise the stability and safety of the system. Passivity has been exploited as one of the the main tools for providing sufficient conditions for stable teleoperation in several controller design approaches, such as the scattering algorithm, timedomain passivity control, energy bounding algorithm, and passive set position modulation. In this work it is presented an innovative energy-based approach, which builds upon existing time-domain passivity controllers, improving and extending their effectiveness and functionality. The set of damping coefficients are prioritized in each degree of freedom, the resulting transparency presents a realistic force feedback in comparison to the other directions. Thus, the prioritization takes effect using a quadratic programming algorithm to find the optimal values for the damping. Finally, the energy tanks approach on passivity control is a solution used to ensure stability in a system for robotics bilateral manipulation. The bilateral telemanipulation must maintain the principle of passivity in all moments to preserve the system\u2019s stability. This work presents a brief introduction to haptic devices as a master component on the telemanipulation chain; the end effector in the slave side is a representation of an interactive object within an environment having a force sensor as feedback signal. The whole interface is designed into a cross-platform framework named ROS, where the user interacts with the system. Experimental results are presented

Expert-in-the-Loop Multilateral Telerobotics for Haptics-Enabled Motor Function and Skills Development

Among medical robotics applications are Robotics-Assisted Mirror Rehabilitation Therapy (RAMRT) and Minimally-Invasive Surgical Training (RAMIST) that extensively rely on motor function development. Haptics-enabled expert-in-the-loop motor function development for such applications is made possible through multilateral telerobotic frameworks. While several studies have validated the benefits of haptic interaction with an expert in motor learning, contradictory results have also been reported. This emphasizes the need for further in-depth studies on the nature of human motor learning through haptic guidance and interaction. The objective of this study was to design and evaluate expert-in-the-loop multilateral telerobotic frameworks with stable and human-safe control loops that enable adaptive “hand-over-hand” haptic guidance for RAMRT and RAMIST. The first prerequisite for such frameworks is active involvement of the patient or trainee, which requires the closed-loop system to remain stable in the presence of an adaptable time-varying dominance factor. To this end, a wave-variable controller is proposed in this study for conventional trilateral teleoperation systems such that system stability is guaranteed in the presence of a time-varying dominance factor and communication delay. Similar to other wave-variable approaches, the controller is initially developed for the Velocity-force Domain (VD) based on the well-known passivity assumption on the human arm in VD. The controller can be applied straightforwardly to the Position-force Domain (PD), eliminating position-error accumulation and position drift, provided that passivity of the human arm in PD is addressed. However, the latter has been ignored in the literature. Therefore, in this study, passivity of the human arm in PD is investigated using mathematical analysis, experimentation as well as user studies involving 12 participants and 48 trials. The results, in conjunction with the proposed wave-variables, can be used to guarantee closed-loop PD stability of the supervised trilateral teleoperation system in its classical format. The classic dual-user teleoperation architecture does not, however, fully satisfy the requirements for properly imparting motor function (skills) in RAMRT (RAMIST). Consequently, the next part of this study focuses on designing novel supervised trilateral frameworks for providing motor learning in RAMRT and RAMIST, each customized according to the requirements of the application. The framework proposed for RAMRT includes the following features: a) therapist-in-the-loop mirror therapy; b) haptic feedback to the therapist from the patient side; c) assist-as-needed therapy realized through an adaptive Guidance Virtual Fixture (GVF); and d) real-time task-independent and patient-specific motor-function assessment. Closed-loop stability of the proposed framework is investigated using a combination of the Circle Criterion and the Small-Gain Theorem. The stability analysis addresses the instabilities caused by: a) communication delays between the therapist and the patient, facilitating haptics-enabled tele- or in-home rehabilitation; and b) the integration of the time-varying nonlinear GVF element into the delayed system. The platform is experimentally evaluated on a trilateral rehabilitation setup consisting of two Quanser rehabilitation robots and one Quanser HD2 robot. The framework proposed for RAMIST includes the following features: a) haptics-enabled expert-in-the-loop surgical training; b) adaptive expertise-oriented training, realized through a Fuzzy Interface System, which actively engages the trainees while providing them with appropriate skills-oriented levels of training; and c) task-independent skills assessment. Closed-loop stability of the architecture is analyzed using the Circle Criterion in the presence and absence of haptic feedback of tool-tissue interactions. In addition to the time-varying elements of the system, the stability analysis approach also addresses communication delays, facilitating tele-surgical training. The platform is implemented on a dual-console surgical setup consisting of the classic da Vinci surgical system (Intuitive Surgical, Inc., Sunnyvale, CA), integrated with the da Vinci Research Kit (dVRK) motor controllers, and the dV-Trainer master console (Mimic Technology Inc., Seattle, WA). In order to save on the expert\u27s (therapist\u27s) time, dual-console architectures can also be expanded to accommodate simultaneous training (rehabilitation) for multiple trainees (patients). As the first step in doing this, the last part of this thesis focuses on the development of a multi-master/single-slave telerobotic framework, along with controller design and closed-loop stability analysis in the presence of communication delays. Various parts of this study are supported with a number of experimental implementations and evaluations. The outcomes of this research include multilateral telerobotic testbeds for further studies on the nature of human motor learning and retention through haptic guidance and interaction. They also enable investigation of the impact of communication time delays on supervised haptics-enabled motor function improvement through tele-rehabilitation and mentoring

Desenvolvimento de ferramentas de treino para teleoperação háptica de um robô humanóide

Mestrado emEngenharia MecânicaIn robotics, the teleoperation of biped humanoids is one of the most exciting topics. It has the possibility to bypass complex dynamic models with learning demonstration algorithms using human interaction. For this procedure, the Humanoid Project at the University of Aveiro - PHUA, ingrained in the production of a 27 degree-of-freedom full body humanoid platform teleoperated by means of haptic devices. The current project also comprises a robot model that has be imported into the Virtual Robot Experimentation Platform: V-REP. The usage of the simulator allows multiple exercises with greater speed and shorted setup times, when compared to the teleoperation of the real robot, besides providing more safety for the platform and the operator during the tests. By using the simulator, the user can perform tests and make achievements towards the reproduction of human movement with the interaction of two haptic devices providing force feedback to the operator. The performed maneuvers have their kinematic and dynamic data stored for later application in learning by demonstration algorithms. However, the production of more complex and detailed movements requires large amounts of motor skill from the operator. Due to the continuous change of users in the PHUA, an adaptation period is required for the newly arrived operators to develop an a nity with the complex control system. This work is focused on developing methodologies to lower the required time for the training process. Thanks to the versatility of customization provided by V-REP, it was possible to implement interfaces which utilized visual and haptic guidance to enhance the learning capabilities of the operator. A dedicate workstation, new formulations and support tools that control the simulation were developed in order to create a more intuitive control over the humanoid platform. Operators were instructed to reproduce complex 3D movements under several training conditions (with visual and haptic feedback, only haptic feedback, only visual feedback, with guidance tools and without guidance). Performance was measured in terms of speed, drift from intended trajectory, response to the drift and amplitude of the movement. Findings of this study indicate that, with the newly implemented mechanisms, operators are able to gain control over the humanoid platform within a relatively short period of training. Operators subjected to the guidance programs present an even shorter period of training needed, exhibiting high performance in the overall system. These facts support the role of haptic guidance in acquiring kinesthetic memory in high DOFs systems.Em robótica, a teleoperação de robôs bípede humanóides é um dos tópicos mais emocionante. Tem a possibilidade de contornar modelos dinâmicos rígidos, com algoritmos de aprendizagem por demonstração utilizando interação humana. Para este procedimento, o Projeto Humanóide da Universidade de Aveiro - PHUA, empanha-se na produção de uma plataforma humanóide de corpo inteiro teleoperado com dispositivos hapticos. O estado presente do projeto apresenta um robô humanóide com 27 graus de liberdade. O projeto actual apresenta um modelo do robô importado para a Virtual Robot Exper- imentation Platform: V-REP. O uso do simulador permite vários exercícios com maior velocidade e tempos de preparação curtos, quando comparado com a teleoperação do robô real, além de proporcionar mais segurança para a plataforma e do operador durante os ensaios. Ao utilizar o simulador, o utilizador pode realizar testes à reprodução de movimento humano com a interacção de dois dispositivos de meios hápticos que fornecem força de retorno para o operador. As manobras realizadas têm os seus dados cinemáticos e dinâmicos armazenados para posterior aplicação na aprendizagem por algoritmos de demonstração. No entanto, a produção de movimentos mais complexos e detalhados requer grandes quantidades de habilidade motora do operador. Devido à mudança contínua de usuários no PHUA, um período de adaptação é necessário para os operadores recém-chegados a desenvolver uma a nidade com o complexo sistema de controlo. Este trabalho é focado no desenvolvimento de metodologias para diminuir o tempo necessário para o processo de formação dos utilizadores. Graças à versatilidade de personalização fornecidos pela V-REP, foi possível implementar interfaces que utilizaram orientação visual e haptica para melhorar as capacidades de aprendizagem do operador. Uma estação de trabalho, novas formulações e ferramentas de apoio que controlam a simulação foram desenvolvidos a m de criar um controle mais intuitivo sobre a plataforma humanóide. Os operadores foram instruídos a reproduzir movimentos complexos em 3D sob diversas condições de treino (com feedback visual e haptico, apenas feedback haptico, apenas feedback visual, com ferramentas de orientação e sem orientação). O desempenho foi medido em termos de velocidade, a desvio de trajectória pretendida, a resposta à desvio e o tempo gasto para a criação do movimento. Os resultados deste estudo indicam que, com os mecanismos recém-implementadas, os operadores são capazes de ganhar o controlo sobre a plataforma humanóide dentro de um período relativamente curto de treino. Operadores submetidos a programas de orientação apresentam um período ainda mais curto de formação necessária, exibindo alto desempenho no sistema global. Estes fatos justi cam o papel da orientação haptica em adquirir memória cinestésica em sistemas DOFs elevados

Development of Cognitive Capabilities in Humanoid Robots

Merged with duplicate record 10026.1/645 on 03.04.2017 by CS (TIS)Building intelligent systems with human level of competence is the ultimate grand challenge for science and technology in general, and especially for the computational intelligence community. Recent theories in autonomous cognitive systems have focused on the close integration (grounding) of communication with perception, categorisation and action. Cognitive systems are essential for integrated multi-platform systems that are capable of sensing and communicating. This thesis presents a cognitive system for a humanoid robot that integrates abilities such as object detection and recognition, which are merged with natural language understanding and refined motor controls. The work includes three studies; (1) the use of generic manipulation of objects using the NMFT algorithm, by successfully testing the extension of the NMFT to control robot behaviour; (2) a study of the development of a robotic simulator; (3) robotic simulation experiments showing that a humanoid robot is able to acquire complex behavioural, cognitive, and linguistic skills through individual and social learning. The robot is able to learn to handle and manipulate objects autonomously, to cooperate with human users, and to adapt its abilities to changes in internal and environmental conditions. The model and the experimental results reported in this thesis, emphasise the importance of embodied cognition, i.e. the humanoid robot's physical interaction between its body and the environment