Assessing Performance, Role Sharing, and Control Mechanisms in Human-Human Physical Interaction for Object Manipulation

abstract: Object manipulation is a common sensorimotor task that humans perform to interact with the physical world. The first aim of this dissertation was to characterize and identify the role of feedback and feedforward mechanisms for force control in object manipulation by introducing a new feature based on force trajectories to quantify the interaction between feedback- and feedforward control. This feature was applied on two grasp contexts: grasping the object at either (1) predetermined or (2) self-selected grasp locations (“constrained” and “unconstrained”, respectively), where unconstrained grasping is thought to involve feedback-driven force corrections to a greater extent than constrained grasping. This proposition was confirmed by force feature analysis. The second aim of this dissertation was to quantify whether force control mechanisms differ between dominant and non-dominant hands. The force feature analysis demonstrated that manipulation by the dominant hand relies on feedforward control more than the non-dominant hand. The third aim was to quantify coordination mechanisms underlying physical interaction by dyads in object manipulation. The results revealed that only individuals with worse solo performance benefit from interpersonal coordination through physical couplings, whereas the better individuals do not. This work showed that naturally emerging leader-follower roles, whereby the leader in dyadic manipulation exhibits significant greater force changes than the follower. Furthermore, brain activity measured through electroencephalography (EEG) could discriminate leader and follower roles as indicated power modulation in the alpha frequency band over centro-parietal areas. Lastly, this dissertation suggested that the relation between force and motion (arm impedance) could be an important means for communicating intended movement direction between biological agents.Dissertation/ThesisDoctoral Dissertation Biomedical Engineering 201

Human factors in instructional augmented reality for intravehicular spaceflight activities and How gravity influences the setup of interfaces operated by direct object selection

In human spaceflight, advanced user interfaces are becoming an interesting mean to facilitate human-machine interaction, enhancing and guaranteeing the sequences of intravehicular space operations. The efforts made to ease such operations have shown strong interests in novel human-computer interaction like Augmented Reality (AR). The work presented in this thesis is directed towards a user-driven design for AR-assisted space operations, iteratively solving issues arisen from the problem space, which also includes the consideration of the effect of altered gravity on handling such interfaces.Auch in der bemannten Raumfahrt steigt das Interesse an neuartigen Benutzerschnittstellen, um nicht nur die Mensch-Maschine-Interaktion effektiver zu gestalten, sondern auch um einen korrekten Arbeitsablauf sicherzustellen. In der Vergangenheit wurden wiederholt Anstrengungen unternommen, Innenbordarbeiten mit Hilfe von Augmented Reality (AR) zu erleichtern. Diese Arbeit konzentriert sich auf einen nutzerorientierten AR-Ansatz, welcher zum Ziel hat, die Probleme schrittweise in einem iterativen Designprozess zu lösen. Dies erfordert auch die Berücksichtigung veränderter Schwerkraftbedingungen

Attention and time constraints in performing and learning a table tennis forehand shot

This is a section on p. S95 of article 'Verbal and Poster: Motor Development, Motor Learning and Control, and Sport and Exercise Psychology' in Journal of Sport and Exercise Psychology, 2010, v.32, p.S36-S237published_or_final_versio

EEG coherence between the verbal-analytical region (T3) and the motor-planning region (Fz) increases under stress in explicit motor learners but not implicit motor learners

This journal supplement contains abstracts of NASPSPA 2010Free Communications - Verbal and Poster: Motor Learning and Controlpublished_or_final_versionThe Annual Conference of the North American Society for the Psychology of Sport and Physical Activity (NASPSPA 2010), Tucson, AZ., 10-12 June 2010. In Journal of Sport and Exercise Psychology, 2010, v. 32 suppl., p. S13

Prototypical Arm Motions from Human Demonstration for Upper-Limb Prosthetic Device Control

Controlling a complex upper limb prosthesis, akin to a healthy arm, is still an open challenge due to the inadequate number of inputs available to amputees. Designs have therefore largely focused on a limited number of controllable degrees of freedom, developing a complex hand and grasp functionality rather than the wrist. This thesis investigates joint coordination based on human demonstrations that aims to vastly simplify the controls of wrist, elbow-wrist, and shoulder-elbow wrist devices.The wide range of motions performed by the human arm during daily tasks makes it desirable to find representative subsets to reduce the dimensionality of these movements for a variety of applications, including the design and control of robotic and prosthetic devices. Here I present the results of an extensive human subjects study and two methods that were used to obtain representative categories of arm use that span naturalistic motions during activities of daily living. First, I sought to identify sets of prototypical upper-limb motions that are functions of a single variable, allowing, for instance, an entire prosthetic or robotic arm to be controlled with a single input from a user, along with a means to select between motions for different tasks. Second, I decouple the orientation from the location of the hand and analyze the hand location in three ways and orientation in three reference frames. Both of these analyses are an application of data driven approaches that reduce the wide range of hand and arm use to a smaller representative set. Together these provide insight into our arm usage in daily life and inform an implementation in prosthetic or robotic devices without the need for additional hardware. To demonstrate the control efficacy of prototypical arm motions in upper-limb prosthetic devices, I developed an immersive virtual reality environment where able-bodied participants tested out different devices and controls. I coined prototypical arm motion control as trajectory control, and I found that as device complexity increased from 3 DOF wrist to 4 DOF elbow-wrist and 7 DOF shoulder-elbow-wrist, it enables users to complete tasks faster with a more intuitive interface without additional body compensation, while featuring better movement cosmesis when compared to standard controls

Hand eye coordination in surgery

The coordination of the hand in response to visual target selection has always been regarded as an essential quality in a range of professional activities. This quality has thus far been elusive to objective scientific measurements, and is usually engulfed in the overall performance of the individuals. Parallels can be drawn to surgery, especially Minimally Invasive Surgery (MIS), where the physical constraints imposed by the arrangements of the instruments and visualisation methods require certain coordination skills that are unprecedented. With the current paradigm shift towards early specialisation in surgical training and shortened focused training time, selection process should identify trainees with the highest potentials in certain specific skills. Although significant effort has been made in objective assessment of surgical skills, it is only currently possible to measure surgeons’ abilities at the time of assessment. It has been particularly difficult to quantify specific details of hand-eye coordination and assess innate ability of future skills development. The purpose of this thesis is to examine hand-eye coordination in laboratory-based simulations, with a particular emphasis on details that are important to MIS. In order to understand the challenges of visuomotor coordination, movement trajectory errors have been used to provide an insight into the innate coordinate mapping of the brain. In MIS, novel spatial transformations, due to a combination of distorted endoscopic image projections and the “fulcrum” effect of the instruments, accentuate movement generation errors. Obvious differences in the quality of movement trajectories have been observed between novices and experts in MIS, however, this is difficult to measure quantitatively. A Hidden Markov Model (HMM) is used in this thesis to reveal the underlying characteristic movement details of a particular MIS manoeuvre and how such features are exaggerated by the introduction of rotation in the endoscopic camera. The proposed method has demonstrated the feasibility of measuring movement trajectory quality by machine learning techniques without prior arbitrary classification of expertise. Experimental results have highlighted these changes in novice laparoscopic surgeons, even after a short period of training. The intricate relationship between the hands and the eyes changes when learning a skilled visuomotor task has been previously studied. Reactive eye movement, when visual input is used primarily as a feedback mechanism for error correction, implies difficulties in hand-eye coordination. As the brain learns to adapt to this new coordinate map, eye movements then become predictive of the action generated. The concept of measuring this spatiotemporal relationship is introduced as a measure of hand-eye coordination in MIS, by comparing the Target Distance Function (TDF) between the eye fixation and the instrument tip position on the laparoscopic screen. Further validation of this concept using high fidelity experimental tasks is presented, where higher cognitive influence and multiple target selection increase the complexity of the data analysis. To this end, Granger-causality is presented as a measure of the predictability of the instrument movement with the eye fixation pattern. Partial Directed Coherence (PDC), a frequency-domain variation of Granger-causality, is used for the first time to measure hand-eye coordination. Experimental results are used to establish the strengths and potential pitfalls of the technique. To further enhance the accuracy of this measurement, a modified Jensen-Shannon Divergence (JSD) measure has been developed for enhancing the signal matching algorithm and trajectory segmentations. The proposed framework incorporates high frequency noise filtering, which represents non-purposeful hand and eye movements. The accuracy of the technique has been demonstrated by quantitative measurement of multiple laparoscopic tasks by expert and novice surgeons. Experimental results supporting visual search behavioural theory are presented, as this underpins the target selection process immediately prior to visual motor action generation. The effects of specialisation and experience on visual search patterns are also examined. Finally, pilot results from functional brain imaging are presented, where the Posterior Parietal Cortical (PPC) activation is measured using optical spectroscopy techniques. PPC has been demonstrated to involve in the calculation of the coordinate transformations between the visual and motor systems, which establishes the possibilities of exciting future studies in hand-eye coordination

Muscle Coordination Contributes to Function after Stroke; Proprioception Contributes to Control of Posture, Movement

More than half of stroke survivors experience persistent upper extremity motor impairments that can negatively impact quality of life and independence. Effective use of the upper extremity requires coordination of agonist/antagonist muscle pairs, as well as coordination of multiple control actions for stabilizing and moving the arm. In this dissertation, I present three studies in which I recorded isometric torque production, single joint movement and stabilization, and clinical measures of function and impairments after stroke to evaluate the extent to which changes in coordination of agonist/antagonist muscles and of sequential control actions contribute to deficits after stroke. In Aim 1, I quantified the extent to which stroke-related deficits in the coordination of agonist/antagonist muscle pairs degraded the ability to generate, maintain, and relax cued torques about the elbow. Participants who survived stroke (SP) and neurologically intact participants (NI) performed pursuit tracking of step-changes in isomeric torque targets to investigate coordination of activation magnitude in elbow agonist/antagonist muscle pairs. SP had marked hypertonia of the primary flexor muscles, which led to increased compensatory activity in the primary extensor muscles. These stroke-related deficits of muscle coordination degraded ability to generate, maintain, and relax cued torque production. In Aim 2, SP and NI performed sequential combinations of elbow stabilization and movements to investigate impairments in execution and coordination of these fundamental control actions. Impaired proprioception in SP was associated with increased impairments in stabilizing the arm against a perturbation compared with SP with intact proprioception. Surprisingly, SP with intact proprioception had greater impairments when moving than did SP with impaired proprioception. These results support the supposition that deficits of somatosensation can differentially impact neural control of limb stabilization and movement. Aim 3 used correlation and forward regression to compare deficits of muscle coordination (Aim 1) and control (Aim 2) to one another in order to quantify the extent to which each could explain deficits of motor function after stroke. Taken together, the three studies revealed that stroke-related deficits in coordination timing and magnitude of muscle activation impact clinically-measured function, and that somatosensory deficits can differentially impair neuromotor stabilization and movement control

Investigating sensory-motor interactions to shape rehabilitation

Over the last decades, robotic devices for neurorehabilitation have been developed with the aim of providing better and faster improvement of motor performance. These devices are being used to help patients repeat movements and (re)learn different dynamic tasks. Over the years, these devices have become bigger and more complex, so as to provide the end user with a more realistic and sophisticated stimuli while still allowing the experimenter to have control over the interaction forces that can potentially shape the motor behaviour. However, experimental results have shown no clear advantage of these complex devices over simpler versions. In this context, this thesis investigates sensory-motor processes of human interaction, which can help us understand the main issues for rehabilitation devices and how to overcome the limitations of simple devices to train particular motor behaviours. Conventional neurorehabilitation of motor function relies on haptic interaction between the patient and physiotherapist. However, how humans deal with human-human interactions is largely unknown, and has been little studied. In this regard, experiments of the first section of the thesis investigate the mechanisms of interaction during human-human collaborative tasks. It goes from identifying the different strategies that dyads can take to proposing methods to measure and understand redundancy and synchrony in haptic interactions. It also shows that one can shape the interaction between partners by modifying only the visual information provided to each agent. Learning a novel skill requires integration of different sensory modalities, in particular vision and proprioception. Hence, one can expect that learning will depend on the mechanical characteristics of the device. For instance, a device with limited degrees of freedom will reduce the amount of information about the environment, modify the dynamics of the task and prevent certain error-based corrections. To investigate this, the second section of the thesis examines whether the lack of proprioceptive feedback that is created due to mechanical constraints or haptic guidance can be substituted with visual information. Psychophysical experiments with healthy subjects and some preliminary experiments with stroke patients presented in this thesis support the idea that by incorporating task-relevant visual feedback into simple devices, one could deliver effective neurorehabilitation protocols. The contributions of the thesis are not limited to the role of visual feedback to shape motor behaviour, but also advance our understanding on the mechanisms of learning and human-human interaction