Brain Functional Connectivity under Teleoperation Latency: a fNIRS Study

Objective: This study aims to understand the cognitive impact of latency in teleoperation and the related mitigation methods, using functional Near-Infrared Spectroscopy (fNIRS) to analyze functional connectivity. Background: Latency between command, execution, and feedback in teleoperation can impair performance and affect operators mental state. The neural underpinnings of these effects are not well understood. Method: A human subject experiment (n = 41) of a simulated remote robot manipulation task was performed. Three conditions were tested: no latency, with visual and haptic latency, with visual latency and no haptic latency. fNIRS and performance data were recorded and analyzed. Results: The presence of latency in teleoperation significantly increased functional connectivity within and between prefrontal and motor cortexes. Maintaining visual latency while providing real-time haptic feedback reduced the average functional connectivity in all cortical networks and showed a significantly different connectivity ratio within prefrontal and motor cortical networks. The performance results showed the worst performance in the all-delayed condition and best performance in no latency condition, which echoes the neural activity patterns. Conclusion: The study provides neurological evidence that latency in teleoperation increases cognitive load, anxiety, and challenges in motion planning and control. Real-time haptic feedback, however, positively influences neural pathways related to cognition, decision-making, and sensorimotor processes. Application: This research can inform the design of ergonomic teleoperation systems that mitigate the effects of latency.Comment: Submitted to Human Factor

Output Feedback Bilateral Teleoperation with Force Estimation in the Presence of Time Delays

This thesis presents a novel bilateral teleoperation algorithm for n degree of freedom nonlinear manipulators connected through time delays. Teleoperation has many practical uses, as there are many benefits that come from being able to operate machines from a distance. For instance, the ability to send a remote controlled robotic vehicle into a hazardous environment can be a great asset in many industrial applications. As well, the field of remote medicine can benefit from these technologies. A highly skilled surgeon could perform surgery on a patient who is located in another city, or even country. Earth to space operations and deep sea exploration are other areas where teleoperation is quite useful. Central to the approach presented in this work is the use of second order sliding mode unknown input observers for estimating the external forces acting on the manipulators. The use of these observers removes the need for both velocity and force sensors, leading to a lower cost hardware setup that provides all of the advantages of a position-force teleoperation algorithm. Stability results for this new algorithm are presented for several cases. Stability of each of the master and slave sides of the teleoperation system is demonstrated, showing that the master and slave are both stabilized by their respective controllers when the unknown input observers are used for state and force estimation. Additionally, closed loop stability results for the teleoperation system connected to a variety of slave side environments are presented. Delay-independent stability results for a linear spring-damper environment as well as a general finite-gain stable nonlinear environment are given. Delay-dependent stability results for the case where the slave environment is a liner spring-damper and the delays are commensurate are also presented. As well, stability results for the closed loop under the assumption that the human operator is modeled as a finite-gain stable nonlinear environment are given. Following the theoretical presentation, numerical simulations illustrating the algorithm are presented, and experimental results verifying the practical application of the approach are given

Consensus control in robot networks and cooperative teleoperation : an operational space approach

An interesting approach in cooperative control is to design distributed control strategies which use only local information so that a multi-agent system achieves specified behaviors. A basic behavior in cooperative control is the consensus. Given a multi-agent system, like a multiple robot network, it is said that the agents reach a consensus if the state of each agent converges to a common state. Examples of cooperative tasks in which consensus algorithms are employed include formation control, flocking theory, rendezvous problems and synchronization. These cooperative tasks have several possible applications, like: transportation systems (intelligent highways, air-traffic control); military systems (formation flight, surveillance, reconnaissance, cooperative attack and rendezvous) and mobile sensor networks (space-based interferometers, environmental sampling). The solution to the consensus problems involves the design of control algorithms such that the agents can reach an agreement on their states. There are two main problems that are studied in consensus, the leader-follower consensus and the leaderless consensus. In the leader-follower consensus problem, there exists a leader that specifies the state for the whole group while in a leaderless consensus problem, there is not a priori reference state. The main goal of this thesis is the design of operational space controllers that solve the leader-follower and the leaderless consensus problems in networks composed of multiple heterogeneous robots. Furthermore, this document proposes novel operational space control schemes for bilateral teleoperation systems. In both scenarios, different conditions are studied, such as the absence of robot velocity measurements, constant and variable time-delays in the robot's interconnection, and uncertainty in the robot's physical parameters. Most of the previous consensus control algorithms, only work with the position or orientation but not with both. On the contrary, this dissertation deals with the entire pose of the robots that contains both the position and the orientation. Moreover, in order to render a singularity-free description of the orientation, the unit-quaternions are employed. The dissertation provides a rigorous stability analysis of the control algorithms and presents simulations and experiments that validate the effectiveness of the proposed controllers.Un enfoque interesante en el control cooperativo es el diseño de estrategias de control distribuido que requieran sólo información local para que un sistema multi-agente logre comportamientos específicos. Un comportamiento básico del control cooperativo es el consenso. Dado un sistema multi-agente, como una red de múltiples robots, se dice que los agentes llegan a un consenso si el estado de cada agente converge a un estado común. Algunos ejemplos de tareas cooperativas en las que los algoritmos de consenso son utilizados son los siguientes: el control de la formación, flocking, rendezvous y sincronización. Estas tareas cooperativas tienen varias aplicaciones posibles, como: sistemas de transporte (carreteras inteligentes , control de tráfico aéreo); sistemas militares (vuelo en formación, vigilancia, reconocimiento, ataque cooperativo) y redes de sensores móviles (interferómetros en el espacio, el muestreo del ambiente). La solución a los problemas de consenso implica el diseño de algoritmos de control de tal manera que los agentes pueden llegar a un acuerdo sobre sus estados. Hay dos problemas principales que se estudian en el consenso, el consenso líder-seguidor y el consenso sin líder. En el problema de consenso líder-seguidor, existe un líder que especifica el estado de todo el grupo, mientras que en un problema de consenso sin líder, no hay ningún estado de referencia definido a priori. El objetivo principal de esta tesis es el diseño de controladores en el espacio operacional que resuelvan los problemas de consenso líder-seguidor y sin líder en redes compuestas de múltiples robots heterogéneos. Además, este documento propone novedosos esquemas de control en el espacio operacional para sistemas de teleoperación bilateral. En ambos escenarios, se estudian diferentes condiciones, tales como la ausencia de medidas de velocidad de los robots, retardos constantes y variables en la interconexión de los robots y la incertidumbre en los parámetros físicos de los robots. La mayoría de los anteriores algoritmos de control que resuelven el consenso, sólo trabajan con la posición o la orientación, pero no con ambos. Por el contrario, esta tesis doctoral se ocupa de toda la pose de los robots que contiene tanto la posición y la orientación. Por otra parte, a fin de usar una representación de la orientación libre de singularidades, se emplean los cuaterniones unitarios. Esta tesis doctoral proporciona un análisis riguroso de la estabilidad de los algoritmos de control y presenta simulaciones y experimentos que validan la eficacia de los controladores propuesto