Towards Intelligent Telerobotics: Visualization and Control of Remote Robot

Human-machine cooperative or co-robotics has been recognized as the next generation of robotics. In contrast to current systems that use limited-reasoning strategies or address problems in narrow contexts, new co-robot systems will be characterized by their flexibility, resourcefulness, varied modeling or reasoning approaches, and use of real-world data in real time, demonstrating a level of intelligence and adaptability seen in humans and animals. The research I focused is in the two sub-field of co-robotics: teleoperation and telepresence. We firstly explore the ways of teleoperation using mixed reality techniques. I proposed a new type of display: hybrid-reality display (HRD) system, which utilizes commodity projection device to project captured video frame onto 3D replica of the actual target surface. It provides a direct alignment between the frame of reference for the human subject and that of the displayed image. The advantage of this approach lies in the fact that no wearing device needed for the users, providing minimal intrusiveness and accommodating users eyes during focusing. The field-of-view is also significantly increased. From a user-centered design standpoint, the HRD is motivated by teleoperation accidents, incidents, and user research in military reconnaissance etc. Teleoperation in these environments is compromised by the Keyhole Effect, which results from the limited field of view of reference. The technique contribution of the proposed HRD system is the multi-system calibration which mainly involves motion sensor, projector, cameras and robotic arm. Due to the purpose of the system, the accuracy of calibration should also be restricted within millimeter level. The followed up research of HRD is focused on high accuracy 3D reconstruction of the replica via commodity devices for better alignment of video frame. Conventional 3D scanner lacks either depth resolution or be very expensive. We proposed a structured light scanning based 3D sensing system with accuracy within 1 millimeter while robust to global illumination and surface reflection. Extensive user study prove the performance of our proposed algorithm. In order to compensate the unsynchronization between the local station and remote station due to latency introduced during data sensing and communication, 1-step-ahead predictive control algorithm is presented. The latency between human control and robot movement can be formulated as a linear equation group with a smooth coefficient ranging from 0 to 1. This predictive control algorithm can be further formulated by optimizing a cost function. We then explore the aspect of telepresence. Many hardware designs have been developed to allow a camera to be placed optically directly behind the screen. The purpose of such setups is to enable two-way video teleconferencing that maintains eye-contact. However, the image from the see-through camera usually exhibits a number of imaging artifacts such as low signal to noise ratio, incorrect color balance, and lost of details. Thus we develop a novel image enhancement framework that utilizes an auxiliary color+depth camera that is mounted on the side of the screen. By fusing the information from both cameras, we are able to significantly improve the quality of the see-through image. Experimental results have demonstrated that our fusion method compares favorably against traditional image enhancement/warping methods that uses only a single image

Wearable haptic systems for the fingertip and the hand: taxonomy, review and perspectives

In the last decade, we have witnessed a drastic change in the form factor of audio and vision technologies, from heavy and grounded machines to lightweight devices that naturally fit our bodies. However, only recently, haptic systems have started to be designed with wearability in mind. The wearability of haptic systems enables novel forms of communication, cooperation, and integration between humans and machines. Wearable haptic interfaces are capable of communicating with the human wearers during their interaction with the environment they share, in a natural and yet private way. This paper presents a taxonomy and review of wearable haptic systems for the fingertip and the hand, focusing on those systems directly addressing wearability challenges. The paper also discusses the main technological and design challenges for the development of wearable haptic interfaces, and it reports on the future perspectives of the field. Finally, the paper includes two tables summarizing the characteristics and features of the most representative wearable haptic systems for the fingertip and the hand

Presence-dependent Performance Differences Between Virtual Simulations And Miniature Worlds

The purpose of simulation is to avoid reality-based constraints by the implemen-tation of a synthetic model. Based on this advantage, interactive simulations have conquered all areas of applications from acquisition, and training, to research. Simulation results are transferred in many ways into reality and conclusions are drawn from the simulation to the application. Many anecdotal observations on human-in-the-loop simulations have shown a significant difference in actor behavior between simulations and reality-based applications. It seems that the factors that makes simulation so attractive, namely the absence of constraints and especially of imminent danger for persons and equipment, influence the behavior and thereby the performance of the user. These differences between simulation and reality may lead to false conclusions based on simulation results. The concept of perceiving a simulation as real and of being in the simulation is called sense of presence. This psychological construct can also be described as level of disbelief towards the simulation. Hence, differences in behavior are based on such users assessment of a simulation and subsequently are supposed to be mediated by a difference in presence. This research established significant differences in presence and performance between a simulation and a miniature-world teleoperation task. Presence and performance changed in identical tasks due to the application type and the connected danger to the robot. Also, the results supported a negative relationship between presence and performance: presence increased in the miniature-world and affected performance so that performance decreased. The causal relationship of application type→ presence→ performance was established and demands the examination of simulation based results with respect to the perceived danger to equipment, before they are transferred into the real application

Advances in Human-Robot Interaction

Rapid advances in the field of robotics have made it possible to use robots not just in industrial automation but also in entertainment, rehabilitation, and home service. Since robots will likely affect many aspects of human existence, fundamental questions of human-robot interaction must be formulated and, if at all possible, resolved. Some of these questions are addressed in this collection of papers by leading HRI researchers

DSAAR: distributed software architecture for autonomous robots

Dissertação apresentada na Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa para obtenção do grau de Mestre em Engenharia ElectrotécnicaThis dissertation presents a software architecture called the Distributed Software Architecture for Autonomous Robots (DSAAR), which is designed to provide the fast development and prototyping of multi-robot systems. The DSAAR building blocks allow engineers to focus on the behavioural model of robots and collectives. This architecture is of special interest in domains where several human, robot, and software agents have to interact continuously. Thus, fast prototyping and reusability is a must. DSAAR tries to cope with these requirements towards an advanced solution to the n-humans and m-robots problem with a set of design good practices and development tools. This dissertation will also focus on Human-Robot Interaction, mainly on the subject of teleoperation. In teleoperation human judgement is an integral part of the process, heavily influenced by the telemetry data received from the remote environment. So the speed in which commands are given and the telemetry data is received, is of crucial importance. Using the DSAAR architecture a teleoperation approach is proposed. This approach was designed to provide all entities present in the network a shared reality, where every entity is an information source in an approach similar to the distributed blackboard. This solution was designed to accomplish a real time response, as well as, the completest perception of the robots’ surroundings. Experimental results obtained with the physical robot suggest that the system is able to guarantee a close interaction between users and robot

Doctor of Philosophy

dissertationThe study of haptic interfaces focuses on the use of the sense of touch in human-machine interaction. This document presents a detailed investigation of lateral skin stretch at the fingertip as a means of direction communication. Such tactile communication has applications in a variety of situations where traditional audio and visual channels are inconvenient, unsafe, or already saturated. Examples include handheld consumer electronics, where tactile communication would allow a user to control a device without having to look at it, or in-car navigation systems, where the audio and visual directions provided by existing GPS devices can distract the driver's attention away from the road. Lateral skin stretch, the displacement of the skin of the fingerpad in a plane tangent to the fingerpad, is a highly effective means of communicating directional information. Users are able to correctly identify the direction of skin stretch stimuli with skin displacements as small as 0.1 mm at rates as slow as 2 mm/s. Such stimuli can be rendered by a small, portable device suitable for integration into handheld devices. The design of the device-finger interface affects the ability of the user to perceive the stimuli accurately. A properly designed conical aperture effectively constrains the motion of the finger and provides an interface that is practical for use in handheld devices. When a handheld device renders directional tactile cues on the fingerpad, the user must often mentally rotate those cues from the reference frame of the finger to the world-centered reference frame where those cues are to be applied. Such mental rotation incurs a cognitive cost, requiring additional time to mentally process the stimuli. The magnitude of these cognitive costs is a function of the angle of rotation, and of the specific orientations of the arm, wrist and finger. Even with the difficulties imposed by required mental rotations, lateral skin stretch is a promising means of communicating information using the sense of touch with potential to substantially improve certain types of human-machine interaction