An Object Template Approach to Manipulation for Semi-autonomous Avatar Robots

Nowadays, the first steps towards the use of mobile robots to perform manipulation tasks in remote environments have been made possible. This opens new possibilities for research and development, since robots can help humans to perform tasks in many scenarios. A remote robot can be used as avatar in applications such as for medical or industrial use, in rescue and disaster recovery tasks which might be hazardous environments for human beings to enter, as well as for more distant scenarios like planetary explorations. Among the most typical applications in recent years, research towards the deployment of robots to mitigate disaster scenarios has been of great interest in the robotics field. Disaster scenarios present challenges that need to be tackled. Their unstructured nature makes them difficult to predict and even though some assumptions can be made for human-designed scenarios, there is no certainty on the expected conditions. Communications with a robot inside these scenarios might also be challenged; wired communications limit reachability and wireless communications are limited by bandwidth. Despite the great progress in the robotics research field, these difficulties have prevented the current autonomous robotic approaches to perform efficiently in unstructured remote scenarios. On one side, acquiring physical and abstract information from unknown objects in a full autonomous way in uncontrolled environmental conditions is still an unsolved problem. Several challenges have to be overcome such as object recognition, grasp planning, manipulation, and mission planning among others. On the other side, purely teleoperated robots require a reliable communication link robust to reachability, bandwidth, and latency which can provide all the necessary feedback that a human operator needs in order to achieve sufficiently good situational awareness, e.g., worldmodel, robot state, forces, and torques exerted. Processing this amount of information plus the necessary training to perform joint motions with the robot represent a high mental workload for the operator which results in very low execution times. Additionally, a pure teleoperated approach is error-prone given that the success in a manipulation task strongly depends on the ability and expertise of the human operating the robot. Both, autonomous and teleoperated robotic approaches have pros and cons, for this reason a middle ground approach has emerged. In an approach where a human supervises a semi-autonomous remote robot, strengths from both, full autonomous and purely teleoperated approaches can be combined while at the same time their weaknesses can be tackled. A remote manipulation task can be divided into sub-tasks such as planning, perception, action, and evaluation. A proper distribution of these sub-tasks between the human operator and the remote robot can increase the efficiency and potential of success in a manipulation task. On the one hand, a human operator can trivially plan a task (planning), identify objects in the sensor data acquired by the robot (perception), and verify the completion of a task (evaluation). On the other hand, it is challenging to remotely control in joint space a robotic system like a humanoid robot that can easily have over 25 degrees of freedom (DOF). For this reason, in this approach the complex sub-tasks such as motion planning, motion execution, and obstacle avoidance (action) are performed autonomously by the remote robot. With this distribution of tasks, the challenge of converting the operator intent into a robot action arises. This thesis investigates concepts of how to efficiently provide a remote robot with the operator intent in a flexible means of interaction. While current approaches focus on an object-grasp-centered means of interaction, this thesis aims at providing physical and abstract properties of the objects of interest. With this information, the robot can perform autonomous subtasks like locomotion through the environment, grasping objects, and manipulating them at an affordance-level avoiding collisions with the environment in order to efficiently accomplish the manipulation task needed. For this purpose, the concept of Object Template (OT) has been developed in this thesis. An OT is a virtual representation of an object of interest that contains information that a remote robot can use to manipulate such object or other similar objects. The object template concept presented here goes beyond state-of-the-art related concepts by extending the robot capabilities to use affordance information of the object. This concept includes physical information (mass, center of mass, inertia tensor) as well as abstract information (potential grasps, affordances, and usabilities). Because humans are very good at analysing a situation, planning new ways of how to solve a task, even using objects for different purposes, it is important to allow communicating the planning and perception performed by the operator such that the robot can execute the action based on the information contained in the OT. This leverages human intelligence with robot capabilities. For example, as an implementation in a 3D environment, an OT can be visualized as a 3D geometry mesh that simulates an object of interest. A human operator can manipulate the OT and move it so that it overlaps with the visualized sensor data of the real object. Information of the object template type and its pose can be compressed and sent using low bandwidth communication. Then, the remote robot can use the information of the OT to approach, grasp, and manipulate the real object. The use of remote humanoid robots as avatars is expected to be intuitive to operators (or potential human response forces) since the kinematic chains and degrees of freedom are similar to humans. This allows operators to visualize themselves in the remote environment and think how to solve a task, however, task requirements such as special tools might not be found. For this reason, a flexible means of interaction that can account for allowing improvisation from the operator is also needed. In this approach, improvisation is described as "a change of a plan on how to achieve a certain task, depending on the current situation". A human operator can then improvise by adapting the affordances of known objects into new unknown objects. For example, by utilizing the affordances defined in an OT on a new object that has similar physical properties or which manipulation skills belong to the same class. The experimental results presented in this thesis validate the proposed approach by demonstrating the successful achievement of several manipulation tasks using object templates. Systematic laboratory experimentation has been performed to evaluate the individual aspects of this approach. The performance of the approach has been tested in three different humanoid robotic systems (one of these robots belongs to another research laboratory). These three robotic platforms also participated in the renowned international competition DARPA Robotics Challenge (DRC) which between 2012 and 2015 was considered the most ambitious and challenging robotic competition

Exploring the Multi-touch Interaction Design Space for 3D Virtual Objects to Support Procedural Training Tasks

Multi-touch interaction has the potential to be an important input method for realistic training in 3D environments. However, multi-touch interaction has not been explored much in 3D tasks, especially when trying to leverage realistic, real-world interaction paradigms. A systematic inquiry into what realistic gestures look like for 3D environments is required to understand how users translate real-world motions to multi-touch motions. Once those gestures are defined, it is important to see how we can leverage those gestures to enhance training tasks. In order to explore the interaction design space for 3D virtual objects, we began by conducting our first study exploring user-defined gestures. From this work we identified a taxonomy and design guidelines for 3D multi-touch gestures and how perspective view plays a role in the chosen gesture. We also identified a desire to use pressure on capacitive touch screens. Since the best way to implement pressure still required some investigation, our second study evaluated two different pressure estimation techniques in two different scenarios. Once we had a taxonomy of gestures we wanted to examine whether implementing these realistic multi-touch interactions in a training environment provided training benefits. Our third study compared multi-touch interaction to standard 2D mouse interaction and to actual physical training and found that multi-touch interaction performed better than 2D mouse and as well as physical training. This study showed us that multi-touch training using a realistic gesture set can perform as well as training on the actual apparatus. One limitation of the first training study was that the user had constrained perspective to allow for us to focus on isolating the gestures. Since users can change their perspective in a real life training scenario and therefore gain spatial knowledge of components, we wanted to see if allowing users to alter their perspective helped or hindered training. Our final study compared training with Unconstrained multi-touch interaction, Constrained multi-touch interaction, or training on the actual physical apparatus. Results show that the Unconstrained multi-touch interaction and the Physical groups had significantly better performance scores than the Constrained multi-touch interaction group, with no significant difference between the Unconstrained multi-touch and Physical groups. Our results demonstrate that allowing users more freedom to manipulate objects as they would in the real world benefits training. In addition to the research already performed, we propose several avenues for future research into the interaction design space for 3D virtual objects that we believe will be of value to researchers and designers of 3D multi-touch training environments

Development of tests for measurement of primary perceptual-motor performance

Tests for measuring primary perceptual-motor performance for assessing space environment effects on human performanc

An Investigation of Skill Acquisition under Conditions of Augmented Reality

Augmented reality is a virtual environment that integrates rendered content with the experience of the real world. There is evidence suggesting that augmented reality provides for important spatial constancy of objects relative to the real world coordinate system and that this quality contributes to rapid skill acquisition. The qualities of simulation, through the use of augmented reality, may be incorporated into actual job activities to produce a condition of just-in-time learning. This may make possible the rapid acquisition of information and reliable completion of novel or infrequently performed tasks by individuals possessing a basic skill-set. The purpose of this research has been to investigate the degree to which the acquisition of a skill is enhanced through the use of an augmented reality training device

The efficiency of bag-valve mask ventilations by medical first responders and basic emergency medical technicians

Bag-valve mask (BVM) ventilation maintains a patient\u27s oxygenation and ventilation until a more definitive artificial airway can be established. In the prehospital setting of a traffic collision or medical aid scene this is performed by an Emerency Medical Technician or medical first responder. Few studies have looked at the effectiveness of Bag-valve masks (BVM) or the complication rate of ventilating an unprotected airway. The purpose and goal of this study is to educate both medical first responders and basic emergency medical technicians

Some NASA contributions to human factors engineering: A survey

This survey presents the NASA contributions to the state of the art of human factors engineering, and indicates that these contributions have a variety of applications to nonaerospace activities. Emphasis is placed on contributions relative to man's sensory, motor, decisionmaking, and cognitive behavior and on applications that advance human factors technology

Haptics Rendering and Applications

There has been significant progress in haptic technologies but the incorporation of haptics into virtual environments is still in its infancy. A wide range of the new society's human activities including communication, education, art, entertainment, commerce and science would forever change if we learned how to capture, manipulate and reproduce haptic sensory stimuli that are nearly indistinguishable from reality. For the field to move forward, many commercial and technological barriers need to be overcome. By rendering how objects feel through haptic technology, we communicate information that might reflect a desire to speak a physically- based language that has never been explored before. Due to constant improvement in haptics technology and increasing levels of research into and development of haptics-related algorithms, protocols and devices, there is a belief that haptics technology has a promising future

Fixtureless automated incremental sheet metal forming

Die-based forming is a technology used by many industries to form metal panels. However, this method of forming lacks flexibility and cost effectiveness. In such cases, manual panel beating is typically undertaken for incremental forming of metal panels. Manual panel forming is a highly skilled operation with very little documentation and is disappearing due to non-observance and a lack of interest. Confederation of British Metal forming (CBM) and Institution of Sheet Metal Engineering (ISME) have realised the need for capturing and understanding manual skills used by panel beaters to preserve the knowledge. At the same time, industries seek for alternative panel forming solutions to produce high quality and cost-effective parts at low volume and reduce the repetitive, yet adaptive parts of the panel forming process to free up skilled workers to concentrate on the forming activities that are more difficult to automate. Incremental forming technologies, currently in practice, lack adaptability as they require substantial fixtures and dedicated tools. In this research a new proof-of-concept fixtureless automated sheet metal forming approach was developed on the basis of human skills captured from panel beaters. The proposed novel approach, named Mechatroforming®, consists of integrated mechanisms to form simple sheet metal parts by manipulating the workpiece using a robotic arm under a repetitive hammering tool. Predictive motion planning based on FEA was analysed and the manual forming skills were captured using a motion capture system. This facilitated the coordinated hammering and motion of the part to produce the intended shape accurately. A 3D measurement system with a vertical resolution of 50μm was also deployed to monitor the formation of the parts and make corrections to the forming path if needed. Therefore, the developed mechatronic system is highly adjustable by robotic motion and was closed loop via the 3D measurement system. The developed automated system has been tested rigorously, initially for bowl shape parts to prove the principle. The developed system which is 98% repeatable for depth and diameter, is able to produce targeted bowl shape parts with ±1% dimensional accuracy, high surface quality, and uniform material thickness of 0.95mm when tested with aluminium. It is envisaged that by further research, the proposed approach can be extended to form irregular and more complicated shapes that are highly in demand in various industries